A signal processing method is proposed for rotor fault diagnosis problem based on variational mode decomposition(VMD). The center frequency and bandwidth of each component are determined in the process of the obtainment of the components by iteratively searching the optimal solution of variational model. Thus the signal frequency domain is adaptively subdivided and effectively separated from each component. The signal instantaneous frequency and amplitude of each single component can be obtained by Hilbert transform. To verify the effectiveness of the proposed method, a comparison is made to show the performance of VMD and empirical mode decomposition(EMD) in the analysis of simulation signal and some typical rotor fault signals. The decomposition results of simulation signal show that the VMD can accurately decompose the intrinsic modes and have no modal mixing. Rotor fault experiment signal analysis results show that the proposed method can effectively extract the evident fault features, so as to accurately diagnose the rotor fault.