In order to solve the boundary distortion problem in the empirical mode decomposition (EMD) method, a novel boundary processing method using a cosine w indow is proposed. The cosine window is firstly defined that its values equal to 1 in middle of the window and reduce according to the laws of cosine／sine func tions in both ends. Secondly, the analyzed signal is preprocessed by multiplying with the defined window, so that the amplitudes of the signal remain in middle and reduce in both ends. Lastly, EMD method is used to analyze the preprocessed signal. With the proposed method, the boundary distortion can be controlled in e nds of the signal and the middle component can be correctly decomposed. The resu lts show that the proposed method can effectively decrease the boundary distorti on of EMD and is useful in friction fault diagnosis of rotor system.