The advantages and disadvantages of several approaches for restraining the end effect in HHT are discussed and a re-optimization slope based method(RO-SBM) to dealing with the end issue is proposed based on the slope-based method( SBM)and improved slope-based method(ISBM).Numerical simulation with a non-stationary, nonlinear data series has proved that the RO-SBM can be used to solve the end issue more effectively than the mirror method (MM) and the ISBM, and the HHT can also get better performance of signal analyzing by using the RO-SBM for data extension. Finally, the HHT can get accurate fault feature because of the radial rub from the radial vibration signal in a rotor system by using the RO-SBM and therefore it can be applied successfully in the field of mechanical failure diagnosis.