Rotor mass unbalance is the most common cause of excessive vibration in rotating machinery and can affect the work performance and safety operations of the system. The traditional least squares algorithm is sensitive to outliers in the process of estimating rotor system unbalance. By manually assigning different weight coefficients, the weighted least squares algorithm can reduce the impact of outliers, but requires practical experience and have deep analysis of the vibration data. In view of these problems, a new unbalance estimation method based on robust regression analysis is proposed. The effect of outliers is automatically eliminated by an optimized objective function. Experimental results show that the proposed method can effectively reduce the influence of outliers and more accurately identify the unbalance of the rotor system.