Most of researcher assume that there exist cubic nonlinear term directly when investigate the non-linear phenomena of torsional vibration of rotor. However, they do not explore the reason why emerge non-linear phenomena. In this paper, softing duffing non-linear terms is gotten in torsional vibration of rotor with geometry deformation. Then modal analysis are carried out by use of ANSYS software without considering the influence of nonlinear terms. A multi-degree of freedom nonlinear modal are proposed, and principal resonance is obtained taking advantage of mode truncation technique and the method of multiple scales. Finally, through the quantitative analysis between numerical simulation and frequency sweep test of torsional vibration, the results indicate the accuracy of softing duffing nonlinear phenomena in a rotor torsional system.