A new structural system identification method based on variational mode decomposition(VMD) and Teager energy operator(TEO) is proposed. The structural modal characteristics are identified based on the pole stability of VMD layer parameter K. In order to satisfy the requirement of TEO for single component, the vibration signal is decomposed into detail signals of different scales by VMD method. For bandlimited intrinsic mode function(BIMF), TEO method isused to estimate the natural frequency and damping ratio. By comparing the pole stability of VMD layer parameter K modal characteristic parameter, the false component is removed. The numerical and experimental results show that the proposed method is effective, accurate and feasible in both traditional modal test modal analysis and environmental excitation modal analysis.