On the basis of the relation between force and deformation when the plastic deformation of soil mass is studied, nonlinear vibratory roller model is built. Based on one type vibratory rollers, the system parameters are selected and the natural frequencies are calculated. And the numerical simulations with different excitation force frequencies are carried out according to the selected numerical value. Meanwhile, the experimental data can be obtained and analyzed according to the vibratory roller experiment. Then the system response characteristic of no nlinear vibration roller is obtained, and the availability of system model is checked, which provide the valuable theoretical basis for the research of vibrating compacting.