In this paper, the nonlinear meshing dynamic model of planetary gear system is proposed. The influence of the variable center distance caused by the installation error and the bending deformation of transmission shaft on the meshing angle, backlash and nonlinear meshing stiffness are considered. Considering the variable center distance and gyroscopic moment and the nonlinear meshing dynamic model of gear, the nonlinear dynamic model of the lateral-torsional coupling of planetary gear system is established. The simulation calculation and test of planetary gear system equipment are carried out. The vibration displacement and dynamic load coefficient of gear ring are compared and analyzed. The results show that the variation trend of the simulation results is basically consistent with the experimental results, and the error is small. The correctness of the nonlinear dynamic model and the nonlinear dynamic meshing model of the gear transmission are verified.