Abstract Based on nonlinear contact theory and multi-body dynamics theory, the multi-body dynamics model of some planetary gear transmission system is built to research on the dynamic characteristic in actual operating condition, then the real-time dynamic contact force of gear teeth and the dynamic equivalent stress of key components are achieved. The simulation results are in good agreement with the theoretical analysis. The results show that using rigid-flexible coupling multi-body dynamics model to study the dynamic performance of planetary gear drive system is more in line with the fact and analytic results are reasonable and reliable. From the research, the reference can be offered for optimal design of gear system and developing new product.