Aimed at the planetary drive system used in vehicles, the rigid-flexible coupling dynamic model of the planetary system is created based on the multi-body dynamics theory and the gear system dynamics theory. Dynamics characteristic of the system is studied in two different operating situations of the ideal steady-state and the actual, complex speed-changing state, and then the real-time dynamic meshing force of key components and some significative results in the transmission process are gained. The simulation data are in good agreement with the theoretical analysis and the simulation results are in line with the actual. From the research above, an effective technical means for engineering application s is offered for optimal design and developing new product on gear drive system.