A nonlinear torsional dynamic model excited by diesel engine is established, wherein the backlash, time-varying mesh stiffness and its engaging phase difference, gear mesh composite errors and time varying planet position angle are considered. On the basis of signal modulation principle, the reason of modulation sidebands of planetary gears is analyzed. According to different rotational speed and torque, engine operational terms are categorized into four work conditions. The torsional vibration angle and meshing force are described in details by time history and frequency spectrum. The results demonstrate that meshing force is at single-sided impact state mostly, and frequency modulation takes more great effect on meshing force. Frequency modulation that associates with rotational frequency of parts relative to meshing frequency widely exists in different working conditions, and it does not come under the influence of working conditions. Frequency modulation that associates with rotational frequency of fluctuant torque to meshing frequency is opposite to the former, and it only operates definitively in high speed heavy load condition. It provides reference for the design of planetary gears in automobile transmission.