A nonlinear dynamic model of gear-rotor-bearing transmission system is developed with synthetically considering dynamic backlash,time-varying meshing stiffness, gear eccentricity and transmission error, using weierstrass-mandelbrot function with dimensional consistency to simulate the variation trend of dynamic backlash. The nonlinear dynamic differential equations are solved using the Runge-Kutta method, and the effect of dynamic backlash on system response is explored. The results show that the vibration amplitude of gear prominently fluctuates in torsion direction, and system kinestate is changed from quasi-periodic motion to chaotic motion with increasing complexity of dynamic backlash curve. The phase plane trajectories of gear in torsion direction present regular variation with the standard deviation value (STD) for short of the dynamic backlash. With increasing complexity of dynamic backlash curve, the diversity of frequency components and the amplitude of noise frequency clearly argument and the amplitude of high frequencies increases sharply.