In order to analyze the complex all-coupling model of gear system, a modeling method is presented based on prototype. Flexible rotor is introduced into mesh coupling system, the influences of varying mesh stiffness, backlashes of teeth and bearing are taken into account, and a prototype model of gearflexible rotor-bearing is established. The dynamic equations of prototype are solved and system nonlinear dynamic responses are obtained. Calculation results show that the peak of contact force is reduced clearly in the flexible rotor coupling system, the beat phenomenon appears in low frequency of torsional vibration when the flexibility of rotor increases, the nonlinear property of mesh vibration is enhanced in high speed and low loading situation, the amplitudes of mesh force is enlarged with large bearing clearance. This research provides a new method for dynamic characteristic analysis of multi rotors, multi clearances and complex coupling gear system.