With a two pairs of helical gear coupled rotor system expander subsystem with three parallel axes being studied，helical gear meshing pair dynamics model and finite element model of rotor system are established, in which the effect of mesh stiffness, gear position angle, transverse pressure angle and helical angle are taken into consideration. Also the gear meshing stiffness matrix is derived. The analysis of natural characteristics and unbalance response of the coupled lateral-torsional expander subsystem are then analyzed based on MSM (mode superposition method). Unbalanced response curves of system loading point are compared with considering meshing or not. The result shows that the vibration between the systems becomes intense due to the coupled gears. Unbalance response of the system maybe accumulated until the existence of the maximum vibration phenomen on in a non-natural frequency. Meanwhile, the generation mechanism of resonance peaks is identified. The system vibration mode is performed as the coupled mode because the coupled gear exerts a profound influence on the dynamic characteristics of the rotor system. So the vibration characteristics of the system must be analyzed with the coupled method, which can provide the basis for preventing the big vibration of the system, and also give the guiding significance to the design and fault analysis of the gear system..