In order to study the influence of friction on the non-linear dynamic characteristics of the gear transmission system, a torsional vibration model with three gears is built that takes into account gear backlash, gear-face friction force, and time-varying mesh stiffness. The effect of layout parameters on gear-face friction force and time-varying mesh stiffness is analyzed. Both the system′s dynamic response under different friction coeficients and its chaos motion in the presence or absence of friction coeficients are discussed. The system′s jump-lag phenomenon and impact phenomenon are discussed using a frequency response curve, while their response to friction coeficients is analyzed. The results indicate that with changes in friction coeficients, the system shows coexistence of the same periodic motion, coexistence of different periodic motion and chaos and system′s chaotic motion; the jump phenomenon is ahead because of the friction, and the gear′s impact motion is increased by the effect of friction. The results provide a reasonable theory reference for the nonlinear dynamic design of the convergent transmission gear system.