Abstract Aiming at the non-linear dynamics characteristic which exits in the gearing transmission while being ignored in the traditional fault diagnosis, cause of detecting signal according to the change of phase space track in the chaotic oscillator detection theory, this paper established a single-tooth fault dynamics model of non-linear gear system based on impact analysis. Through analysis, we find that given determinate parameters the dynamics characteristic will turn into chaos state; in addition, using the same parameters, after single-tooth pulse fault appears and reaches to a certain degree, gear system will turn into periodic phase state which shows different dynamics characteristic from normal gear system obviously. Simulation results show that this method can distinguish the single-tooth fault effectively which offers a new method for fault diagnosis of the rotary machinery.