With the difficulty of gear system diagnosis at the complex incipient tooth failure under complex structure and work condition, the transmission error rule of different incipient gear fault is analyzed. The different transmission error models are established and analyzed. According to the order feature of the gear system transmission error signal, the fault component of the transmission error signal from the transmission error signal is isolated by adopting improved notch filtering method. Experimental examples are conducted to evaluate the effectiveness of the proposed approach. The results show that transmission error signal had high SNR and the fault component of the transmission error signal is good to diagnosis different incipient tooth wear under the light degree load and different speed work condition.