Certain factors bring tremendous trouble to the fault diagnosis of the planetary gear system of wind turbines, such as a complex signal transfer path, severe noise pollution and signal couple modulation. These factors become more severe under unstable work conditions and time-varying speed. In order to solve the fault diagnosis problems caused by these factors, a method based on order envelop analysis (OEA) was proposed, and its principle and implementation were elaborated. The OEA method was used to analyze the vibration acceleration signals of a planetary system with tooth faults and time-varying speeds. The fault characteristic order of different parts (planet, sun gear and ring gear) was obtained through comparison to the order feature of a normal gear system. Experimental tests show that by combining the advantages of both order analysis and envelope analysis, the OEA method was efficient and has the advantage of strong anti-noise ability. Furthermore, the fault characteristic orders of different parts were shown to be useful for detecting and localizing faults in planetary gear systems under time-varying speed conditions.