This paper proposes an adaptive short-time Fourier transform (ASTFT) method based on fast path optimization (FPO). The proposed method is applied to the fault diagnosis of planetary gearbox. This method can estimate the instantaneous frequency from the time-frequency distribution. Thus, the time window size can be adaptively adjusted by considering the instantaneous frequency. According to the characteristics of the planetary box, this method can be used to effectively extract the speed information and resample the fault signal in angle domain from which we can get the fault orders of the planetary gearbox. The vibration analysis of a gearbox demonstrates its effectiveness for fault diagnosis.The performance of the proposed method in this research work is validated by both the simulated and experimental signals including a vibration signal. Compared with the STFT, the proposed method demonstrates its superb accuracy and perfection in characterizing the time-varying time-frequency features of the signal.