In the light of frequency aliasing and parameter custom caused by doubletree complex wavelet transform, a fault diagnosis method of adaptive improved dual-tree complex wavelet transform is proposed. This method integrates dual-tree complex wavelet transform-variational mode decomposition (DTCWT-VMD). First, the signal is decomposed and reconstructed by dual-tree complex wavelet transform. Particle swarm optimization (PSO) is used to determine the component kurtosis value as a fitness function to select the optimal decomposition level of doubletree complex wavelet. Second, the reconstructed low-frequency signal is subjected to spectrum analysis to extract the fault characteristic signal. The high-frequency components are reconstructed by variational mode decomposition, and through the kurtosis value, the main frequency component signal of each high-frequency component decomposed by variational mode is obtained. Finally, the spectrum of the main frequency component signals is analyzed to identify the fault frequency of the gearbox. The experimental results show that the proposed method eliminates frequency aliasing and improves the correctness of signal-to-noise ratio and frequency band selection compared with that process by the dual-tree complex wavelet transform and variational mode decomposition. Besides, it improves the ability to extract transient shock characteristics from a strong noisy environment.