The conventional wavelet denoising method based on the simple threshold principle cannot always successfully extract the weak fault feature from the vibration signal with strong background noise, as the noise of different layers is individually estimated. Inspired by the fact that the wavelet transform coefficients of adjacent layers have some similarities where the signal is singular, a new method based on dual-tree complex wavelet transform and correlation filter is proposed. The method is an interactive process in which its parameters are adaptively selected and the noise can be efficiently reduced. More important, the faint component that is expected to be extracted will be retained. The defect frequency can be accurately found by the envelope demodulation analysis. As an improvement to the conventional wavelet transform domain correlation filter, it takes full advantage of dual-tree complex wavelet transform. Experimental and engineering application examples show the method's effectiveness in incipient gearbox fault diagnosis.