The algorithm of employing order tracking based on the time-frequency representation is limited in rolling bearing fault diagnosis under a variable rotational speed and gear vibration noise, because of the lack of extractable rotational frequency components. The algorithm of angle domain resampling based on fault characteristic frequency can extract the bearing fault characteristics, however the location of the fault cannot be found and error may occur in this algorithm. The method is proposed in this paper based on auto regressive (AR) model filtering in angle domain to solve this problem.To estimate the rotational speed, the gear instantaneous meshing frequency is extracted from the down sampling mixed signal using chirplet path pursuit algorithm. The mixed signal is re-sampled by a constant angular interval based on the estimated rotational speed. The gear noise is removed in the angle domain signal used AR model. Finally, the fault diagnosis is completed by observing the order spectrum gotten by Hilbert transformation and FFT. The effectiveness of the method is tested by the analysis of the simulation signal and experimental signal.