Rolling mill vibration signals that are time-varying and random are often affected by noise interference, so that frequency domain analysis cannot identify and extract fault features. The adaptive frequency domain filtering method has been proposed to successfully separate noise signals and vibration characteristics based on short-time Fourier transform (STFT). First, according to STFT and a threshold set by the power method, the dominant frequencies of vibration signals in different time periods are automatically captured. After the vibration signal, hydraulic cylinder pressure signal, and servo valve reference signal have been captured by STFT, the hot continuous rolling mill is diagnosed with mechanical-hydraulic coupling vibration. With the combination method of discrete wavelet transform and S transform, the rolling mill vibration signal is analyzed, which determines that the rolling mill vibration start-up time is the investment time of the hydraulic pressure system, thus proving the existence of mechanical-hydraulic coupling vibration.