Machining chatter caused by the regeneration effect of dynamic cutting thickness in milling process restricts the machining quality and efficiency. Online monitoring of machining process can effectively identify chatter signals, which is the basis of chatter suppression and parameter optimization. The traditional chatter recognition methods based on signal statistical features have the difficulty of designing indicators and threshold determination. The existing feature adaptive extraction method of converting signals into image has the problems of redundant input information, resulting in complex model structure. This paper considers the frequency domain characteristics of chatter signals, and designs an adaptive frequency domain feature extraction network based on channel attention mechanism, which greatly reduces the number of model parameters and calculation. Experimental results show that the accuracy of the proposed method is more than 97%, and the recognition efficiency is significantly improved.