For the problem of rotary machine nonlinear feature extraction, a method based on generalized fractal dimension (GFD) and kernel principal component analysis (KPCA) is proposed. Firstly, GFD is used for feature extraction and formed a high dimensions feature space. Secondly, KPCA is used for dimensionality reduction in high dimensions space and feature extraction ulteriorly. Finally, data in different running conditions of a rotor system and faulty bearing are classified using the methods of KPCA and K nearest neighbor (KNN). The result shows that this GFD-KPCA method can effectively extract features, accurately classify data in different conditions, and has a low dependence on selecting parameters. Bearing weak fault vibration feature extraction results show that the performance of GFD-KPCA is better than that of conventional KPCA feature extraction algorithm, which has better accuracy and scope of application.