The coupling problem of the features selection and classifier parameters with detection accuracy of diesel engine abnormal detection is analyzed, an abnormal detection multi-objective optimization method based on non-dominated sorting Particle Swarm Optimization is proposed. The multi-aspects feature extraction including time domain, frequency domain and time-frequency domain of diesel engine vibration signal based on dual-tree complex wavelet packet transform is studied. After analysis the detection accuracy influences of features selection and classifier parameters, Non-dominated Particle Swarm Optimization algorithm is used for feature set and classifier parameters optimization. The test vibration signal analysis results illustrate that the proposed method could find the optimal feature set and classifier parameters with minimum dimensions and maximum classification accuracy, and improve the abnormal detection accuracy and efficiency .