In order to study the detection and diagnosis problem of railway out-of-round wheels, a method based on improved Hilbert-Huang transform(HHT) was utilized. First, in light of the mode mixing phenomenon of HHT, a novel morphology filtering and energy principle algorithm was put forward to eliminate mode mixing. Then, the vehicle track coupling dynamics model and representative wheel fault models were set up to calculate the vertical vibration dynamic response of the axle box. Finally, the axle box vertical vibration characteristic induced by a normal wheel, polygonal wheel and flat wheel were studied with revised HHT. The research results show that there was a significant difference in the Hilbert spectrum between healthy wheels and fault wheels, as well as fault wheels with different fault types. This method can thus effectively diagnose faults of out-of-round wheels.