Ensemble empirical mode decomposition (EEMD) is a good method for suppressing the mode mixing in empirical mode decomposition (EMD), but its decomposition effect depends on two important parameters, which are the amplitude of added noise and the number of ensemble for EEMD. On the present time, these two important parameters mainly depend on experience and lack of reliability. In order tosolve above problems, an improved algorithm named adaptive ensemble empirical mode decomposition (AEEMD) is proposed. Firstly, the amplitude standard deviation of original signal is computed; secondly, the original signal is decomposed by high-pass filters, and then the amplitude standard deviation of high frequency components is computed. Thus the amplitude standard deviation coefficient of added noise can be determined by the amplitude standard deviation of original signal and its high frequency components. Thirdly, the number of ensemble can be determined by the expectation AEEMD decomposition error and the amplitude standard deviation coefficient of added noise. Lastly, in order to decrease the mode mixing further and enhance the orthogonality of adjacent intrinsic mode function, an AEEMD post-processing method using EMD is proposed. The results of the simulation and experiments indicate that the proposed AEEMD algorithm can overcome mode mixing, and is successfully applied in the fault diagnosis of rotary machine, the result demonstrates that mode mixing can be controlled effectively, and then is compared with basic EMD algorithm. The results show that AEEMD is more precise,and it can exactly recognize rotary machine faults.