An optimization method for vibration reduction of small-scaled smart rotor with trailing edge flaps is presented. Both the inertial forces and aerodynamic forces due to the deflection of trailing edge flaps are concerned in this model. A surrogate model is developed to calculate the aerodynamic forces of flapped airfoils. The aeroelastic dynamic equations are solved with the implicit trapezoid method to get the elastic response of blade, and the vibratory blade loads and hub loads are predicted with a force integration method. The flap deflection harmonics are the design variables and the amplitutes of vibratory hub load are chosen as the objective function. The best flap deflection law for hub vibration control is found with the steepest descent method. Results show that both structural and aerodynamic loads of rotor can be precisely calculated with the current model. Hub vertical vibratory load can be effectively reduced with properly controlled flaps at different advance ratios. The lack of deflection ability can be simulated with the direct constraint method or the objective weight method. The deflection ability of trailing edge flaps significantly influences the vibration reduction effect. Despite the limited deflection angle due to the ability of actuator, vibratory loads can still be reduced with actively controlled flaps.