A driveline system model of six degrees of freedom is established, and the meshing clearance between the geared gears in the transmission are modeled. The dynamic response is calculated using the developed model to estimate angular acceleration for each rigid body in the model. Taking the angular acceleration of the input shaft for transmission as an evaluation index, the influence of model parameters on the angular acceleration of the input shaft is studied. Based on the developed model, the parameters of the moments of inertia for flywheel and stiffness of clutch are optimized by sequential quadratic programming. Then, in the experiment, the torsional angle acceleration of the input shaft for transmission, the vibration acceleration of transmission and the sound pressure at the transmission side at an engine compartment are measured and compared on the baseline clutch and the optimized clutch to measure. The results show that the angular acceleration amplitude of the input shaft and the gear impact of unloaded gear rattle can be effectively decreased by increasing the inertia of the flywheel and the damping torque of clutch, or by reducing the inertia of the unloaded gear on the output shaft. The unloaded gear rattle is decreased greatly using the optimized clutch, which validates the calculation and experiment methods of this study.