The torsional vibration of a large equipment transmission system affects its normal operation and even leads to serious damage. An effective means to relieve the torsional vibration is avoiding resonance. A large mill equipment was targeted, and its transmission system was analyzed based on the revising theory of the finite element model (FEM) of the response surface. A response surface model was built to approach the complicated implicit function of the characteristic quantity and design parameters. After choosing the significant parameters, the probable errors existing in the FEM model of the system were revised. Then, the transmission system was optimized based on the new FEM, and its modal frequency was adjusted to avoid the working frequency by updating the FEM. Finally, the optimization of the transmission system was achieved, which indicates that this method can guide the design of the transmission system.