Since the main cause of vibration and noise during transmission is unsteady gear meshing, it is necessary to examine its dynamic process and laws. This paper adopts the finite element method and slice theory to study a pair of helical gears in an automotive transmission (AT) and calculate its transmission error (TE), in order to evaluate the steadiness level of the transmission process. An effective way to reduce noise and vibration is gear modification; based on the empirical equations of gear modification, the micro-geometric size range was first decided. In order to determine the optimal method for gear tooth modification, the enumeration method was applied to the targets of transmission error reduction and load distribution improvement on the tooth surface, based on thin slice theory. The analysis results of the bench test of the AT showed the validity and effectiveness of the proposed method and model.