According to the elastic-plastic deformation of vibration-compacted material on slant excitation mode, a novel dynamical model is proposed in terms of the segmental line horizontal symmetric hysteretic model and vertical asymmetric hysteretic model according to soil property parameters. Based on first order approximation, the equivalent damping and stiffness coefficients are deduced. Meanwhile, the response equation of the system and its analytic solution are derived through the KBM approximation method. Numerical analysis shows that the equivalent natural frequency of the compaction system decreases and the wheel vibration waveform distorts as the excitation amplitude increases due to hysteretic nonlinearity: multiple harmonics present vertically and only odd harmonics occurs horizontally in addition to the fundamental wave. Simulation results are verified through the on-site construction process. The conclusion of the article is valuable for theoretically monitoring and improving the compaction quality.