Traditional signal processing methods are applied to nonlinear Lamb waves, but the precision and noise immunity are insufficient. Duffing van-der-pol system is a kind of nonlinear dynamical system, and a method is proposed to detect the weak changes of nonlinear Lamb wave, which are caused by the material nonlinearity based on this system. Due to the limitation of traditional method, the geometric characteristics of the system (the average periodic area) are used as the characteristic parameters to enhance the weak second harmonic and to quantify the nonlinear characteristics of the material. In this paper, the internal force of the system is determined by bifurcation diagram. Then, according to the system phase trajectory, the existence of the second harmonic is determined by the system state identified, and the geometric characteristics are used to calculate the amplitude of second harmonic. Furthermore, the linear relationship between the second harmonic amplitude and the geometric characteristic parameters is analyzed. Ultimately, the nonlinear coefficient of the material can be acquired accurately. Compared with the wavelet transform and Kalman filtering, this method can still maintain high precision under strong noise.