Order to probe into the original nonlinear characteristics of fictional brake groan, two test conditions are devised, which are starting on level road under D shift and coasting on gradient road under neutral gear. Under these conditions caliper 3D vibration signals are sampled and analyzed in time domain, time-frequency domain and phase diagram. The analysis results show that the brake groan during D shift starting on level road conditions is an obvious short-time impact with a wide frequency band. The vibration amplitudes increase with the augment of brake pressure, and the durations rise up with the increase of the pressure reduction rate. Brake groan during neutral gear coasting on gradient road conditions, can be a short-time impact with a wide frequency band, or a periodic stationary vibration with multi-frequencies. Under different relative velocities, periodic, period-doubling, multi coupled period vibrations, even chaotic vibration will occur due to the nonlinear friction-velocity characteristics. Three basic frequencies and their coupled relationships indicate that the frictional mechanism of brake groan is not only induced by the stick-slip effect, but also the nonlinear mode coupling effect in a chassis corner system.