Piezoelectric actuators have been widely applied in scanning tunneling microscope (STM). A controller combining inverse hysteresis model and PID control law was designed for the STM to compensate for the hysteresis nonlinearity of piezoelectric actuators and to improve the scanning accuracy. For the purpose of comparing the proposed controller with classical displacement feedback control law, experimental system was built in this investigation. The relative measuring error of grating period and width of grating line were 4.41% and 2.65% using the classic displacement feedback control method respectively. And they were reduced to 1.26% and 0.27% in proposed method. Experimental results indicate that the error caused by hysteresis nonlinearity has been well compensated, and the scanning accuracy is improved with the help of such controller.