This paper presents an in-plane vibration measurement technique to reduce the complexity of traditional vibration test system and eliminate its added mass influence on the measured structure. The corresponding relations between the displacement and the image edge recognition result of the measured structure are established. Because of the influence of discontinuous edges and false edges obtained by the traditional image edge recognition method on the test results， a data fitting method is adopted and the traditional image edge recognition method is improved， which make the testing results more accurate. Moreover， to verify the effectiveness of the proposed test method， a vibration test system is set up and corresponding typical experiments are carried out. Results show that a more precise edge of object is obtained by the proposed improved method compared with the traditional method. The precision of displacement calculation with the proposed improved approach is 12% higher than that with the traditional approach. Furthermore， both the vibration magnitude and direction of target object are captured precisely within the test frequency band of 0~60 Hz. It can be concluded that the in-plane vibration testing system proposed in this paper has the advantages of simple set-up， flexible frequency range and sufficient test data， and it is promising in finding an alternative method with limited testing conditions.