With the consideration of railway irregularity, a finite element model of vehicle/bridge system is established in this study, and the damage detection of different bridge patterns is achieved using vehicle response with genetic algorithm (GA). The damage location and degree of bridge are taken as recognition factors. The objective function is constructed by comparing the simulated and measured vehicle response data with some vehicle-response-analysis data. For the assumed damage cases, every detection process is independently calculated for several times, and the successful rate and the average iterative number are used to evaluate the detection efficiency of GA when the first optimal solution appears. This study shows that GA can achieve the bridge damage detection with a high detection efficiency not only for single object detection but also for multi-object detection. The search space size has a great impact on the GA detection efficiency. Therefore, this method is much more stable for the damage located at the middle and 3/4 span than at the ends.