Crack detection is the main task of full-size aircraft structural fatigue test. Crack recognition based on computer vision can replace manual visual inspection to achieve timely， accurate and reliable damage detection in large， complex and dangerous test environment. At present， the main challenge of computer vision recognition model is how to identify fatigue cracks from surface textures， scratches and stains at the same millimeter scale. Therefore， a crack detection method for aircraft structures based on bidirectional fusion network （BF-Net） and siamese difference is proposed. Firstly， a hierarchical detection strategy is employed to extract the key regions where cracks may occur. Then， the BF-Net for small object recognition is designed to extract high-quality feature presentation of crack from regional images， through top-down and bottom-up feature fusion operations. Finally， the siamese difference is proposed， to comprehensively identify cracks， based on the characteristic information of the key region and the characteristic difference information between the region and the template. Experimental results illustrate that the method proposed in this paper can realize the accurate detection for small cracks， indicating a promising technique for automatic crack detection in fatigue test.