Based on the characteristics of failures inside an aero-engine turbine, we introduced a three-dimensional fiber-optic tip clearance measuring method using the displacement sensor of a two-circle reflective coaxial fiber bundle. Next, we built a three-dimensional modulation function for the sensor when there was radial clearance and inclination in both the axial and circumferential directions of the reflector. We also built a compensation algorithm for the overlapping area between the receiving fiber and light spot, and a correction algorithm of average light intensity on the surface of the virtual image of the receiving fiber. Then, we obtained the simulation results of the three-dimensional modulation function when the reflector was parallel or declining relative to the probe, and compared them with the experimental results. Finally, we simulated output behavior with different design parameters of the sensor. We concluded that the newly built three-dimensional modulation function of the two-circle reflective coaxial fiber bundle not only reproduced the real output behavior of the sensor, but also provided theoretical guidance in the design process of the fiber-optic probe based on any measuring requirement for three-dimensional displacement.