Nonlinear bearing forces and moments of roller bearing and dynamic equations of roller bearing system are established. The bifurcation and the stability of the periodic motion of the system are studied by continuation-shooting algorithm for periodic solutions of nonlinear non-autonomous dynamics system, and floquet factors are used to judge the bifurcation behavior of the system. Taking a rigid rotor system in cylinder roller bearing as an example, the bifurcation of the rotor system in parameter domains of radial clearance-rotating speed, damping-rotating speed and bending moment-rotating speed are studied. Results show that the periodic motion of the rotor loses stability by Hopf bifurcation or doubling period bifurcation when radial clearance, damping and bending moment change. And the parameters of rotor bearing system are chosen to obtain periodic motion.