In this paper, the influence of tooth surface wear on dynamic characteristics of the gear-bearing system is discussed. The gear-bearing system considering the factors of tooth surface friction, time-varying meshing stiffness, backlash and et al. is established. Based on the Fractal theory and the Archard theory, the tooth surface wear is calculated, and it is substituted into the calculation of backlash and stiffness to obtain the time-varying backlash and the time-varying meshing stiffness. The Runge–Kutta method is used to solve the dynamic differential equation of the gear system, then the phase diagrams, the Poincare section diagrams and the bifurcation diagrams of the system are obtained. The results show that compared with the constant backlash and normal-distribution backlash under the wear condition, the wear backlash calculated by the Fractal theory and the Archard theory can better show the influence of wear on the system response. With the increasing of the accumulated wear on the tooth surface, the decrement of the time-varying meshing stiffness increases, and the system is more unstable. As the friction coefficient decreases, the regions of periodic and quasi-periodic motion state increase, and the boundaries between the periodic and quasi-periodic motion state become clear. Therefore, friction mainly plays a hysteretic role on the gear-bearing system.