A complete dynamic model of a timing belt drive system with an oval cogged pulley and an auto-tensioner is established in this work. Periodic torsional vibrations of all accessory pulleys and the tensioner arm are calculated using a modified incremental harmonic balance (MIHB) method based on the complete dynamic model. Calculated results from the MIHB method are verified by comparing them with those obtained from Runge–Kutta method. Influences of tensioner parameters and oval pulley parameters on torsional vibrations of camshafts and other accessory pulleys are investigated. A sequence quadratic programing (SQP) method with oval pulley parameters selected as design variables is applied to minimize the overall torsional vibration amplitude of all the accessory pulleys and the tensioner arm in the timing belt drive system at different operational speeds. It is demonstrated that torsional vibrations of the timing belt drive system are significantly reduced by matching belt stretch with speed variations of the crankshaft and fluctuating torque loads on camshafts. The timing belt drive system with optimal oval parameters given in this work has better performance in the overall torsional vibration of the system than that with oval parameters provided by the kinematic model and the simplified dynamic model in previous research.