Wellbore instability is a critical issue restricting efficient well drilling and successful development of oil and gas field. Most instability problems originate from shale formations because of their distinct laminated structures that cause significant anisotropy and moderate to high clay contents that are prone to shrinkage and swelling. To account for these influences on the mechanical responses of shales, this study aims to identify an appropriate strength criterion for stability analyses. Two anisotropic criteria including single plane of weakness and the modified Hoek–Brown criteria were compared to evaluate their suitability in characterizing the anisotropic strength of layered rocks including shale, schist, and slate under different confining pressures. Comparative case studies indicated that the single plane of weakness criterion overestimates the strength of layered rocks at some orientation angles. The modified Hoek–Brown criterion can fit well with the experimental data of layered rocks. Moreover, wellbore stability analysis models for shale gas wells were built, respectively, for each criterion and applied to in situ scenarios. The single plane of weakness and modified Hoek–Brown criteria provide similar results of collapse pressure, and the shale failure is mainly determined by the bedding plane. This further validates that the modified Hoek–Brown criterion is a good choice for wellbore stability analysis in shale formations with bedding planes. This study shows the potential of using the modified Hoek–Brown criterion to enhance the safety and efficiency of well drilling and operation in shale formations.