Extensive prior studies aimed at the development of diagnostic methods for planetary gearboxes have mainly examined acceleration and acoustic emission signals. Recently, due to the relationship between gear mesh stiffness and transmission error (TE), TE-based techniques have emerged as a promising way to analyze dynamic behavior of spur and helical gears. However, to date, TE has not been used as a measure to detect faults in planetary gears. In this paper, we propose a new methodology for model-based fault diagnostics of planetary gears using TE signals. A lumped parametric model of planetary gear dynamics was built to extract simulated TE signals, while accounting for the planet phasing effect, which is a peculiar characteristic of the planetary gear. Next, gear dynamic analysis was performed using TE signals, and TE-based damage features were calculated from the processed TE signals to quantitatively represent health conditions. The procedures described aforesaid were then applied to a case study of a planetary gear in a wind turbine gear train. From the results, we conclude that TE signals can be used to detect the faults, while enhancing understanding of the complex dynamic behaviors of planetary gears.