Reinforcement Learning from Human Feedback (RLHF) has become the dominant paradigm for aligning large language models (LLMs) with human values, yet its environmental cost remains nearly invisible in both academic discourse and practitioner tooling. This paper presents the first phase-resolved empirical characterization of RLHF’s carbon footprint, combining Marginal Operating Emission Rate (MOER) based carbon accounting, real-time grid carbon intensity (CI) signals, and a multi-region 90-day deployment study, a scope and resolution not addressed by prior machine-learning (ML) carbon measurement work. We find that the Proximal Policy Optimization (PPO) phase alone accounts for 59–64% of total training emissions across all model scales, reaching 78.9 kgCO2eq for a single LLaMA-65B alignment run. We introduce CarbonAware-RLHF, an open-source scheduling framework that adaptively pauses, reorders, or temporally shifts RLHF phases toward low-carbon windows using real-time CI forecasts. In a 90-day deployment across six United States (US) Independent System Operator (ISO) grid regions (312 jobs, three independent seeds per condition), CarbonAware-RLHF achieves a mean carbon reduction of 38.9% with only a − 0.22 pp (percentage-point) reward model accuracy degradation and no statistically significant change on MT-Bench safety evaluation, the best carbon-quality trade-off among all evaluated strategies. Carbon savings correlate strongly with grid CI variability (coefficient of determination R2 = 0.91), providing a practical deployment heuristic. We discuss geographic equity, carbon accounting standards, and the Jevons paradox as structural challenges that technical scheduling alone cannot resolve. All data, code, and reproducibility artifacts are released at https://github.com/Saher-Elsayed/carbonaware-rlhf.