Building temperature prediction is crucial for energy optimization and control in smart cities. We present a physics-enhanced XGBoost framework in a multi-stage sequential scaling approach. Starting from single-zone, single-day predictions, we progressively scale to multi-zone, multi-year forecasts using real-world data from Google's Smart Building Simulator. Our method incorporates physics-enhanced features, temporal encodings, and inter-zone interactions, achieving mean absolute errors (MAE) as low as 0.169°F for weekly multi-zone predictions. For longer horizons, we employ ensemble strategies, demonstrating robust performance up to 2.5 years. Compared to baseline models, our framework consistently improves long-term prediction fidelity. This work advances urban AI by enabling accurate long-term building dynamics modeling for downstream control tasks and bridges machine learning with physics-based modeling approaches.