Heating, ventilation, and air-conditioning (HVAC) systems account for around 16.4% of global final energy consumption and about 14% of global operational CO2emissions. Controlling them is a partially observable, sequential decision problem: relying solely on instantaneous sensor readings as inputs overlooks the full sequence of past conditions that shape future dynamics. To tackle this challenge and fully exploit the available temporal context, this study integrates a transformer encoder–decoder architecture into a Double Deep Q-Network (DDQN), forming a "Q-Transformer" capable of processing 24-hour sequences of observations. This approach is benchmarked against a conventional DDQN using a multilayer perceptron (MLP) and a sequence-aware Bi-LSTM within an EnergyPlus model of a 400 m2university amphitheater. In a weekly adaptation test, the Q-Transformer demonstrated rapid learning convergence, reducing energy consumption by up to 48% and 66% (occupied and unoccupied periods) compared to Bi-LSTM and MLP, respectively, while maintaining thermal and air quality comfort. After one year of simulated training on weather and occupancy data from a reference site (Luxembourg), the agent was evaluated across 4 different climatic locations, resulting energy reductions of 40% and 74%, and comfort violation reductions of 11% and 29%, compared to Bi-LSTM and MLP, respectively. These results signifies the ability of generalization and control capabilities of transformer-based reinforcement learning for adaptive HVAC management.

Buildings are one of the prominent sectors among global primary energy consumption. A large portion of this energy consumption is influenced by occupancy interaction with the buildings. Occupancy prediction in buildings without intruding their privacy helps to enhance the building energy management. Due to the complex relations of the inputs and the temporal dependency, modeling accurate occupancy predictions is highly difficult. The use of Deep Learning (DL) algorithms is one of the best approaches for accomplishing this goal. This paper provides an analysis using horizontally cascaded Long Short-Term Memory (LSTM) model as a baseline for occupancy prediction. The proposed horizontally cascaded LSTM model focuses on learning local patterns and dependencies within their input sequences, and both short term and long terms dependencies in temporal direction along with the relation between other input features, allowing for a more comprehensive understanding of the input data. This architecture can capture a broader range of information from the collected building data and learn more heterogeneity in occupancy presence behavior. The models are also compared for different prediction window sizes. The OPTUNA optimization is utilized for hyper-parameter tuning and to determine number of LSTMs to be cascaded. The proposed models function better for smaller window size and optimization of number of cascaded LSTMs are essential for improving the accuracy of the model. The paper also shows that for window sizes, 2–4 LSTMs are optimal to cascade.

The HVAC system accounted for a significant portion of the building's energy consumption, resulting in enormous CO2emissions. Among the numerous HVAC control methods, reinforcement learning (RL) gives the ability to control complex systems without requiring an explicit model of the building's thermal dynamics. This study conducted a concise review of previous research on the application of RL to HVAC systems in buildings, it offered a thorough explanation of the theoretical foundations of RL and a summary of several recent RL studies that employ a particular variant of each main component of the RL: environment, state-space, action-space, rewards function, number of time steps and training episodes. Most studies construct the training environment as a stationary MDP due to the use of a predefined single sequence of transitions for non-action-controllable state vector components (e.g., outdoor temperature and occupancy schedule). This type of MDP is solved using tabular RL and DRL algorithms. Future research should focus on using the Meta-RL approach for HVAC systems, which solves the problem of non-stationarity in the environment (non-stationarity-MDP) where the sequence of transitions for (outdoor temperature and occupancy schedule) are changeable, as is the case during the real-world implementation of the RL controller in buildings.

International audience

This paper deals with the implementation of a Model Predictive Control (MPC) system for a classroom in a container building ventilation system and the associated indoor climate through controlling the airflow rate to the zone. A dynamic thermal model for the building system is formulated using the three resistors and two capacitors (3R2C) lumped capacitance method, and linearized using the Taylor's series expansion. This model is used for the proposed MPC implementation for thermal comfort management with energy optimization. Simulation results demonstrate the significance of the MPC controller in handling the constraints, multi-objective control, and producing optimal control strategy. The energy optimization results of the MPC have shown 31% of energy consumption reduction compared to a conventional controller.