Building energy management is essential for achieving carbon reduction goals, improving occupant comfort and reducing energy costs. Coordinated building energy management faces critical challenges in exploiting spatial-temporal dependencies while ensuring operational safety across multi-building systems. Current multi-building energy systems face three key challenges: insufficient spatial-temporal information exploitation, lack of rigorous safety guarantees, and system complexity. This paper proposes Spatial-Temporal Enhanced Safe Multi-Agent Coordination (STEMS), a novel safety-constrained multi-agent reinforcement learning framework for coordinated building energy management. STEMS integrates two core components: (1) a spatial-temporal graph representation learning framework using GCN-Transformer fusion architecture to capture inter-building relationships and temporal patterns, and (2) a safety-constrained multi-agent RL algorithm incorporating Control Barrier Functions to provide mathematical safety guarantees. Extensive experiments on real-world building datasets demonstrate STEMS’s strong performance over existing methods, showing that STEMS achieves 21% cost reduction and 18% emission reduction over traditional methods, 65% safety violation reduction compared with learning-based baselines, and maintains the lowest discomfort rate. The framework also demonstrates strong robustness during extreme weather conditions and maintains effectiveness across different building types.

Accurate electric load forecasting is of critical importance for modern power grids. It can help optimize energy management, reduce operational costs, and enhance grid stability. Existing load forecasting tools typically perform well when modelling long-term trends with substantive data upon which to build a model, but can perform poorly for short-term load forecasting or when data is sparse or incomplete. Diffusion models have recently emerged as powerful generative tools that excel in modelling complex distributions, making them a promising approach for electric load forecasting. In this paper, we build upon recent diffusion models for time series forecasting and explore the potential of combining diffusion models with prior models to improve performance. Additionally, we propose a metric-based meta-learning approach for fast data adaptation. Experimental results with this metric-based meta-learning approach on real-world load forecasting datasets outperform state-of-the-art baselines, showcasing the potential of diffusion-based refinement in practical forecasting applications.

Time series data permeates our daily existence and has been recognized as of significant importance for many sectors, such as energy, transportation, telecommunication, and health care. Ensemble learning stands as a prevalent technique in time series forecasting, adept at handling dynamic data distributions and enhancing robustness. Nevertheless, the determination of model combination weights for computing ensemble outputs remains an unresolved issue. In this endeavour, we introduce a comprehensive reinforcement learning-based approach to tackle this challenge. Moreover, our method remains agnostic to the base models within the ensemble and exhibits versatility across a spectrum of time series forecasting tasks, thereby holding promise for application across diverse real-world scenarios. In this work, we extensively test the performance of our method on real-world datasets. Experimental results show that our proposed method achieves an average 6.2% forecasting accuracy improvement over all other single baseline methods.

Multivariate time series (MTS) forecasting has a wide range of applications in both industry and academia. Recently, spatial-temporal graph neural networks (STGNNs) have gained popularity as MTS forecasting methods. However, current STGNNs can only use the finite length of MTS input data due to the computational complexity. Moreover, they lack the ability to identify similar patterns throughout the entire dataset and struggle with data that exhibit sparsely and discontinuously distributed correlations among variables over an extensive historical period, resulting in only marginal improvements. In this article, we introduce a simple yet effective k-nearest neighbor MTS forecasting (kNN-MTS) framework, which forecasts with a nearest neighbor retrieval mechanism over a large datastore of cached series, using representations from the MTS model for similarity search. This approach requires no additional training and scales to give the MTS model direct access to the whole dataset at test time, resulting in a highly expressive model that consistently improves performance, and has the ability to extract sparse distributed but similar patterns span over multivariables from the entire dataset. Furthermore, a hybrid spatial-temporal encoder (HSTEncoder) is designed for kNN-MTS which can capture both long-term temporal and short-term spatial-temporal dependencies and is shown to provide accurate representation for kNN-MTS for better forecasting. Experimental results on several real-world datasets show a significant improvement in the forecasting performance of kNN-MTS. The quantitative analysis also illustrates the interpretability and efficiency of kNN-MTS, showing better application prospects and opening up a new path for efficiently using the large dataset in MTS models.

In this work, we investigate how to leverage pre-trained visual-language models (VLM) for online Reinforcement Learning (RL). In particular, we focus on sparse reward tasks with pre-defined textual task descriptions. We first identify the problem of reward misalignment when applying VLM as a reward in RL tasks. To address this issue, we introduce a lightweight fine-tuning method, named Fuzzy VLM reward-aided RL (FuRL), based on reward alignment and relay RL. Specifically, we enhance the performance of SAC/DrQ baseline agents on sparse reward tasks by fine-tuning VLM representations and using relay RL to avoid local minima. Extensive experiments on the Meta-world benchmark tasks demonstrate the efficacy of the proposed method. Code is available at: https://github.com/fuyw/FuRL.

Traffic signal control (TSC) has seen substantial advancements through the application of reinforcement learning (RL) algorithms, which have shown remarkable potential in enhancing traffic flow efficiency. These RL-based approaches often surpass traditional rule-based methods, particularly in dynamic traffic environments. However, current RL solutions for TSC predominantly rely on model-free methods, necessitating extensive environmental interactions during training. This requirement can be prohibitively expensive or unfeasible in real-world implementations. Furthermore, existing methods have frequently neglected the issue of fairness in multi-intersection control, resulting in unbalanced congestion across different intersections. To address these challenges, we present FM2Light, a fairness-aware model-based multi-agent RL framework for TSC. Our approach leverages an ensemble of global world models for generating synthetic samples to enhance sample efficiency, thereby mitigating the data-intensive nature of the training process. Additionally, FM2Light incorporates a refined reward structure to promote fairness and improve coordination across multiple intersections. Extensive evaluations conducted in diverse real-world scenarios demonstrate that FM2Light achieves performance comparable to or exceeding that of model-free RL (MFRL) methods, while significantly reducing sample requirements and ensuring more equitable control among multiple agents.

Reward specification is one of the most tricky problems in Reinforcement Learning, which usually requires tedious hand engineering in practice. One promising approach to tackle this challenge is to adopt existing expert video demonstrations for policy learning. Some recent work investigates how to learn robot policies from only a single/few expert video demonstrations. For example, reward labeling via Optimal Transport (OT) has been shown to be an effective strategy to generate a proxy reward by measuring the alignment between the robot trajectory and the expert demonstrations. However, previous work mostly overlooks that the OT reward is invariant to temporal order information, which could bring extra noise to the reward signal. To address this issue, in this paper, we introduce the Temporal Optimal Transport (TemporalOT) reward to incorporate temporal order information for learning a more accurate OT-based proxy reward. Extensive experiments on the Meta-world benchmark tasks validate the efficacy of the proposed method. Our code is available at: https://github.com/fuyw/TemporalOT.

The penetration of electric vehicles (EV) has been increasing rapidly in recent years. Electric vehicle charging load poses a huge demand on the power grids. The forecasting for electric vehicle charging load, especially for the charging load of EV charging stations, is of significant importance for the safe operation of power grids. However, most of the existing forecasting methods fail to capture the long-term dependencies efficiently and assume the availability of a large amount of training data. Hence, they cannot address newly built charging stations with scarce historical charging load data. Meanwhile, most of the methods focus on point forecasting, which lacks risk consideration. In this work, we aim to leverage the benefits of Transformer-based models for EV charging forecasting. Specifically, we propos Probformer, a Transformer-based forecasting model for charging load forecasting. To enable Probformer to adapt fast to unseen environments, we further extend it to MetaProbformer, a meta-learning-based forecasting framework. Extensive experiments have been done on real-world datasets for both point forecasting and probabilistic forecasting. Experimental results show that our methods can consistently outperform baseline methods by a large margin.

Time series data appears in many real-world fields such as energy, transportation, communication systems. Accurate modelling and forecasting of time series data can be of significant importance to improve the efficiency of these systems. Extensive research efforts have been taken for time series problems. Different types of approaches, including both statistical-based methods and machine learning-based methods, have been investigated. Among these methods, ensemble learning has shown to be effective and robust. However, it is still an open question that how we should determine weights for base models in the ensemble. Sub-optimal weights may prevent the final model from reaching its full potential. To deal with this challenge, we propose a reinforcement learning (RL) based model combination (RLMC) framework for determining model weights in an ensemble for time series forecasting tasks. By formulating model selection as a sequential decision-making problem, RLMC learns a deterministic policy to output dynamic model weights for non-stationary time series data. RLMC further leverages deep learning to learn hidden features from raw time series data to adapt fast to the changing data distribution. Extensive experiments on multiple real-world datasets have been implemented to showcase the effectiveness of the proposed method.

Despite recent advances in the field of Offline Reinforcement Learning (RL), less attention has been paid to understanding the behaviors of learned RL agents. As a result, there remain some gaps in our understandings, i.e., why is one offline RL agent more performant than another? In this work, we first introduce a set of experiments to evaluate offline RL agents, focusing on three fundamental aspects: representations, value functions and policies. Counterintuitively, we show that a more performant offline RL agent can learn relatively low-quality representations and inaccurate value functions. Furthermore, we showcase that the proposed experiment setups can be effectively used to diagnose the bottleneck of offline RL agents. Inspired by the evaluation results, a novel offline RL algorithm is proposed by a simple modification of IQL and achieves SOTA performance. Finally, we investigate when a learned dynamics model is helpful to model-free offline RL agents, and introduce an uncertainty-based sample selection method to mitigate the problem of model noises. Code is available at: https://github.com/fuyw/RIQL.

Catheter/guidewire manipulation in endovascu-lar intervention procedures are associated with risks of injury on vessel wall and embolization. Determination of catheter/guidewire-vessel interaction contact forces can improve the navigation process safety and efficiency which prevent injuries in both manual and robotic vascular interventions. This study proposes a sensor-less sensing solution to estimate multiple contact point forces at the side of catheter/guidewire exerted on the vasculature. This goal is achieved by using image feedback of catheter-vessel interaction and numerical finite element modeling (FEM). Real-time image processing algorithms are implemented to track interaction contact points on catheter/guidewire. Image-based deflection measurement and contact points tracking data are given to a nonlinear finite element beam model to estimate the forces. The variable equivalent bending modulus of the guidewire is found through a series of three-point-bending tests. To directly measure contact point forces, an experimental platform is prepared which simulates catheter/guidewire-vessel interaction with two, three and four contact points. The effectiveness of the proposed approach is tested in six scenarios in which force estimation accuracy of more than 87.9% is achieved. The proposed approach can be applied to various types of under-actuated catheter/guidewire in endovascular intervention procedures. This study proves that multiple catheter/guidewire side contact forces can be estimated by using the deflected shape and equivalent bending modulus property without embedding any force sensor.