Traffic Signal Control(TSC), a pivotal and challenging research area in the transportation domain, aims to alleviate congestion at urban intersections by optimizing vehicular flows from different inflow directions. While large efforts have been focused on using Reinforcement Learning(RL) based methods to tackle the TSC problem, it possesses constraints such as unpredictable training duration and risks of online exploration, limiting its real-world deployment. Recently, offline RL has emerged as a new solution by transitioning from learning through online interactions to deriving policies from pre-collected datasets, which guarantees a safer and more efficient learning process. However, existing offline methods overlook the crucial temporal and spatial intricacy among data from different traffic signals at different timesteps, which leads to suboptimal performance. To this end, in this paper, we present an innovative formulation of the offline TSC problem by introducing a spatio-temporal graph to model the historical Markov Decision Process sequences across all traffic signals within the road network. Along this line, we propose STLight, a novel spatio-temporal sequence modeling approach to predict optimal actions for the signals from historical data, accounting for the inherent inter-dependencies among them. Specifically, we incorporate a spatio-temporal encoder to represent states, actions, and returns by capturing dynamic and spatially dependent information. The ordered space-time-aware representations are further fed to the Action Decoder to predict signal phase actions in an auto-regressive manner, accounting for the hidden dependencies between the actions and the reward and state tokens. Furthermore, to adaptively handle tasks with different levels of congestion scenarios, we incorporate space-aware return-based contrastive learning to automatically differentiate data samples with disparate traffic flow patterns. Finally, extensive experiments conducted on two public real-world traffic datasets clearly demonstrate the superior performance of the proposed model over both the state-of-the-art online and offline traffic signal control baselines.

The recent advancements in Traffic Signal Control (TSC) have highlighted the potential of Reinforcement Learning (RL) as a promising solution to alleviate traffic congestion. Current research in this area primarily concentrates on either online or offline learning strategies, aiming to create optimized policies for specific cities. Nevertheless, the transferability of these policies to new cities is impeded by constraints such as the limited availability of high-quality data and the expensive and risky exploration process. To this end, in this paper, we present an innovative cross-city Traffic Signal Control (TSC) paradigm called CrossLight. Our approach involves meta training using offline data from source cities and adaptively fine-tuning in the target city. This novel methodology aims to address the challenges of transferring TSC policies across different cities effectively. In our proposed approach, we start by acquiring meta-decision pattern knowledge through trajectory dynamics reconstruction via pre-training in source cities. To address disparities in road network topologies between cities, we dynamically construct city topological structures based on the extracted meta-knowledge during the offline meta-training phase. These structures are then used to distill pattern-structure aware representations of decision trajectories from the source cities. To identify effective initial parameters for the learnable components, we employ the Model-Agnostic Meta-Learning (MAML) framework, a popular meta-learning approach. During adaptive fine-tuning in the target city, we introduce a replay buffer that is iteratively updated using online interactions with a rank and filter mechanism. This mechanism, along with a carefully designed exploration strategy, ensures a balance between exploitation and exploration, thereby fostering both the diversity and quality of the trajectories for fine-tuning. Finally, extensive experiments across four cities validate that CrossLight achieves comparable performance in new cities with minimal fine-tuning iterations, surpassing both existing online and offline methods. This success underscores that our CrossLight framework emerges as a groundbreaking and potent paradigm, offering a feasible and effective solution to the intelligent transportation community.

Recent years have witnessed the rapid development of the Cooperative Vehicle Infrastructure System (CVIS), where road infrastructures such as traffic lights (TL) and autonomous vehicles (AVs) can share information among each other and work collaboratively to provide safer and more comfortable transportation experience to human beings. While many efforts have been made to develop efficient and sustainable CVIS solutions, existing approaches on urban intersections heavily rely on domain knowledge and physical assumptions, preventing them from being practically applied. To this end, this paper proposes NavTL, a learning-based framework to jointly control traffic signal plans and autonomous vehicle rerouting in mixed traffic scenarios where human-driven vehicles and AVs co-exist. The objective is to improve travel efficiency and reduce total travel time by minimizing congestion at the intersections while guiding AVs to avoid the temporally congested roads. Specifically, we design a graph-enhanced multi-agent decentralized bi-directional hierarchical reinforcement learning framework by regarding TLs as manager agents and AVs as worker agents. At lower temporal resolution timesteps, each manager sets a goal for the workers within its controlled region. Simultaneously, managers learn to take the signal actions based on the observation from the environment as well as an intention information extracted from its workers. At higher temporal resolution timesteps, each worker makes rerouting decisions along its way to the destination based on its observation from the environment, an intention-enhanced manager state representation, and a goal from its present manager. Finally, extensive experiments on one synthetic and two real-world network-level datasets demonstrate the effectiveness of our proposed framework in terms of improving travel efficiency.

Over the years we have seen recommender systems shifting focus from optimizing short-term engagement toward improving long-term user experience on the platforms. While defining good long-term user experience is still an active research area, we focus on one specific aspect of improved long-term user experience here, which is user revisiting the platform. These long term outcomes however are much harder to optimize due to the sparsity in observing these events and low signal-to-noise ratio (weak connection) between these long-term outcomes and a single recommendation. To address these challenges, we propose to establish the association between these long-term outcomes and a set of more immediate term user behavior signals that can serve as surrogates for optimization.