Federated Deep Reinforcement Learning (FDRL) aims to enable distributed collaborative training of multiple DRL models while preserving privacy. Existing FDRL methods function in static client environments, but real-world scenarios often involve dynamic state transitions, such as noise, which render static model topologies inadequate and result in biased policy loss. This degrades client performance and leads to suboptimal global policies. To address this challenge, we develop a generic solution, referred to as the self-regulating training framework, which can be seamlessly integrated into existing FDRL approaches to address dynamic state transitions. Specifically, we propose a Sparse Training (ST) method that dynamically sparsifies and adjusts the topology of each model during training to maximize model performance and reduce model complexity. Additionally, we introduce an auxiliary model to adaptively regulate the policy loss of client models, mitigating loss bias and facilitating updates that yield improved returns. Experimental results demonstrate that our method enhances six state-of-the-art (SOTA) FDRL approaches across nine tasks in terms of return.

Deep Reinforcement Learning (DRL) methods have shown remarkable success in many applications, yet their high energy consumption limits their practicability. Recent studies incorporated energy-efficient Spiking Neural Networks (SNNs) to build Spiking DRL methods and lower energy consumption by setting a shorter simulation duration for SNNs to compute fewer gradients. However, these existing Spiking DRL methods fail to sample sufficient high-quality samples within a fixed-size replay buffer and perform poorly when the simulation duration is small, introducing the challenging tradeoff between energy consumption and model performance. Motivated by such observations, we develop a generic resilient experience replay method that can be seamlessly integrated into existing spiking DRL methods to effectively address the above tradeoff. Specifically, we allow the replay buffer to dynamically expand as the number of training samples increases, thereby accommodating more potentially valuable candidate samples for policy training. Meanwhile, we introduce an adaptive approach to manage the buffer size by determining when to shrink the replay buffer and removing redundant samples automatically. This strategy prevents the buffer from expanding unnecessarily, thereby mitigating the potential negative impact on model performance. Extensive experimental results demonstrate that our approach significantly enhances the performance of five state-of-the-art (SOTA) spiking DRL methods across various simulation durations in sixteen tasks, in terms of return, without compromising their energy efficiency.

In the field of Deep reinforcement learning (DRL), enhancing exploration capabilities and improving the accuracy of Q-value estimation remain two major challenges. Recently, double-actor DRL methods have emerged as a promising class of DRL approaches, achieving substantial advancements in both exploration and Q-value estimation. However, existing double-actor DRL methods feature actors that operate independently in exploring the environment, lacking mutual learning and collaboration, which leads to suboptimal policies. To address this challenge, this work proposes a generic solution that can be seamlessly integrated into existing double-actor DRL methods by promoting mutual learning among the actors to develop improved policies. Specifically, we calculate the difference in actions output by the actors and minimize this difference as a loss during training to facilitate mutual imitation among the actors. Simultaneously, we also minimize the differences in Q-values output by the various critics as part of the loss, thereby avoiding significant discrepancies in value estimation for the imitated actions. We present two specific implementations of our method and extend these implementations beyond double-actor DRL methods to other DRL approaches to encourage broader adoption. Experimental results demonstrate that our method significantly improves twenty state-of-the-art (SOTA) DRL methods, including SOTA double-actor DRL methods, across eleven tasks, as measured by return and other metrics.

Deep learning has revolutionized autonomous driving; nevertheless, its inherent opacity hinders explainability, an essential requirement for public trust and regulatory approval. Existing explainable autonomous driving research typically employs a multi-task framework, simultaneously generating driving actions and their corresponding explanations (collectively called categories). Most methods use a two-stage approach: extracting category-related features and modeling category correlations separately. This separation overlooks the potential synergy between these two processes. Moreover, existing approaches often rely on simple linear combinations of task-specific losses, which may fail to optimally balance action and explanation objectives. To address these limitations, we propose Dual-Attention with Multi-Objective optimization (DAMO). DAMO introduces a dual-attention mechanism that alternates between cross-attention for category representation learning and self-attention for category correlation modeling, fostering mutual enhancement. Additionally, we devise a multi-objective optimization algorithm that dynamically balances tasks and achieves Pareto optimality with theoretical guarantees. Extensive evaluations on two benchmarks show that DAMO surpasses state-of-the-art baselines and a large vision-language model, delivering up to 13.9% performance improvement and enhanced generalization across diverse driving scenarios.

Stereo matching achieves significant progress with iterative algorithms like RAFT-Stereo and IGEV-Stereo. However, these methods struggle in ill-posed regions with occlusions, textureless, or repetitive patterns, due to a lack of global context and geometric information for effective iterative refinement. To enable the existing iterative approaches to incorporate global context, we propose the Global Regulation and Excitation via Attention Tuning (GREAT) framework which encompasses three attention modules. Specifically, Spatial Attention (SA) captures the global context within the spatial dimension, Matching Attention (MA) extracts global context along epipolar lines, and Volume Attention (VA) works in conjunction with SA and MA to construct a more robust cost-volume excited by global context and geometric details. To verify the universality and effectiveness of this framework, we integrate it into several representative iterative stereo-matching methods and validate it through extensive experiments, collectively denoted as GREAT-Stereo. This framework demonstrates superior performance in challenging ill-posed regions. Applied to IGEV-Stereo, among all published methods, our GREAT-IGEV ranks first on the Scene Flow test set, KITTI 2015, and ETH3D leaderboards, and achieves second on the Middlebury benchmark. Code is available at https://github.com/JarvisLee0423/GREAT-Stereo.

Collaborative perception, fusing information from multiple agents, can extend perception range so as to improve perception performance. However, temporal asynchrony in real-world environments, caused by communication delays, clock misalignment, or sampling configuration differences, can lead to information mismatches. If this is not well handled, then the collaborative performance is patchy, and what's worse safety accidents may occur. To tackle this challenge, we propose CoDynTrust, an uncertainty-encoded asynchronous fusion perception framework that is robust to the information mismatches caused by temporal asynchrony. CoDynTrust generates dynamic feature trust modulus (DFTM) for each region of interest by modeling aleatoric and epistemic uncertainty as well as selectively suppressing or retaining single-vehicle features, thereby mitigating information mismatches. We then design a multi-scale fusion module to handle multi-scale feature maps processed by DFTM. Compared to existing works that also consider asynchronous collaborative perception, CoDynTrust combats various low-quality information in temporally asynchronous scenarios and allows uncertainty to be propagated to downstream tasks such as planning and control. Experimental results demonstrate that CoDynTrust significantly reduces performance degradation caused by temporal asynchrony across multiple datasets, achieving state-of-the-art detection performance even with temporal asynchrony. The code is available at https://github.com/CrazyShout/CoDynTrust.

Lifelong deep reinforcement learning (DRL) approaches are commonly employed to adapt continuously to new tasks without forgetting previously acquired knowledge. While current lifelong DRL methods have shown promising advancements in retaining acquired knowledge, they suffer from significant adaptation efforts (i.e., longer training duration) and suboptimal policy when transferring to a new task that significantly deviates from previously learned tasks, a phenomenon known as the few-shot generalization challenge. In this work, we propose a generic approach that equips existing lifelong DRL methods with the capability of few-shot generalization. First, we employ selective experience reuse by leveraging the experience of encountered states, improving adaptation training for new tasks. Then, a relaxed softmax function is applied to the target Q values to improve the accuracy of evaluated Q values, leading to more optimal policies. Finally, we measure and reduce the discrepancy in data distribution between the policy and off-policy samples, resulting in improved adaptation efficiency. Extensive experiments have been conducted on three typical benchmarks to compare our approach with six representative lifelong DRL methods and two state-of-the-art (SOTA) few-shot DRL methods regarding their training speed, episode return, and average return of all episodes. Experimental results substantiate that our method improves the return of six lifelong DRL methods by at least 25%.

Deep reinforcement learning (DRL) has been widely adopted in various applications, yet it faces practical limitations due to high storage and computational demands. Dynamic sparse training (DST) has recently emerged as a prominent approach to reduce these demands during training and inference phases, but existing DST methods achieve high sparsity levels by sacrificing policy performance as they rely on the absolute magnitude of connections for pruning and randomly generating connections. Addressing this, our study presents a generic method that can be seamlessly integrated into existing DST methods in DRL to enhance their policy performance while preserving their sparsity levels. Specifically, we develop a novel method for calculating the importance of connections within the model. Subsequently, we dynamically adjust the sparse network topology by dropping existing connections and introducing new connections based on their respective importance values. Through validation on eight widely used simulation tasks, our method improves two state-of-the-art (SOTA) DST approaches by up to 70% in episode return and average return across all episodes under various sparsity levels.

Deep reinforcement learning (DRL) has been widely applied to various applications, but improving the exploration and the accuracy of Q-value estimation remain key challenges. Recently, the double-actor architecture has emerged as a promising DRL framework that can enhance both exploration and Q-value estimation. Existing double-actor DRL methods sample from the replay buffer to update the two actors; however, the samples used to update each actor are generated by its previous versions and the other actor, resulting in a different data distribution compared with the current actor being updated, which can negatively impact the actor's update and lead to suboptimal policies. To this end, this work proposes a generic solution that can be seamlessly integrated into existing double-actor DRL methods to mitigate the adverse effects of data distribution differences on actor updates, thereby learning better policies. Specifically, we decompose the updates of double-actor DRL methods into two stages, each of which uses the same sampling approach to train a pair of actor-critic. This sampling approach classifies the samples in the replay buffer into distinct categories using a clustering technique, such as K-means, and subsequently employs the Jensen-Shannon (JS) divergence to evaluate the distributional differences between each sample category and the actor currently being updated. Samples are then prioritized from the categories with smaller distribution differences to the current actor to update it. In this way, we can effectively mitigate the distribution difference between the samples and the current actor being updated. Experiments demonstrate that our method enhances the performance of five state-of-the-art (SOTA) double-actor DRL methods and outperforms eight SOTA single-actor DRL methods across eight tasks.

Deep reinforcement learning (DRL) methods, which train a policy to obtain the sequence of actions required to complete a task, have achieved remarkable success across diverse applications. It is a long-standing open issue in the DRL community to make the trained policy gradually approach the theoretically globally optimal policy, and existing research has also explored several challenges, such as exploration-exploitation, to improve the quality of the obtained policy. However, most DRL methods rely solely on the current state for decision-making, leading to short-sightedness and suboptimal learning. To overcome this, we propose a neighboring state-aware policy that enhances existing DRL methods by incorporating a neighboring state sequence in the decision-making process. Specifically, our approach saves multiple past and future states and concatenates them as the neighboring state sequence, along with the current state, and inputs them to the actor to generate an action during the training process. This global perspective, provided by neighboring states, is similar to human decision-making and helps the agent better understand state evolution, leading to improved policy learning. We present two specific implementations of our approach and demonstrate through extensive experiments that it effectively enhances ten representative DRL methods across nine tasks, based on three metrics, including return.

Lifelong deep reinforcement learning (DRL) methods enable continuous adaptation to new tasks and retention of old knowledge. However, these methods often necessitate large model sizes, leading to substantial computational and storage resource requirements during training and inference. Unfortunately, existing research has not yet provided a lightweight solution to address this issue. This work aims to develop a generic method that can be seamlessly integrated into existing lifelong DRL methods to facilitate their achievement of lightweight models while also yielding higher returns. While sparse training (ST) methods have been extensively used in the DRL community to achieve lightweight models, they exacerbate the issue of catastrophic forgetting and compromise generalization when applied in lifelong DRL. To improve generalization, we develop a gradient optimization method that leverages sharpness-aware minimization (SAM) to smooth the gradient surface of the model without introducing excessive computational complexity. In addition, to alleviate catastrophic forgetting and promote model convergence, we introduce a priority-based approach that samples effective past experiences from the replay buffer. Extensive experiments demonstrate that our approach achieves 90% sparsity in five representative lifelong DRL methods while achieving higher episode return and average return (up to 34% improvement) across all episodes compared to the dense models.

Recent studies have extensively explored the causal connections between emotions and their underlying causes in textual data. Most research aims to identify clauses within documents that are causally related. However, these studies have overlooked the fact that such causal relationships are often context-dependent and valid only within specific contextual clauses. To bridge this gap, we present a novel task of determining the presence of a valid causal relationship between a given pair of emotion and cause clauses in different contexts, while also identifying the specific contextual clauses involved. Since this task is novel and lacks an existing dataset for testing, we manually annotate a benchmark dataset to obtain labels for our task and classify the types of context clauses, which can also be beneficial for other applications. By leveraging negative sampling, we create a balanced final dataset that includes documents with and without causal relationships. Building upon this dataset, we propose an end-to-end multi-task framework that incorporates two innovative modules aimed at achieving the objectives of our task. We introduce a context masking module to identify the contextual clauses that contribute to causal relationships and a prediction aggregation module to refine predictions by determining the reliance of emotion and cause clauses on specific contextual clauses. Extensive comparative experiments and ablation studies validate the effectiveness and robustness of our proposed framework. The annotated dataset provides a novel way for exploring complex reasoning in causal analysis.

Autonomous driving systems rely on precise trajectory prediction for safe and efficient motion planning. Despite considerable efforts to enhance prediction accuracy, inherent uncertainties persist due to data noise and incomplete observations. Many strategies entail formalizing prediction outcomes into distributions and utilizing variance to represent uncertainty. However, our experimental investigation reveals that existing trajectory prediction models yield unreliable uncertainty estimates, necessitating additional customized calibration processes. On the other hand, directly applying current calibration techniques to prediction outputs may yield sub-optimal results due to using a universal scaler for all predictions and neglecting informative data cues. In this paper, we propose Customized Calibration Temperature with Regularizer (CCTR), a generic framework that calibrates the output distribution. Specifically, CCTR 1) employs a calibration-based regularizer to align output variance with the discrepancy between prediction and ground truth and 2) generates a tailor-made temperature scaler for each prediction using a post-processing network guided by context and historical information. Extensive evaluation involving multiple prediction and planning methods demonstrates the superiority of CCTR over existing calibration algorithms and uncertainty-aware methods, with significant improvements of 11%-22% in calibration quality and 17%-46% in motion planning.

Simulating realistic behaviors of traffic agents is pivotal for efficiently validating the safety of autonomous driving systems. Existing data-driven simulators primarily use an encoder-decoder architecture to encode the historical trajectories before decoding the future. However, the heterogeneity between encoders and decoders complicates the models, and the manual separation of historical and future trajectories leads to low data utilization. Given these limitations, we propose BehaviorGPT, a homogeneous and fully autoregressive Transformer designed to simulate the sequential behavior of multiple agents. Crucially, our approach discards the traditional separation between "history" and "future" by modeling each time step as the "current" one for motion generation, leading to a simpler, more parameter- and data-efficient agent simulator. We further introduce the Next-Patch Prediction Paradigm (NP3) to mitigate the negative effects of autoregressive modeling, in which models are trained to reason at the patch level of trajectories and capture long-range spatial-temporal interactions. Despite having merely 3M model parameters, BehaviorGPT won first place in the 2024 Waymo Open Sim Agents Challenge with a realism score of 0.7473 and a minADE score of 1.4147, demonstrating its exceptional performance in traffic agent simulation.

Multi-agent reinforcement learning (MARL) has proven effective in training multi-robot confrontation, such as StarCraft and robot soccer games. However, the current joint action policies utilized in MARL have been unsuccessful in recognizing and preventing actions that often lead to failures on our side. This exacerbates the cooperation dilemma, ultimately resulting in our agents acting independently and being defeated individually by their opponents. To tackle this challenge, we propose a novel joint action policy, referred to as the consensus action policy (CAP). Specifically, CAP records the number of times each joint action has caused our side to fail in the past and computes a cooperation tendency, which is integrated with each agent’sQ-value and Nash bargaining solution to determine a joint action. The cooperation tendency promotes team cooperation by selecting joint actions that have a high tendency of cooperation and avoiding actions that may lead to team failure. Moreover, the proposed CAP policy can be extended to partially observable scenarios by combining it with DeepQnetwork or actor-critic–based methods. We conducted extensive experiments to compare the proposed method with seven existing joint action policies, including four commonly used methods and three state-of-the-art methods, in terms of episode rewards, winning rates, and other metrics. Our results demonstrate that this approach holds great promise for multi-robot confrontation scenarios.

In autonomous driving, an accurate understanding of environment, e.g., the vehicle-to-vehicle and vehicle-to-lane interactions, plays a critical role in many driving tasks such as trajectory prediction and motion planning. Environment information comes from high-definition (HD) map and historical trajectories of vehicles. Due to the heterogeneity of the map data and trajectory data, many data-driven models for trajectory prediction and motion planning extract vehicle-to-vehicle and vehicle-to-lane interactions in a separate and sequential manner. However, such a manner may capture biased interpretation of interactions, causing lower prediction and planning accuracy. Moreover, separate extraction leads to a complicated model structure and hence the overall efficiency and scalability are sacrificed. To address the above issues, we propose an environment representation, Temporal Occupancy Flow Graph (TOFG). Specifically, the occupancy flow-based representation unifies the map information and vehicle trajectories into a homogeneous data format and enables a consistent prediction. The temporal dependencies among vehicles can help capture the change of occupancy flow timely to further promote model performance. To demonstrate that TOFG is capable of simplifying the model architecture, we incorporate TOFG with a simple graph attention (GAT) based neural network and propose TOFG-GAT, which can be used for both trajectory prediction and motion planning. Experiment results show that TOFG-GAT achieves better or competitive performance than all the SOTA baselines with less training time.

Recently, many efforts have been devoted to promoting the Emotion-Cause Pair Extraction (ECPE) task, as jointly extracting emotions and their causes is considered more helpful than only identifying the emotions in many applications. Among the existing efforts, end-to-end approaches are getting popular as the main trend, while others like pipeline models have been overlooked due to their potential issues of cascading errors. Nevertheless, the advantages of the pipeline models, such as logically dividing a complicated task into multiple easier subtasks, are underestimated and not well exploited. Moreover, the existing end-to-end approaches fail to capture the implicit co-occurrence or exclusion patterns between multiple pairs of emotions and causes since they are extracted independently. In view of these limitations, we propose a novel two-stage model to address the ECPE task and incorporate reinforcement learning (RL) to tackle the cascading error issue. In particular, our two-stage model first detects emotion clauses and then recognizes cause clauses for each detected emotion clause sequentially. By representing the error of each decision as an explicit reward, our model clearly knows how the error at each stage affects the final performance, hence the model can adjust itself for better performance. Furthermore, the sequential prediction enables our model to use the results achieved in the previous stages as auxiliary information in the subsequent stages. Extensive experiments on the benchmark dataset demonstrate the effectiveness of our proposed two-stage model, and the ablation comparison shows the promising effect of reducing cascading errors by incorporating RL.

Modeling interactions among vehicles is critical in improving the efficiency and safety of autonomous driving since complex interactions are ubiquitous in many traffic scenarios. To model interactions under different traffic scenarios, most existing works consider interaction information implicitly in their specific tasks with hand-crafted features and predefined maneuvers. Extracting interaction representation, which can be commonly used among different downstream tasks, is not explored. In this article, we propose a general and novel graph self-attention network (GSAN) to learn the spatial–temporal interaction representation among vehicles by a framework consisting of pretraining and fine-tuning. Specifically, in the pretraining step, we construct the GSAN module based on a graph self-attention layer and a gated recurrent unit layer, and use trajectory autoregression to learn the interaction information among vehicles. In the fine-tuning step, we propose two different adaptation schemes to utilize the learned interaction information in various downstream tasks and fine-tune the entire model with only a few steps. To illustrate the effectiveness and generality of our spatial–temporal interaction model, we conduct extensive experiments on two typical interaction-related tasks, namely, lane-changing classification and trajectory prediction. The experiment results demonstrate that our approach significantly outperforms the state-of-the-art solutions of these two tasks. We also visualize the impact of surrounding vehicles on the ego vehicle in different interaction scenes. The visualization offers an intuitive explanation on how our model captures the dynamic changing interactions among vehicles and makes good predictions in various interaction-related tasks.

Abstract Exploiting hand-crafted lexicon knowledge to enhance emotional or sentimental features at word-level has become a widely adopted method in emotion-relevant classification studies. However, few attempts have been made to explore the emotion construction in the classification task, which provides insights to how a sentence’s emotion is constructed. The major challenge of exploring emotion construction is that the current studies assume the dataset labels as relatively independent emotions, which overlooks the connections among different emotions. This work aims to understand the coarse-grained emotion construction and their dependency by incorporating fine-grained emotions from domain knowledge. Incorporating domain knowledge and dimensional sentiment lexicons, our previous work proposes a novel method namedEmoChannelto capture the intensity variation of a particular emotion in time series. We utilize the resultant knowledge of 151 available fine-grained emotions to comprise the representation of sentence-level emotion construction. Furthermore, this work explicitly employs a self-attention module to extract the dependency relationship within all emotions and proposeEmoChannel-SANetwork to enhance emotion classification performance. We conducted experiments to demonstrate that the proposed method produces competitive performances against the state-of-the-art baselines on both multi-class datasets and sentiment analysis datasets.

Emotion-cause pair extraction (ECPE) aims at extracting emotions and causes as pairs from documents, where each pair contains an emotion clause and a set of cause clauses.Existing approaches address the task by first extracting emotion and cause clauses via two binary classifiers separately, and then training another binary classifier to pair them up.However, the extracted emotion-cause pairs of different emotion types cannot be distinguished from each other through simple binary classifiers, which limits the applicability of the existing approaches.Moreover, such two-step approaches may suffer from possible cascading errors.In this paper, to address the first problem, we assign emotion type labels to emotion and cause clauses so that emotioncause pairs of different emotion types can be easily distinguished.As for the second problem, we reformulate the ECPE task as a unified sequence labeling task, which can extract multiple emotion-cause pairs in an end-to-end fashion.We propose an approach composed of a convolution neural network for encoding neighboring information and two Bidirectional Long-Short Term Memory networks for two auxiliary tasks.Experiment results demonstrate the feasibility and effectiveness of our approaches.

The causal relationships between emotions and causes in text have recently received a lot of attention.Most of the existing works focus on the extraction of the causally related clauses from documents.However, none of these works has considered the possibility that the causal relationships among the extracted emotion and cause clauses may only be valid under a specific context, without which the extracted clauses may not be causally related.To address such an issue, we propose a new task of determining whether or not an input pair of emotion and cause has a valid causal relationship under different contexts, and construct a corresponding dataset via manual annotation and negative sampling based on an existing benchmark dataset.Furthermore, we propose a prediction aggregation module with low computational overhead to fine-tune the prediction results based on the characteristics of the input clauses.Experiments demonstrate the effectiveness and generality of our aggregation module.

Modeling the interactions among vehicles has been considered essential in improving efficiency and safety in autonomous driving, since the real traffic scenarios, such as merging lanes, intersection, and lane change, are full of complex interactions. In the literature, interaction is considered implicitly in individual tasks, which makes it hard to extract the interactions for other related downstream tasks. In this paper, we propose a novel Graph Self-Attention Network (GSAN) to quickly capture and quantify the influence of interactions among vehicles from historical trajectories, which can be used as a tool to introduce the impact of interactions into different downstream tasks and further analyze the dominating features affecting the interactions among vehicles. We conduct experiments on the trajectory prediction task as one example to illustrate how to use the spatial-temporal interaction vector to improve the performance of interaction related tasks. The experiment results demonstrate that the GSAN module outperforms the state-of-the-art solutions in terms of the trajectory prediction accuracy. Also, we visualize the effects from all surrounding vehicles on the ego vehicle by heat maps using the trained attention values from the GSAN module.