Task decomposition has shown promise in complex cooperative multi-agent reinforcement learning (MARL) tasks, which enables efficient hierarchical learning for long-horizon tasks in dynamic and uncertain environments. However, learning dynamic task decomposition from scratch generally requires a large number of training samples, especially exploring the large joint action space under partial observability. In this paper, we present the Conditional Diffusion Model for Dynamic Task Decomposition (CD3T), a novel two-level hierarchical MARL framework designed to automatically infer subtask and coordination patterns. The high-level policy learns subtask representation to generate a subtask selection strategy based on subtask effects. To capture the effects of subtasks on the environment, CD3T predicts the next observation and reward using a conditional diffusion model. At the low level, agents collaboratively learn and share specialized skills within their assigned subtasks. Moreover, the learned subtask representation is also used as additional semantic information in a multi-head attention mixing network to enhance value decomposition and provide an efficient reasoning bridge between individual and joint value functions. Experimental results on various benchmarks demonstrate that CD3T achieves better performance than existing baselines.

In Internet of Things (IoT)-enabled marine sensor networks, underwater object detection faces challenges due to resource constraints, such as small object identification and scale variations. These challenges result in sample imbalance and ambiguities in label assignments. While frameworks like the YOLO series are efficient, their underwater performance is often inadequate due to these issues. This paper introduces a training enhancement strategy tailored for underwater object detection (UWDET) in IoT settings, aimed at reducing inference resource consumption while maintaining high detection accuracy. Importantly, our approach preserves existing network architectures and does not extend inference time. The methodology comprises three main elements: Gaussian Overlap Loss (GOL), which interprets bounding boxes through two-dimensional Gaussian distributions, thereby enhancing localization and addressing scale imbalance for small objects in resource-limited environments. Dynamic Task Joint Assignment (DTJA) modifies positive sample assignments based on classification confidence and regression quality, thereby minimizing false positive rates during training. Normative Focal Loss (NFL) employs a normalized joint assignment metric as continuous labels to effectively address sample imbalance. Experimental evaluations on underwater detection benchmarks reveal that our approach markedly enhances precision and recall, stabilizes gradient signals, and improves training efficiency. We also report training-side GPU memory/time/energy and edge-side memory and latency, confirming unchanged inference cost and reduced training resource usage. Our training enhancement strategy applies meticulously designed lightweight generic object detection models to the underwater domain. Without requiring complex modifications to the network architecture, it enables rapid training and facilitates the deployment and inference of high-precision models within resource-constrained IoT underwater devices.

Multiagent reinforcement learning (MARL) has been widely investigated, ranging from theoretical analysis to real-life applications. However, the utilization of existing non-transparent neural network architectures has resulted in opaque decision-making processes, making it difficult for humans to understand and trust the models being used. Fundamentally, all data is a topological structure, which provides reliable transparency for MARL tasks due to its powerful relational expression capability, scalability, and explicit structural relationships. In this article, we propose a novel approach of graph cooperation modeling (GCM), explicitly capturing and comprehending the complex dynamics of collaborative relationships among agents with the graph structure. GCM learns a metric function to discern beneficial interactions among agents, integrating it into the agent aggregation strategy of a graph neural network (GNN) capable of modeling arbitrary-order interactions. Furthermore, GCM utilizes identity semantics together with global state and individual value functions to estimate the credit of each agent, enhancing each agent's distinct focus on task-related regions. Extensive experiments on a range of challenging MARL benchmarks demonstrate that GCM not only delivers up to 28.75% relative performance gains on super-hard maps but also offers clear interpretability that provides insights into the underlying cooperative patterns.

The ability to model interactions among agents is crucial for effective coordination and understanding their cooperation mechanisms in multi-agent reinforcement learning (MARL). However, previous efforts to model high-order interactions have been primarily hindered by the combinatorial explosion or the opaque nature of their black-box network structures. In this paper, we propose a novel value decomposition framework, called Continued Fraction Q-Learning (QCoFr), which can flexibly capture arbitrary-order agent interactions with only linear complexity $\mathcal{O}\left({n}\right)$ in the number of agents, thus avoiding the combinatorial explosion when modeling rich cooperation. Furthermore, we introduce the variational information bottleneck to extract latent information for estimating credits. This latent information helps agents filter out noisy interactions, thereby significantly enhancing both cooperation and interpretability. Extensive experiments demonstrate that QCoFr not only consistently achieves better performance but also provides interpretability that aligns with our theoretical analysis.

Efficient exploration in multi-agent reinforcement learning (MARL) is a challenging problem when receiving only a team reward, especially in environments with sparse rewards. A powerful method to mitigate this issue involves crafting dense individual rewards to guide the agents toward efficient exploration. However, individual rewards generally rely on manually engineered shaping-reward functions that lack high-order intelligence, thus it behaves ineffectively than humans regarding learning and generalization in complex problems. To tackle these issues, we combine the above two paradigms and propose a novel framework, LIGHT (Learning Individual Intrinsic reward via Incorporating Generalized Human experTise), which can integrate human knowledge into MARL algorithms in an end-to-end manner. LIGHT guides each agent to avoid unnecessary exploration by considering both individual action distribution and human expertise preference distribution. Then, LIGHT designs individual intrinsic rewards for each agent based on actionable representational transformation relevant to Q-learning so that the agents align their action preferences with the human expertise while maximizing the joint action value. Experimental results demonstrate the superiority of our method over representative baselines regarding performance and better knowledge reusability across different sparse-reward tasks on challenging scenarios.

While achieving tremendous success in various fields, existing multi-agent reinforcement learning (MARL) with a black-box neural network makes decisions in an opaque manner that hinders humans from understanding the learned knowledge and how input observations influence decisions. In contrast, existing interpretable approaches usually suffer from weak expressivity and low performance. To bridge this gap, we propose MIXing Recurrent soft decision Trees (MIXRTs), a novel interpretable architecture that can represent explicit decision processes via the root-to-leaf path and reflect each agent's contribution to the team. Specifically, we construct a novel soft decision tree using a recurrent structure and demonstrate which features influence the decision-making process. Then, based on the value decomposition framework, we linearly assign credit to each agent by explicitly mixing individual action values to estimate the joint action value using only local observations, providing new insights into interpreting the cooperation mechanism. Theoretical analysis confirms that MIXRTs guarantee additivity and monotonicity in the factorization of joint action values. Evaluations on complex tasks like Spread and StarCraft II demonstrate that MIXRTs compete with existing methods while providing clear explanations, paving the way for interpretable and high-performing MARL systems.

The CNN-Transformer structure is widely applied to underwater image enhancement (UIE) tasks. However, previous studies have typically used structures similar to those in other image restoration scenarios, without specifically designing a unified structure that fully integrates the characteristics of convolution and Transformers for real-world underwater scenarios. Moreover, the inconsistency of color channel and spatial region attenuation of underwater images has not been given sufficient attention. To this end, this paper proposes a new UIE network, UMCTN, based on multi-kernel convolution Transformer. A multi-kernel convolution residual self-attention block (MCRA) was constructed. By designing a multi-kernel convolutional residual structure, it addresses the issue of critical feature information loss when convolution is applied to small-sized image patches in Transformers, which are widely adopted to reduce computational costs. It elegantly combines the characteristics of convolution and Transformers, endowing the network with a strong capability to capture both local and global dependencies. In addition, a feature fusion compensation module (FFCM) is proposed to supplement richer global perceptual features for MCRA, and It can effectively remove and restore spatial and color channels with more severe attenuation. Tests on the UIEB, UFO-120, and LSUI datasets show that UMCTN achieves better quantitative evaluation and visual performance compared to state-of-the-art (SOTA) schemes, with a maximum peak signal-to-noise ratio (PSNR) improvement of 1.93 dB. Detailed ablation studies validate the effectiveness of each component.

For on-policy reinforcement learning (RL), discretizing action space for continuous control can easily express multiple modes and is straightforward to optimize. However, without considering the inherent ordering between the discrete atomic actions, the explosion in the number of discrete actions can possess undesired properties and induce a higher variance for the policy gradient (PG) estimator. In this article, we introduce a straightforward architecture that addresses this issue by constraining the discrete policy to be unimodal using Poisson probability distributions. This unimodal architecture can better leverage the continuity in the underlying continuous action space using explicit unimodal probability distributions. We conduct extensive experiments to show that the discrete policy with the unimodal probability distribution provides significantly faster convergence and higher performance for on-policy RL algorithms in challenging control tasks, especially in highly complex tasks such as Humanoid. We provide theoretical analysis on the variance of the PG estimator, which suggests that our attentively designed unimodal discrete policy can retain a lower variance and yield a stable learning process.

Evolutionary reinforcement learning (ERL) algorithms recently raise attention in tackling complex reinforcement learning (RL) problems due to high parallelism, while they are prone to insufficient exploration or model collapse without carefully tuning hyperparameters (aka meta-parameters). In the paper, we propose a general meta ERL framework via bilevel optimization (BiERL) to jointly update hyperparameters in parallel to training the ERL model within a single agent, which relieves the need for prior domain knowledge or costly optimization procedure before model deployment. We design an elegant meta-level architecture that embeds the inner-level’s evolving experience into an informative population representation and introduce a simple and feasible evaluation of the meta-level fitness function to facilitate learning efficiency. We perform extensive experiments in MuJoCo and Box2D tasks to verify that as a general framework, BiERL outperforms various baselines and consistently improves the learning performance for a diversity of ERL algorithms.

Deep reinforcement learning (DRL) has achieved impressive success in traffic signal control systems (TSCS). However, since a key component of many DRL models is the complex deep neural networks (DNNs), it hinders humans or experts from understanding and explaining the learned policy. Recently, many works have focused on developing interpretable techniques to compress or distill complex DNNs into smaller, faster, or more understandable models. The decision trees (DTs) are viewed as the de facto technique for interpretable and transparent machine learning, and can provide an easy-understanding decision path from the root to the leaf node. In this work, we utilize modified DTs to extract models with simpler hierarchical structures from premium policy achieved by DRL methods. First, we use a DRL algorithm to learn a premium policy for traffic signal control. Then, we collect a dataset with the learned premium policy by interacting with the environment. Finally, we extract the DTs with the collected dataset of state-actions pairs. We evaluate our method on a Simulation of Urban Mobility simulator in a simulation way. Simulation results show that the extracted DTs can generate human-understandable decision processes and provide explicit knowledge from the DNNs reference of the DRL.

Determining the shortest-path distance between vertices in the weighted graph is an important problem for a broad range of fields, such as context-aware search and route selection. While many efficient methods for querying shortest-path distance have been proposed, they are poorly suited for parallel architectures, such as multi-core CPUs or computer clusters, due to the strong task dependencies. In this paper, we propose ParaPLL, a new parallelism-friendly framework for fast shortest-path distance query on large-scale weighted graphs. ParaPLL exploits intra-node and inter-node parallelism by using shared memory and message passing paradigms respectively. We also design task assignment and synchronization policies, which allow ParaPLL to reach remarkable speedups compared to state-of-the-art solutions. Moreover, we also prove the correctness of ParaPLL. To the best of our knowledge, ParaPLL is the first parallel framework that utilizing pruned landmark labeling to accelerate shortest-path distance queries on large-scale weighted graphs. Our evaluation results show that ParaPLL is 9.46 times faster than the corresponding serial version on a weighted 0.3M-vertex graph using a 12-core computer. ParaPLL on a 6-node computer cluster can also achieve a speedup of up to 5.6 over the single-node implementation.