Graph classification is crucial for diverse real-world applications, such as drug discovery and social network analysis. Reinforcement learning (RL) has been introduced to graph tasks for its advantage in sequential decision-making and exploration. However, existing RL-based methods face challenges: improper state/action definitions can trap policies in local optima, while contrastive learning approaches often rely on random sampling, struggling to guarantee effective negative samples. To address these issues, this paper proposes a graph classification framework with stepwise optimization embedded with multi-agent collaboration. Specifically, we first optimize the pooling ratio adjustment agent, adopt more advanced reinforcement learning algorithms, and design richer and more informative action spaces and reward functions to support the agent in conducting more sufficient environmental exploration, thereby approximating the global optimal policy. Additionally, we introduce a sampling agent that adaptively optimizes its sampling strategy during the contrastive learning process, aiming to generate more distinguishable hard negative samples. Extensive experiments on six classic benchmark datasets demonstrate that the proposed method achieves superior performance in graph classification tasks, significantly outperforming existing baseline methods and exhibiting strong competitiveness.

Foundation models have revolutionized language modeling, while whether this success is replicated in scientific computing remains unexplored. We present OmniArch, the first prototype aiming at solving multi-scale and multi-physics scientific computing problems with physical alignment. We addressed all three challenges with one unified architecture. Its pre-training stage contains a Fourier Encoder-decoder fading out the disharmony across separated dimensions and a Transformer backbone integrating quantities through temporal dynamics, and the novel PDE-Aligner performs physics-informed fine-tuning under flexible conditions. As far as we know, we first conduct 1D-2D-3D united pre-training on the PDEBench, and it sets not only new performance benchmarks for 1D, 2D, and 3D PDEs but also demonstrates exceptional adaptability to new physics via in-context and zero-shot learning approaches, which supports realistic engineering applications and foresight physics discovery.

The development and application of large language models (LLM) have demonstrated that foundational models can be utilized to solve a wide array of tasks. However, their performance in multi-agent path finding (MAPF) tasks has been less than satisfactory, with only a few studies exploring this area. MAPF is a complex problem requiring both planning and multi-agent coordination. To improve the performance of LLM in MAPF tasks, we propose a novel framework, LLM-NAR, which leverages neural algorithmic reasoners (NAR) to inform LLM for MAPF. LLM-NAR consists of three key components: an LLM for MAPF, a pre-trained graph neural network-based NAR, and a cross-attention mechanism. This is the first work to propose using a neural algorithmic reasoner to integrate GNNs with the map information for MAPF, thereby guiding LLM to achieve superior performance. LLM-NAR can be easily adapted to various LLM models. Both simulation and real-world experiments demonstrate that our method significantly outperforms existing LLM-based approaches in solving MAPF problems.

Goal assignment is a critical challenge in multi-robot systems. The emergence of large language models (LLMs) has enabled the use of natural language commands for tackling goal assignment problems. However, applying LLMs directly to these tasks presents two limitations: 1) limited accuracy and 2) excessive decision delays due to their autoregressive nature, hindering adaptability to unexpected changes. To address these issues, inspired by dual-process theory, we propose a framework called Collaborative LLMs for dynamic Goal Assignment (CLGA). Specifically, we leverage LLMs for pre-planning tasks and invoke an external solver to generate an initial goal assignment solution, ensuring solution accuracy. During execution, small-scale models enable real-time adjustments to respond to dynamic environmental changes. This approach integrates the strengths of slow, precise pre-planning and fast, adaptive online adjustments, allowing agents to efficiently handle real-world challenges. Additionally, we introduce a benchmark dataset for NLP-based goal assignment to advance research in this domain. Simulation and real-world experiments demonstrate that CLGA significantly enhances task execution efficiency and flexibility in multi-robot systems. The prompt, experimental videos, and datasets associated with this work are available at https://sites.google.com/view/project-clga/.

Multi-Agent Reinforcement Learning (MARL) has shown great potential in solving complex tasks. Despite great success, low training efficiency remains a pervasive and long-standing challenge in MARL. To tackle this issue, it is promising to leverage prior knowledge or environmental properties to inform and improve the MARL. We notice that many multi-agent tasks specify certain goal states where special rewards are granted, guiding agents to achieve the goal. Inspired by the theory of Lyapunov stability, an intuitive optimal policy to the tasks should be able to asymptotically converge to the goal states from any initial, making the goal states stable equilibria. Focusing on this type of tasks, we introduce the concept of Lyapunov Markov game (LMG), a new subclass of the cooperative Markov game, featuring a set of goal states and goal-oriented reward function. We then provide a theoretical bound on scaled value distance as a necessary condition to obtain a stable suboptimal policy in LMG. Motivated by this insight, we further propose the Lyapunov-informed MARL, which leverages a newly-designed Lyapunov-informed reward. Theoretical work is conducted to show that the Lyapunov-informed MARL enjoys a broadened bound, facilitating the training process to find a stable suboptimal policy more easily and then converge to an optimal policy more efficiently. Extensive experiments and real-world multi-robot implementations are conducted to show the superior performance of the proposed approach over advanced baseline models.

Uncrewed aerial vehicle-mounted base stations (UAV-MBSs) provide flexible wireless connectivity, extending communication coverage in underserved areas. Recently, multi-agent reinforcement learning (MARL) has shown great potential for cooperative UAV swarm control to support efficient communication coverage in dynamic and complex environments. However, existing MARL-based methods often suffer from low sample efficiency due to its trial-and-error training characteristics, limiting its ability to control large UAV swarms with continuous state-action space and partial observation. We notice that UAV swarm systems in communication coverage tasks exhibit a spatial symmetry property, e.g., a rotation in the spatial observation of a UAV results in a same rotation in its optimal action. Exploiting this property, we formulate the task as a symmetric decentralized partially observable Markov decision process and introduce symmetry-informed MARL, featuring a novel network called the symmetry-informed graph neural network (SiGNN) to serve as the policy/value networks. SiGNN leverages the inherent symmetry in multi-UAV systems by embedding the symmetry into the network structure, thereby enhancing the training efficiency to handle large swarms with continuous control. Theoretical analysis shows that the SiGNN strictly preserves symmetry properties, which guarantees the effectiveness of the approach. Experiments in simulation were conducted to handle communication coverage using up to 20 UAVs with continuous control. Experimental results demonstrate that SiGNN-based MARL outperforms advanced baselines, verifying its superior sample efficiency, scalability and robustness.

Incorporating symmetry as an inductive bias into multi-agent reinforcement learning (MARL) has led to improvements in generalization, data efficiency, and physical consistency. While prior research has succeeded in using perfect symmetry prior, the realm of partial symmetry in the multi-agent domain remains unexplored. To fill in this gap, we introduce the partially symmetric Markov game, a new subclass of the Markov game. We then theoretically show that the performance error introduced by utilizing symmetry in MARL is bounded, implying that the symmetry prior can still be useful in MARL even in partial symmetry situations. Motivated by this insight, we propose the Partial Symmetry Exploitation (PSE) framework that is able to adaptively incorporate symmetry prior in MARL under different symmetry-breaking conditions. Specifically, by adaptively adjusting the exploitation of symmetry, our framework is able to achieve superior sample efficiency and overall performance of MARL algorithms. Extensive experiments are conducted to demonstrate the superior performance of the proposed framework over baselines. Finally, we implement the proposed framework in real-world multi-robot testbed to show its superiority.

Achieving high sample efficiency is a critical research area in reinforcement learning. This becomes extremely difficult in multi-agent reinforcement learning (MARL), as the capacity of the joint state and action space grows exponentially with the number of agents. The reliance of MARL solely on exploration and trial-and-error, without incorporating prior knowledge, exacerbates the issue of low sample efficiency. Currently, introducing symmetry into MARL is an effective approach to address this issue. Yet the concept of hierarchical symmetry, which maintains symmetry across different levels of a multi-agent system (MAS), has not been explored in existing methods. This paper focuses on multi-agent cooperative tasks and proposes a method incorporating hierarchical symmetry, termed the Hierarchical Equivariant Policy Network (HEPN) which is O(n)-equivariant. Specifically, HEPN utilizes clustering to perform hierarchical information extraction in MAS, and employs graph neural networks to model agent interactions. We conducted extensive experiments across various multi-agent tasks. The results indicate that our method achieves faster convergence speeds and higher convergence rewards compared to baseline algorithms. Additionally, we have deployed our algorithm in a physical multi-robot system, confirming its effectiveness in real-world environments. Supplementary materials are available at https://yongkai-tian.github.io/HEPN/.

Multi-agent reinforcement learning has emerged as a promising approach for the control of multi-robot systems. Nevertheless, the low sample efficiency of MARL poses a significant obstacle to its broader application in robotics. While data augmentation appears to be a straightforward solution for improving sample efficiency, it usually incurs training instability, making the sample efficiency worse. Moreover, manually choosing suitable augmentations for a variety of tasks is a tedious and time-consuming process. To mitigate these challenges, our research theoretically analyzes the implications of data augmentation on MARL algorithms. Guided by these insights, we present AdaptAUG, an adaptive framework designed to selectively identify beneficial data augmentations, thereby achieving superior sample efficiency and overall performance in multi-robot tasks. Extensive experiments in both simulated and real-world multi-robot scenarios validate the effectiveness of our proposed framework.

In multi-agent reinforcement learning (MARL), the Centralized Training with Decentralized Execution (CTDE) framework is pivotal but struggles due to a gap: global state guidance in training versus reliance on local observations in execution, lacking global signals. Inspired by human societal consensus mechanisms, we introduce the Hierarchical Consensus-based Multi-Agent Reinforcement Learning (HC-MARL) framework to address this limitation. HC-MARL employs contrastive learning to foster a global consensus among agents, enabling cooperative behavior without direct communication. This approach enables agents to form a global consensus from local observations, using it as an additional piece of information to guide collaborative actions during execution. To cater to the dynamic requirements of various tasks, consensus is divided into multiple layers, encompassing both short-term and long-term considerations. Short-term observations prompt the creation of an immediate, low-layer consensus, while long-term observations contribute to the formation of a strategic, high-layer consensus. This process is further refined through an adaptive attention mechanism that dynamically adjusts the influence of each consensus layer. This mechanism optimizes the balance between immediate reactions and strategic planning, tailoring it to the specific demands of the task at hand. Extensive experiments and real-world applications in multi-robot systems showcase our framework’s superior performance, marking significant advancements over baselines.

For multiple smart sensors with limited energy supply, the relationship between the energy supply and sensor fault-free region is often unknown, and neither of the interior structure nor exterior circumstance modeling the smart sensors is easy to accurately achieve, so how to guarantee the fault-free smart sensors to harvest the most energy in the fault-free way is a very challenging topic. To address this issue, this paper first formulates the individual smart sensor as a single-input single-output (SISO) model-free system (MFS), with its energy supply and sensing error as the input and output, respectively, then makes use of Lyapunov function to deduce an upper bound of the fault-free region to realize the fault-detection of any smart sensor and disclose the relationship between the energy supply and fault-free region, and finally achieves the optimal energy supply guiding the overall energy harvest of all fault-free smart sensors to converge to the maximum with the convergence rate no larger than$\kappa$,$\kappa\in[0,1]$, while enjoying the real-time fault-detection. An algorithm based on the sound theoretical foundations is further proposed to implement the optimal energy supply. Theoretical analysis, simulations and field experiments jointly verify the performance of our method. To our best knowledge, it is the initial work towards this issue.Note to Practitioners—This paper addresses the interesting issue of how to guarantee multiple smart sensors to harvest the most energy in the free-fault way. Through the insightful disclosure of relationship between the energy supply and sensor fault-free region, and the optimal control of energy supply, this paper facilitates the overall energy harvest of all fault-free smart sensors that work under the environments, where the available energy is limited, to converge to the maximum while enjoying the real-time fault-detection, which we believe could push the development of Internet of Things (IoT) or Cyber-Physical System (CPS) that employs multiple smart sensors to sense the physical world. Simulations and field experimental investigations jointly show that the proposed solution outperforms the existing solutions.

Knowledge tracing models based on deep learning can achieve impressive predictive performance by leveraging attention mechanisms. However, there still exist two challenges in attentive knowledge tracing (AKT): First, the mechanism of classical models of AKT demonstrates relatively low attention when processing exercise sequences with shifting knowledge concepts (KC), making it difficult to capture the comprehensive state of knowledge across sequences. Second, classical models do not consider stochastic behaviors, which negatively affects models of AKT in terms of capturing anomalous knowledge states. This article proposes a model of AKT, called Enhancing Locality for Attentive Knowledge Tracing (ELAKT), that is a variant of the deep KT model. The proposed model leverages the encoder module of the transformer to aggregate knowledge embedding generated by both exercises and responses over all timesteps. In addition, it uses causal convolutions to aggregate and smooth the states of local knowledge. The ELAKT model uses the states of comprehensive KCs to introduce a prediction correction module to forecast the future responses of students to deal with noise caused by stochastic behaviors. The results of experiments demonstrated that the ELAKT model consistently outperforms state-of-the-art baseline KT models.

Multi-agent reinforcement learning (MARL) has achieved promising results in recent years. However, most existing reinforcement learning methods require a large amount of data for model training. In addition, data-efficient reinforcement learning requires the construction of strong inductive biases, which are ignored in the current MARL approaches. Inspired by the symmetry phenomenon in multi-agent systems, this paper proposes a framework for exploiting prior knowledge by integrating data augmentation and a well-designed consistency loss into the existing MARL methods. In addition, the proposed framework is model-agnostic and can be applied to most of the current MARL algorithms. Experimental tests on multiple challenging tasks demonstrate the effectiveness of the proposed framework. Moreover, the proposed framework is applied to a physical multi-robot testbed to show its superiority.

Reinforcement learning for swarms of flying robots is a challenging task that requires a large number of data samples. Moreover, the problem of sim-to-real transfer has long been a challenge in robotics algorithm deployment. To address these issues, we propose Air-M, a platform that facilitates large-scale drone swarm learning in a distributed docker container environment and deployment in a virtual reality setting. Air-M trains the policy network using physics engines and creates replicas of agents in docker containers, which helps amortize the computational cost. In addition, Air-M establishes an intermediate link between the simulation and the real world, allowing real drones to interact with virtual objects via virtual sensors. This enables the policy network to be trained using virtual agents and seamlessly transferred to real drones. Air-Mis highly scalable, accommodating hundreds of agents with dynamic models and virtual sensors. We evaluate the effectiveness of our approach by conducting experiments in three representative virtual scenarios with an increasing number of agents. Our results demonstrate that our method outperforms the state-of- the-art in terms of training efficiency and transferability, making it a promising platform for swarm robotics applications.

In order to reduce the measurement error of low cost sensor in the real-time mobile sensing network, rendezvous calibration mechanism is widely used. To tackle the sparsity of reference data and the lack of calibration opportunities, we propose ST-ICM: a Spatial-Temporal Inference Calibration Model based on Gaussian Process Regression, assisting the calibration task by creating more calibration grids in both spatial and temporal dimensions. By using the GPR, the inferred grids generated by ST-ICM are associated with various confidence levels. Based on this property, we propose to make use of a hyperparameter, i.e., variance threshold, to balance the tradeoff between the quantity and quality of the inferred grids. Specifically, only the grids with variances below the threshold will be employed. We conducted experiments using a real-world dataset collected in Nanjing, China, to evaluate the performance of the proposed ST-ICM. The experimenal results show that our model achieves 24% improvement on error calibration compared to the baseline.

Brain–computer interface (BCI) is a promising intelligent healthcare technology to improve human living quality across the lifespan, which enables assistance of movement and communication, rehabilitation of exercise and nerves, monitoring sleep quality, fatigue and emotion. Most BCI systems are based on motor imagery electroencephalogram (MI-EEG) due to its advantages of sensory organs affection, operation at free will and etc. However, MI-EEG classification, a core problem in BCI systems, suffers from two critical challenges: the EEG signal’s temporal non-stationarity and the nonuniform information distribution over different electrode channels. To address these two challenges, this paper proposes TCACNet, a temporal and channel attention convolutional network for MI-EEG classification. TCACNet leverages a novel attention mechanism module and a well-designed network architecture to process the EEG signals. The former enables the TCACNet to pay more attention to signals of task-related time slices and electrode channels, supporting the latter to make accurate classification decisions. We compare the proposed TCACNet with other state-of-the-art deep learning baselines on two open source EEG datasets. Experimental results show that TCACNet achieves 11.4% and 7.9% classification accuracy improvement on two datasets respectively. Additionally, TCACNet achieves the same accuracy as other baselines with about 50% less training data. In terms of classification accuracy and data efficiency, the superiority of the TCACNet over advanced baselines demonstrates its practical value for BCI systems.

Traffic state estimation (TSE) bifurcates into two main categories, model-driven and data-driven (e.g., machine learning, ML) approaches, while each suffers from either deficient physics or small data. To mitigate these limitations, recent studies introduced hybrid methods, such as physics-informed deep learning (PIDL), which contains both model-driven and data-driven components. This paper contributes an improved paradigm, called physics-informed deep learning with a fundamental diagram learner (PIDL + FDL), which integrates ML terms into the model-driven component to learn a functional form of a fundamental diagram (FD), i.e., a mapping from traffic density to flow or velocity. The proposed PIDL + FDL has the advantages of performing the TSE learning, model parameter identification, and FD estimation simultaneously. This paper focuses on highway TSE with observed data from loop detectors, using traffic density or velocity as traffic variables. We demonstrate the use of PIDL + FDL to solve popular first-order and second-order traffic flow models and reconstruct the FD relation as well as model parameters that are outside the FD term. We then evaluate the PIDL + FDL-based TSE using the Next Generation SIMulation (NGSIM) dataset. The experimental results show the superiority of the PIDL + FDL in terms of improved estimation accuracy and data efficiency over advanced baseline TSE methods, and additionally, the capacity to properly learn the unknown underlying FD relation.

In this paper, a general location selection strategy is proposed based on active learning, which involves iterations of a selector and an estimator. We implement four instances of this general strategy to embody it: KAL (Active Learning based on Kriging), TAL (Active Learning based on Regression Tree), KMAL (Active Learning based on Kriging and MPGR) and TMAL (Active Learning based on Regression Tree and MPGR). The estimator of KAL or TAL can estimate the air quality at remaining locations from air quality samples at monitoring locations leveraging spatial or cross-domain correlation of air quality. The selecting indicators of their selectors are designed to measure the uncertainty of unlabeled samples according to their estimators. KMAL and TMAL are upgrades of the former two respectively, by introducing MPGR (Manifold Preserving Graph Reduction) to also take the representativeness of unlabeled samples into account. The experimental results show that the proposed strategy can achieve low estimation error with few monitoring locations. Particularly, given the same budget (i.e., the number of monitoring locations), the estimation error is reduced from about 20% of baselines to 15% by KAL and to 5% by KMAL, and TAML likewise.

Traffic state estimation (TSE) reconstructs the traffic variables (e.g., density or average velocity) on road segments using partially observed data, which is important for traffic managements. Traditional TSE approaches mainly bifurcate into two categories: model-driven and data-driven, and each of them has shortcomings. To mitigate these limitations, hybrid TSE methods, which combine both model-driven and data-driven, are becoming a promising solution. This paper introduces a hybrid framework, physics-informed deep learning (PIDL), to combine second-order traffic flow models and neural networks to solve the TSE problem. PIDL can encode traffic flow models into deep neural networks to regularize the learning process to achieve improved data efficiency and estimation accuracy. We focus on highway TSE with observed data from loop detectors and probe vehicles, using both density and average velocity as the traffic variables. With numerical examples, we show the use of PIDL to solve a popular second-order traffic flow model, i.e., a Greenshields-based Aw-Rascle-Zhang (ARZ) model, and discover the model parameters. We then evaluate the PIDL-based TSE method using the Next Generation SIMulation (NGSIM) dataset. Experimental results demonstrate the proposed PIDL-based approach to outperform advanced baseline methods in terms of data efficiency and estimation accuracy.

The rapid growth in urban population poses significant challenges to moving city dwellers in a fast and convenient manner. This paper contributes to solving the challenges from the viewpoint of passengers by improving their on-vehicle experience. Specifically, we focus on the problem: Given an urban public transit network and a number of passengers, with some of them controllable and the rest uncontrollable, how can we plan for the controllable passengers to improve their experience in terms of their service preference? We formalize this problem as a multi-agent path planning (MAPP) problem with soft collisions, where multiple controllable passengers are allowed to share on-vehicle service resources with one another under certain constraints. We then propose a customized version of the SC-M* algorithm to efficiently solve the MAPP task for bus transit system in complex urban environments, where we have a large passenger size and multiple types of passengers requesting various types of service resources. We demonstrate the use of SC-M* in a case study of the bus transit system in Porto, Portugal. In the case study, we implement a data-driven on-vehicle experience simulator for the bus transit system, which simulates the passenger behaviors and on-vehicle resource dynamics, and evaluate the SC-M* on it. The experimental results show the advantages of the SC-M* in terms of path cost, collision-free constraint, and the scalability in run time and success rate.

Automated data collection in urban transportation systems produces a large volume of passenger data. However, quite a few of the data are still incomplete, limiting the insight into passenger mobility. The unavailability of destination information in entry-only passenger data is a very common issue. Traditional approaches for estimating passenger destinations rely on heuristics that can recover only some of the missing destinations. To deal with the remaining incomplete data, this paper, for the first time, proposes a second-order inference methodology to leverage semi-supervised self-training to infer the missing destinations. The methodology involves the design of a base learner to predict the missing destinations based on the statistics of a selected similarity-based "training set", and the design of a selection strategy to select new data with high prediction confidence to update the training set. To further improve the inference, we incorporate personal history priors to modify the base learner. We evaluate our designs using two data sources: a real-data inspired traffic-passenger behavior simulation in the city of Porto, Portugal, and the real bus Automated Fare Collection (AFC) data collected from the same city. The experimental results show that compared to baseline methods that do not use self-training, our approach significantly improves the inference performance and achieves notably high accuracies.

Application-specific integrated circuit (ASIC) implementations for Deep Neural Networks (DNNs) have been adopted in many systems because of their higher classification speed. However, although they may be characterized by better accuracy, larger DNNs require significant energy and area, thereby limiting their wide adoption. The energy consumption of DNNs is driven by both memory accesses and computation. Binarized Neural Networks (BNNs), as a trade-off between accuracy and energy consumption, can achieve great energy reduction, and have good accuracy for large DNNs due to its regularization effect. However, BNNs show poor accuracy when a smaller DNN configuration is adopted. In this paper, we propose a new DNN model, LightNN, which replaces the multiplications to one shift or a constrained number of shifts and adds. For a fixed DNN configuration, LightNNs have better accuracy at a slight energy increase than BNNs, yet are more energy efficient with only slightly less accuracy than conventional DNNs. Therefore, LightNNs provide more options for hardware designers to make trade-offs between accuracy and energy. Moreover, for large DNN configurations, LightNNs have a regularization effect, making them better in accuracy than conventional DNNs. These conclusions are verified by experiment using the MNIST and CIFAR-10 datasets for different DNN configurations.

This paper introduces a novel design of phase locked loop (PLL) based oscillatory neural networks (ONNs) to mitigate the frequency clustering phenomenon caused by transmission delays in real systems. Theoretical analysis of the ONN reveals that transmission delays can produce frequency clustering that leads to synchronization and convergence failure. This paper describes the redesign of ONN dynamics and associated system-level architecture to achieve robustness. Specifically, we first demonstrate that using the phase information of zero-crossing points of inputs as the PLL error signal enables the ONN dynamical model to correctly synchronize under uniform transmission delays. A Type-II PLL based ONN architecture is shown via simulation to provide this property in hardware. Furthermore, to accommodate non-uniform transmission delays in hardware, a phase synchronization technique is proposed that is shown to provide the correct synchronization behavior.