Behavior trees (BTs) have been widely adopted in autonomous task planning because of their modularity and reactivity. Recently, automatic BT generation based on Large Language Models (LLMs) has attracted growing research attention. However, synthesizing BTs for long-horizon tasks without relying on predefined expert rules remains an open problem, and it poses two key challenges: ensuring the logical consistency of the generated BTs, and maintaining their factual alignment with the ground truth of the environment. To address these challenges, this paper presents Veritas, a novel verification-driven framework for automatically generating logically consistent BTs. Veritas integrates STRIPS-like symbolic operators with a multi-layered verification mechanism, thereby transforming the planning process into a coherent chain of logical derivations. We further introduce Veritas+, which augments this framework with a memory module that accumulates both successful and failed execution experiences, enabling dynamic self-correction and improving factual consistency. We evaluate the framework on 67 long-horizon tasks in Minecraft and 116 real-world tasks in AndroidWorld. Experimental results show that Veritas and Veritas+ are highly effective and significantly outperform state-of-the-art baselines.

Multi-modal large language models (MLLMs) are expanded from large language models (LLMs) with additional capabilities to infer multi-modal data. Current MLLM workflows, when dealing with complex tasks, typically begin by using an LLM to decompose the task into multiple subtasks, then heuristically select a specific pre-trained model to complete a subtask to get a result, and finally integrate all the results to obtain the final response. However, heuristically binding one model to one subtask may generate a less satisfying subtask result, thereby affecting the overall performance. Therefore, we propose CompeMLLM, which introduces an innovative method of dynamic orchestration of the workflows. It allows different models to compete on the same subtask instead of statically binding them. By dynamically integrating the results from diverse models, the optimal subtask result is determined, thereby improving the overall performance of MLLM. Specifically, given a certain complex task, CompeMLLM first decomposes it into subtasks, then employs multiple pre-trained models to execute the same subtask in parallel to compete, and then the optimal subtask result is chosen by dynamically evaluating these results using ensemble learning idea, and finally integrates these optimal results into a complete workflow, thus obtaining the best overall performance. We conducted extensive experiments using six advanced MLLMs as baselines across seven diverse datasets. The experimental results robustly demonstrate that CompeMLLM achieves significantly improved performance on all datasets, demonstrating its effectiveness.

The surge of multimedia content-spanning images, audio, video, and text-on digital platforms has heightened the need for sophisticated bundle recommendation techniques to curate cohesive item sets that resonate with users' preferences amidst rich media environments. Bundle recommendation aims to recommend thematically related item sets to users based on their preferences by simulating their cognitive decision-making process. It not only enhances user experience but also significantly boosts merchant profits, making it an increasingly crucial research area. However, existing studies often face cognitive limitations in modeling user preferences. They typically adopt a unified rule to learn user and bundle representations from items, failing to capture the diversity of bundling strategies and the underlying factors of user preferences. Therefore, we propose a Category-Aware Bundle Recommendation model, called CABRec. Specifically, CABRec learns user preferences from three views: User-Bundle (UB), User-Item (UI) and Bundle-Item (BI). From the UB view, we model users' direct preferences for bundles based on user-bundle interactions. From the UI and BI views, we model users' indirect preferences for bundles by constructing item-bundle layer and item-user layer to learn category-aware representations of bundles and users, respectively. Then, to capture the preference variations of different users on different views, we propose a user-specific prediction layer to learn a set of personalized preference weights for each user. Finally, we apply contrastive learning across these three views to utilize the complementary information they provide, enabling the model to learn more robust and generalizable representations. Extensive experiments on three datasets demonstrate that our method surpasses the strongest baseline, achieving a 2.33% ~ 17.02% improvement on recall.

We explore visual reinforcement learning (RL) using two complementary visual modalities: frame-based RGB cam-era and event-based Dynamic Vision Sensor (DVS). Ex-isting multi-modality visual RL methods often encounter challenges in effectively extracting task-relevant information from multiple modalities while suppressing the in-creased noise, only using indirect reward signals instead of pixel-level supervision. To tackle this, we propose a Decomposed Multi-Modality Representation (DMR) framework for visual RL. It explicitly decomposes the inputs into three distinct components: combined task-relevant features (co-features), RGB-specific noise, and DVS-specific noise. The co-features represent the full information from both modalities that is relevant to the RL task; the two noise components, each constrained by a data reconstruction loss to avoid information leak, are contrasted with the co-features to maximize their difference. Extensive experiments demonstrate that, by explicitly separating the different types of information, our approach achieves substan-tially improved policy performance compared to state-of-the-art approaches.

The black-box characteristic of deep reinforcement learning restricts the safe and scalable application of decision models in practical deployment. Existing interpretability methods for deep reinforcement learning models are often inadequate in providing comprehensive insights and generating logical sequential decisions. In this study, we propose an innovative framework called XRLBT, which introduces the behavior tree structure to explainable reinforcement learning. XRLBT clusters state space by aggregating temporally related states. Based on these clustered states, XRLBT constructs behavior tree structures that align with the target deep reinforcement learning model. In this way, XRLBT incorporates an exploration technique specifically designed for temporal policies in DRL. Through extensive experimentation across six benchmark environments, we showcase the superiority of the discovered behavior tree structures over state-of-the-art algorithms. This work represents a significant stride towards addressing the challenges of explainability and performance in DRL applications.

Multi-agent reinforcement learning (MARL) effectively improves the learning speed of agents in sparse reward tasks with the guide of subgoals. However, existing works sever the consistency of the learning objectives of the subgoal generation and subgoal reached stages, thereby significantly inhibiting the effectiveness of subgoal learning. To address this problem, we propose a novel Potential field Subgoal-based Multi-Agent reinforcement learning (PSMA) method, which introduces the potential field (PF) to unify the two-stage learning objectives. Specifically, we design a state-to-PF representation model that describes agents' states as potential fields, allowing easy measurement of the interaction effect for both allied and enemy agents. With the PF representation, a subgoal selector is designed to automatically generate multiple subgoals for each agent, drawn from the experience replay buffer that contains both individual and total PF values. Based on the determined subgoals, we define an intrinsic reward function to guide the agent to reach their respective subgoals while maximizing the joint action-value. Experimental results show that our method outperforms the state-of-the-art MARL method on both StarCraft II micro-management (SMAC) and Google Research Football (GRF) tasks with sparse reward settings.

The quadrotor unmanned aerial vehicle (UAV) is widely used due to its low maintenance cost, high maneuverability and strong hovering capability. Modeling the quadrotor UAV maneuver and simulating its performance can effectively support airborne intelligent algorithms training such as mission planning and scheduling. Traditional quadrotor UAV maneuver modeling method construct high-order mathematical model based on physics analysis, which require significant expertise and difficult to generalize. In this paper, we analyze the quadrotor UAV maneuvering process and propose a cascaded quadrotor UAV maneuvering model generating framework based on deep neural network. Using long short-term memory (LSTM) network to model each part of the quadrotor UAV maneuvering process individually, and flexibly combine network of each part to obtain varying granularity models. A variable-dimensional particle swarm optimization (PSO) algorithm based on detour foraging strategy is proposed to simultaneously determine the LSTM network's hidden layers and neurons of each hidden layer. We validate the effectiveness of the maneuvering model generation framework and the improved PSO algorithm through comparative experiments.

In some complex multi-agent environments, the types of relationships between agents are diverse and their intensity changes during the policy learning process. Theoretically, some of these relationships can facilitate cooperative policy learning. However, acquiring these relationships is an intractable problem. To tackle the problem, we propose a diverse effective relationship exploration based multi-agent reinforcement learning (DERE) method. Specifically, a potential fields model is firstly designed to represent relationships between agents. Then to encourage the exploration of effective relationships, we define an information-theoretic objective function. Finally, an intrinsic reward function is designed to optimize the information-theoretic objective, meanwhile, guide agents to learn more effective collaborative policies. Experimental results show that our method outperforms state-of-the-art methods on both super hard StarCraft II micromanagement tasks (SMAC) and Google Research Football (GRF).

The role-oriented learning approach could improve the performance of multi-agent reinforcement learning by decomposing complex multi-agent tasks into different roles. However, due to the dynamic environment and interactions among agents, the role undertaken by an agent changes rapidly with time going on. Therefore, the roles of agents should be adapted to the varying situation during the learning process. In this paper, we propose a role-oriented graph convolution based multi-agent reinforcement learning framework (ROGC). Firstly, we design a role assigner based on samples generated from the environment to learn roles for classifying agents into different groups. To further enhance cooperation among agents in the same group for higher performance, we design a graph convolutional module to achieve intra-role communications based on discovered roles. With roles and extracted role features, we design a role-oriented policy learning module that embeds the role information into the algorithm and generates effective policies for individuals. Further, we introduce an auto-encoder to learn the intra-role cooperation knowledge in the graph convolutional module, which ensures our framework executes in a decentralized way. Extensive experiments show that our framework can learn dynamic roles and make full use of learned roles, which makes it outperform popular MARL methods.

Modeling user preferences from users' historical sequences is one of the core problems of sequential recommendation. Previous studies only considered transition patterns between user-items, ignoring transition patterns between item features and item-item interactions. Recently, there has been interest in integrating knowledge graphs as auxiliary information into sequential recommendation. Most of the existing methods deal with the heterogeneous information in the knowledge graph in a coarse-grained manner. We believe that fine-grained processing of information in knowledge graphs can help recommendation systems understand changes in user preferences, i.e. dividing hetero-geneous information into item-to-item relationships and item-to-attribute relationships. In this paper, we propose a dual-level self-attention network for sequential recommendation. Specifically, we divide the knowledge graph heterogeneous information about items into relation level and attribute level, representing item-item relationship and item-attribute relationship, respectively. Afterwards, the self-attention network is used to learn user preferences at dual-level, respectively. Then, the outputs of the above dual levels are integrated for next item recommendation. Based on extensive experiments on three real-world data sets, our model achieves significant improvements compared to state-of-the-art baseline methods.

The distributed system Ray has attracted much attention for many decision-making applications. It provides a flexible and powerful distributed running mechanism for the training of the learning algorithms, which could map the computation tasks to the resources automatically. Task scheduling is a critical component in Ray, adopting a two-layer structure. It uses a simple general scheduling principle, which leaves much space to optimize. In this paper, we will study the two-layer scheduling problem in Ray, setting it as an optimization problem. We firstly present a comprehensive formulation for the problem and point out that it is a NP-hard problem. Then we design a hierarchical reinforcement learning method, named HierRL, which consists of a high-level agent and a low-level agent. Sophisticated state space, action space, and reward function are designed for this method. In the high level, we devise a value-based reinforcement learning method, which allocates a task to an appropriate node of the low level. With tasks allocated from the high level and generated from applications, a low-level reinforcement learning method is constructed to select tasks from the queue to be executed. A hierarchical policy learning method is introduced for the training of the two-layer agents. Finally, we simulate the two-layer scheduling procedure in a public platform, Cloudsim, with tasks from a real Dataset generated by the Alibaba Cluster Trace Program. The results show that the proposed method performs much better than the original scheduling method of Ray.

Heterogeneous behaviours are widespread in many multi-agent tasks, which have not been paid much attention in the community of multi-agent reinforcement learning. It would be a key factor for improving the learning performance to efficiently characterize and automatically find heterogeneous behaviours. In this paper, we introduce the concept of the skill to explore the ability of heterogeneous behaviours. We propose a novel skill-based multi-agent reinforcement learning framework to enable agents to master diverse skills. Specifically, our framework consists of the skill representation mechanism, the skill selector and the skill-based policy learning mechanism. We design an auto-encoder model to generate the latent variable as the skill representation by incorporating the environment information, which ensures the distinguishable of agents for skill selection and the discriminability for the skill learning. With the representation, a skill selection mechanism is invented to realize the assignment from agents to skills. Meanwhile, diverse skill-based policies are generated through a novel skill-based policy learning method. To promote efficient skill discovery, a mutual information based intrinsic reward function is constructed. Empirical results show that our framework obtains the best performance on three challenging benchmarks, i.e., StarCraft II micromanagement tasks, Google Research Football and GoBigger, over state-of-the-art MARL methods.

Knowledge graph completion can complete knowledge by predicting missing facts, which is a increasingly hot research topic in knowledge graph construction.Prevalent approaches propose to embed knowledge graphs in a lowdimensional vector space and use these embedding to predict, but they neglect either semantic information or graph structures.We propose a new approach to knowledge graph completion named as ATTWALK, which learns embedding by exploiting both structural and semantic features of a knowledge graph.This is achieved by leveraging a key insight that an entities' embedding is influenced by its multi-hop neighbors', which can be further distinguished by their semantic importance to the entity.ATTWALK orchestrates a two-step workflow by first evaluating neighbors' semantic weights using graph attention networks for each entity, then exploring the entities' local structural features by performing a semantic weight guided walk.We evaluate ATTWALK by conducting extensive experiments, which show that ATTWALK outperforms 12 representative approaches on average across 3 publicly available datasets.

The distributed system Ray has attracted much attention in decision-making applications, which could greatly accelerate the training efficiency for intelligent algorithms. Task scheduling is one of the critical technologies in Ray, in which the number of resource nodes could be auto-scaling, i.e., automatically increasing or decreasing according to the workload. The adopted scheduling strategy is simple, which leaves much space to be optimized. In this paper, we consider designing a reinforcement learning method to optimize the scheduling problem in Ray. We propose an attentive reinforcement learning method, designing an attention-based state encoder that could efficiently extract the system state in the situation of the varying number of resource nodes. At the same time, an action mask mechanism filters invalid actions. Further, to improve the learning efficiency in the environment with the varied number of nodes, we design a curriculum learning method, which trains the method by gradually increasing the number of nodes in the scheduling process. Finally, we use the real data generated by the Alibaba Cluster Trace Program to test in the simulation platform CloudSim. The experimental results show that the proposed method effectively scales down the completion time of tasks compared to the original algorithm in Ray.

Nowadays, simulation environments are widely used in different research fields. As the complexity of the problem increases, Cameras-Sampling in a simulation environment is important since it can provide highly realistic training data for online machine learning and reinforcement learning, etc. This paper attempts to improve the throughput and efficiency when multiple camera sampling are performed simultaneously. We implement a high-throughput platform for Real-time Cameras-Sampling based on Unreal Engine 4. The platform uses multi-graphics queues to support multi-cameras sampling in parallel. We construct a virtual desert scene to verify the correctness and effectiveness of the proposed platform. The experiments show that the proposed platform can generate eight 960x640 pixels pictures at a frequency of 30Hz. The throughput and GPU utilization have been increased by 2.57x and 1.43x, respectively, compared with Unreal Engine4.

Abstract Scene graph generation refers to the task of identifying the objects and specifically the relationships between the objects from an image. Existing scene graph generation methods generally use the bounding boxes region features of objects to identify the relationships between objects. However, we feel that the overlap region features of two objects may play an important role in fine‐grained relationship identification. In fact, some fine‐grained relationships can only be obtained from the overlap region features of two objects. Therefore, we propose the Multi‐Branch Feature Combination (MFC) module and Overlap Region Transformer (ORT) module to comprehensively obtain the visual features contained in the overlap regions of two objects. Concretely, the MFC module uses deconvolution and multi‐branch dilation convolution to obtain high‐pixels and multi‐receptive field features in the overlap regions. The ORT module uses the vision transformer to obtain the self‐attention of the overlap regions. The joint use of these two modules achieves the mutual complementation of local connectivity properties of convolution and the global connectivity properties of attention. We also design a Geometrical Center Augmented (GCA) module to obtain the relative position information of the geometric centers between two objects, to prevent the problem that only relying on the scale of the overlap region cannot accurately capture the relationship between two objects. Experiments show that our model ORGC (Overlap Region and Geometrical Center), the combination of the MFC module, the ORT module, and the GCA module, can enhance the performance of fine‐grained relation identification. On the Visual Genome dataset, our model outperforms the current state‐of‐the‐art model by 4.4% on the R@50 evaluation metric, reaching a state‐of‐the‐art result of 33.88.

Pommerman is a recently-proposed multi-agent benchmark, which is very challenging for Reinforcement Learning (RL). The main obstacles to adopt RL in Pommerman are the delayed action effects and sparse rewards. This paper presents novel approaches to mitigate these problems by introducing Artificial Potential Field (APF) in the two-dimensional Pommerman world. We propose a new framework to generate hybrid features from both APF computation and raw environment data. Meanwhile, a new reward shaping method through APF is developed to give the learning agent faster and more efficient policy iteration. The training results show that the learning speed and convergence reward are both improved on a 1v1 mode of the Pommerman game, compared to the conventional learning algorithms, A2C and ACKTR.

Reinforcement Learning (RL) platforms play an important role in translating the rapid advances of RL algorithms into the successes of real-world tasks. These platforms integrate multiple simulation environments, allowing testing, evaluating and finally applying RL algorithms in different scenarios. However, the algorithm code is required to execute in the same runtime system with the underlying environments, which limits platforms' compatibility when adapting an algorithm and flexibility when switching between different algorithms. We propose GLUE to resolve this issue, by decoupling the executions of algorithms and environments first, then leveraging the RPC protocol to orchestrate a seamless workflow between them. GLUE is further implemented as a library, which hides the handling of language-specific RPCs from users. We evaluate GLUE by adapting 6 RL algorithm implementations to a representative RL platform. Compared with the baseline approach, GLUE enables algorithms to achieve competitive performance, but reduces lines of algorithm code to be changed in adaption by 27.77% , at the cost of 5.40% longer training time, on average.

Summary For parallel multiphase flows, the core procedure is solving linear systems using preconditioned iterative methods, and the iterative convergence rate is crucial to the overall efficiency. How the mesh is partitioned influences the iterative convergence rate. However, the numerical characteristics of the linear systems vary significantly along with the time steps because of the dramatic change of the flow fields. Traditional static mesh partition cannot guarantee a good convergence feature throughout the simulation. A dynamic mesh re‐partitioning scheme MDMRPar (Multiphase Dynamic Mesh Re‐Partitioning) is proposed and implemented in OpenFOAM (Open Field Operation And Manipulation). MDMRPar employs a new surface field to record the linear system information in every time step. For simple multiphase problems with topo‐invariant mesh, MDMRPar periodically adopts the numerical information from the previous time step and calculates a weighted graph from the mesh topology in a straightforward way. For multiphase flows using adaptive mesh refinement method, a coarse graph with vertex weights is built based on the mesh topology and refinement history. The numerical information on the surface field is integrated into the coarse graph as edge weights. In both cases, weighted graphs are re‐partitioned by ParMetis, a general multi‐level parallel graph re‐partitioning package. Experimental results on three multiphase flows show that MDMRPar significantly outperforms the traditional partitioning/re‐partitioning schemes both in the iterative convergence rate and the total simulation time.

The method of Brownian configuration fields (BCF) is a promising multi-scale approach for the simulationof viscoelastic fluids, however, it is a computationally expensive method, which restricts its application in complex scenarios. Therefore, it is of great importance to optimize the parallel implementation in order to improve computational efficiency. In this paper we propose a hybrid decomposition parallel algorithm named: MCDPar, whichenables the simulation problem to bedecomposed simultaneously over mesh cells andthe Brownian configuration fields. Compution processes are split into multiple groups and Brownian configuration fields are equally associated with these groups. Meanwhile, within each group the processes are concurrently executed based on the traditional mesh decomposition approach. Finally we implemented the MCDPar algorithm in a micro-macro numerical solver based on OpenFOAM. Experimental results show that the micro-macro simulation time of viscoelastic fluids issignificantly reduced with improved scalability and parallel efficiency. In the test case with Nf= 2000 and Ncell= 262144, the speedup of the MCDParis up to 9.23x with a 7.5x increase in number of cores compared to the original parallel algorithm.