Multi-robot reinforcement learning (MRRL) is a promising approach to solving cooperation problems and has been widely adopted in many applications. In the past decades, researchers have proposed various approaches to improve the efficiency of MRRL. However, most of them are trained and evaluated only in simulated environments with simple interaction scenarios. The problem of how these methods perform in the real-world environment with complex interaction scenarios remains unsolved. To meet this emergent need, we introduce a scalable multi-robot reinforcement learning platform (SMART) for training and evaluation. Specifically, SMART consists of two components: 1) a simulation environment with an uncertainty-aware social agent model that provides a variety of complex interaction scenarios for training and 2) a real-world multi-robot system for realistic performance evaluation. To evaluate the generalizability of MRRL baselines, we introduce a novel generalization metric that takes into account their performance across changes in the environment as well as the policies of other agents. Furthermore, we conduct a case study on the multi-vehicle cooperative lane change and summarize the unique challenges of MRRL, which are rarely considered previously. Finally, we open-source the simulation environments, associated benchmark tasks, and state-of-the-art baselines to encourage and empower MRRL research. Our code is available at https://github.com/Blackmamba-xuan/MRST.

Pipelines have become one of the most important infrastructures in the city. Over time, they are prone to aging, cracks, corrosion, and the demand for regular inspection is gradually increasing. Robotic solutions are effective methods for in-pipe inspection. However, existing In-pipe Inspection Robots (IPIR) require that the inner diameter of the pipe is fixed in the application scenarios, and need extra labor to control the robot and handle the cable. In this work, we design and develop a deformable robot to adapt to pipes with different inner diameters. Specifically, the passive elastic hinge is used by us to make the robot fully in contact with the pipe, generating enough friction to ensure that the robot is attached to the inner wall of the pipe. An edge device is deployed on the robot, generating velocity commands of wheels through the data from Inertial Measurement Unit (IMU), which eliminates the need for external devices. Experimental results demonstrate that the robot can move in horizontal and vertical pipelines, as well as traverse through pipe joints and scenarios where there is dirty or small obstacle.

Many real-world applications can be formulated as multi-agent cooperation problems, such as network packet routing and coordination of autonomous vehicles. The emergence of deep reinforcement learning (DRL) provides a promising approach for multi-agent cooperation through the interaction of the agents and environments. However, traditional DRL solutions suffer from the high dimensions of multiple agents with continuous action space during policy search. Besides, the dynamicity of agents’ policies makes the training non-stationary. To tackle the issues, we propose a hierarchical reinforcement learning approach with high-level decision-making and low-level individual control for efficient policy search. In particular, the cooperation of multiple agents can be learned in high-level discrete action space efficiently. At the same time, the low-level individual control can be reduced to single-agent reinforcement learning. In addition to hierarchical reinforcement learning, we propose an opponent modeling network to model other agents’ policies during the learning process. In contrast to end-to-end DRL approaches, our approach reduces the learning complexity by decomposing the overall task into sub-tasks in a hierarchical way. To evaluate the efficiency of our approach, we conduct a real-world case study in the cooperative lane change scenario. Both simulation and real-world experiments show the superiority of our approach in the collision rate and convergence speed.

Although molecular dynamics (MD) simulations of biomolecular systems often run for days to months, many events of great scientific interest and pharmaceutical relevance occur on long time scales that remain beyond reach. We present several new algorithms and implementation techniques that significantly accelerate parallel MD simulations compared with current state-of-the-art codes. These include a novel parallel decomposition method and message-passing techniques that reduce communication requirements, as well as novel communication primitives that further reduce communication time. We have also developed numerical techniques that maintain high accuracy while using single precision computation in order to exploit processor-level vector instructions. These methods are embodied in a newly developed MD code called Desmond that achieves unprecedented simulation throughput and parallel scalability on commodity clusters. Our results suggest that Desmond's parallel performance substantially surpasses that of any previously described code. For example, on a standard benchmark, Desmond's performance on a conventional Opteron cluster with 2K processors slightly exceeded the reported performance of IBM's Blue Gene/L machine with 32K processors running its Blue Matter MD code.