When planning intra-urban control measures for epidemics with significant societal impact, it is essential to consider the spatiotemporal orderliness of interventions, thus mitigating the disruption to daily life. For instance, improving intervention consistency among highly interacted sub-regions and avoid frequent and significant changes of interventions over time can be effective. However, existing studies on optimizing epidemic control overlooked the need for spatiotemporal consistency and stability of the interventions, potentially impacting their practicality and public compliance. To fill this gap, this study systematically conceptualized and quantified spatiotemporal orderliness for intra-urban epidemic control. A deep-reinforcement-learning (DRL) framework integrating the spatiotemporal orderliness was proposed to optimize the interventions across sub-regions over time. Taking Shenzhen, China as a study area, we solve a joint control plan for 74 sub-regions based on a meta-population SEIR epidemic model with a real-world intra-urban mobility network. The results demonstrate that the proposed model can effectively generate tailored dynamic interventions for sub-regions, significantly improving spatiotemporal orderliness. Furthermore, the effectiveness and generalizability of proposed model are demonstrated under different urban structures and transmissibility of respiratory viruses. Overall, this study provides a DRL-based tool for planning intra-urban epidemic control measures with enhanced spatiotemporal orderliness, potentially aiding future epidemic preparedness.

Gaussian process (GP) offers a robust solution for modeling the dynamics of unmanned surface vehicles (USV) in model-based reinforcement learning (MBRL). However, the rapidly increasing computational complexity with a large sample capacity of GP limits its application in complex scenarios that require substantial samples to cover the state space. In this article, a novel probabilistic MBRL approach, probabilistic neural networks model predictive control (PNMPC) is proposed to tackle this issue. With an iterative learning framework, PNMPC properly models the USV dynamics using neural networks from a probabilistic perspective to avoid the computational complexity associated with sample capacity. Employing this model to effectively propagate system uncertainties, a model predictive control (MPC) policy is developed to robustly control the USV against external disturbances. Evaluated by position-keeping and multiple targets-tracking scenarios on a real USV data-driven simulation, the proposed method consistently demonstrates its significant superiority in both model accuracy and control performance compared to not only GP model-based approaches but also the probabilistic neural networks-based MBRL baselines, across various scales of external disturbances.Note to Practitioners—Modelling the system dynamics and maintaining computational efficiency with a large sample set has been challenging for MBRL in the USV domain. We propose a novel neural network modeling method to capture the dynamic features of USV within an RL loop and develop a robust MPC policy based on its uncertainty propagation. Our method achieves computational complexity independent of the sample capacity and outperforms related baselines in model accuracy and control performance.

In this paper, we propose practical model-based policy optimization (PMBPO) to address the time efficiency issue caused by overly frequent model updates in recent probabilistic model-based reinforcement learning (MBRL) methods that accelerate learning by generating samples from the model. PMBPO enhances the reliability of the generated samples by introducing an expressive probabilistic model that focuses on the system’s dynamic features over continuous time steps. A time-efficient learning framework is proposed by offline updating and interacting with the model at the end of each epoch. One policy fallback mechanism is further designed to mitigate the negative impact of model bias on the learned policy. Evaluated on five Mujoco control benchmarks and one quadruped robot control scenario, PMBPO reduces the one-step computation time by 90% while achieving 70% more cumulative rewards compared to the state-of-the-art MBRL approaches. It extends the feasibility of MBRL in practical control scenarios. The code of PMBPO is available at https: //github.com/mrjun123/PMBPO.

This paper focused on developing an effective Multi-Agent Reinforcement Learning (MARL) approach that quickly explores optimal control policies of multiple agents through interactions with unknown environments. Multi-Agent Continuous Dynamic Policy Gradient (MACDPP) was proposed to tackle the issues of limited capability and sample efficiency in the current MARL approaches. It alleviates the inconsistency of multiple agents’ policy updates by introducing the relative entropy regularization to the Centralized Training with Decentralized Execution (CTDE) framework with the Actor-Critic (AC) structure. Evaluated by multi-agent cooperation and competition tasks and traditional control tasks including OpenAI benchmarks and robot arm manipulation, MACDPP demonstrates its significant superiority in learning capability and sample efficiency compared with both related multi-agent and widely implemented signal-agent baselines. It converges to$62\%$higher average return and uses$38\%$fewer samples compared with the suboptimal baseline over all tasks, indicating the potential of MARL in challenging control scenarios, especially when the number of interactions is limited. The open source code of MACDPP is available at https://github.com/AdrienLin1/MACDPP.Note to Practitioners—Learning proper cooperation strategy over multiple agents in complicated systems has been a challenge in the domain of Reinforcement Learning. Our work extends the traditional MARL approach FKDPP that has been successfully implemented in the real-world chemical plant by Yokogawa to the CTDE framework and AC structure that supports continuous actions. This extension significantly expands its range of applications from cooperative/competitive tasks to the joint control of one complex system while maintaining its effectiveness.

This article addresses the robustness of unmanned surface vehicles (USV) using model-based reinforcement learning (MBRL). A novel MBRL approach, Lyapunov probabilistic model predictive control (LPMPC) is proposed to simultaneously learn both the probabilistic model of a USV and its corresponding estimated Lyapunov region of attraction (ROA) under one reinforcement learning framework. Unlike the existing MBRL USV systems with less consideration of robustness and safety, our method naturally learns a general indicator of system stability based on the probabilistic model’s belief and employs it to guide its policy. Evaluated by different navigation tasks in a simulation driven by real boat data, LPMPC demonstrated significant advantages in both control robustness and task completion against various levels of environmental disturbances compared with the baseline approach without Lyapunov ROA’s guidance. Note to Practitioners—Modelling the system stability without human prior knowledge is challenging in the domain of USV. This work proposed a data-driven method to iteratively learn a task-relevant stability model of USV in a probabilistic view. Based on the evaluation of a real boat data-driven simulation, the learned stability model contributed to superior driving skills in different USV scenarios by properly indicating and avoiding potentially risky states. In future research, we plan to expand the definition of risks in different tasks, such as loss of control, overlarge sway, and excessive energy consumption and investigate the proposed approach in real-world USV.

In the field of mineral processing, controlling the paste thickener is a highly challenging and critical task because of the high complexity, incomplete observation space, and excessive environmental noises. In this article, we propose an offline-data-driven controlling strategy to optimize the operational indices in the thickening system based on offline reinforcement learning (RL). Compared to common RL methods that rely on online interactive training, our approach ensures the safety of the production process by training the controller solely using offline datasets, thereby avoiding dangerous online exploration. In terms of offline dataset collection, this study utilizes the prior knowledge of the thickening mechanism to design a proportional–integral–derivative controller as the behavior policy to collect operational trajectories as the offline dataset. In addition, to tackle a critical issue in controlling the thickening system: constrained observation space, this article analyzes the dynamical properties of the thickening system and introduces a novel offline RL algorithm, temporal batch-constrained Q-learning (TBCQ). The algorithm and associated model framework are specifically developed for controlling partially observed Markov decision processes. The TBCQ and trained policy are evaluated in both a simulated thickening environment and a real industrial paste thickener in a copper mine. The real-world experiments demonstrate that the proposed controller outperforms the baselines and effectively reduces the tracking error of underflow concentration by over 12%. The successful application of our pipeline in paste thickener also offers an innovative perspective on addressing optimization problems in complex industrial systems: performing offline RL on a dataset sampled from a suboptimal policy.

This article addresses the prediction stability, prediction accuracy, and control capability of the current probabilistic model-based reinforcement learning (MBRL) built on neural networks. A novel approach to dropout-based probabilistic ensembles with trajectory sampling (DPETS) is proposed, where the system uncertainty is stably predicted by combining the Monte Carlo dropout (MC Dropout) and trajectory sampling in one framework. Its loss function is designed to correct the fitting error of neural networks for more accurate prediction of probabilistic models. The state propagation in its policy is extended to filter the aleatoric uncertainty for superior control capability. Evaluated by several Mujoco benchmark control tasks under additional disturbances and one practical robot arm manipulation task, DPETS outperforms related MBRL approaches in both average return and convergence velocity while achieving superior performance than well-known model-free baselines with significant sample efficiency. The open-source code of DPETS is available at https://github.com/mrjun123/DPETS.

In this article, a novel reinforcement learning (RL) approach, continuous dynamic policy programming (CDPP), is proposed to tackle the issues of both learning stability and sample efficiency in the current RL methods with continuous actions. The proposed method naturally extends the relative entropy regularization from the value function-based framework to the actor-critic (AC) framework of deep deterministic policy gradient (DDPG) to stabilize the learning process in continuous action space. It tackles the intractable softmax operation over continuous actions in the critic by Monte Carlo estimation and explores the practical advantages of the Mellowmax operator. A Boltzmann sampling policy is proposed to guide the exploration of actor following the relative entropy regularized critic for superior learning capability, exploration efficiency, and robustness. Evaluated by several benchmark and real-robot-based simulation tasks, the proposed method illustrates the positive impact of the relative entropy regularization including efficient exploration behavior and stable policy update in RL with continuous action space and successfully outperforms the related baseline approaches in both sample efficiency and learning stability.

In this article, we focus on the computational efficiency of probabilistic model-based reinforcement learning (MBRL) in unmanned surface vehicles (USV) under unforeseeable and unobservable external disturbances. A novel MBRL approach, local update spectrum probabilistic model predictive control (LUSPMPC), is proposed to fully release the superiority of the probabilistic model approximated in the frequency domain in computational efficiency while mitigating its risk of overfitting during the learning procedure. It employs a local update strategy to relieve the violation of Bochner's theory, and a frequency clipping trick to encourage the approximated model to focus on the features in the low-frequency domain. Evaluated by the position-keeping task in a real USV data-driven simulation, LUSPMPC shows its significant advantages in computational efficiency while achieving better learning capability, generalization capability, and control performances in a wide range of sparse scales compared with the baseline MBRL approaches that approximate their models in sample space and frequency domain, and therefore becomes an appealing solution for MBRL USV system defending against rapidly changing ocean disturbances.

In this article, we address the difficulty of controlling unmanned surface vehicles (USVs) under unforeseeable and unobservable external disturbances using model-based reinforcement learning (MBRL) without human’s prior knowledge. A novel MBRL approach, filtered probabilistic model predictive control (FPMPC) is proposed to iteratively learn the USV model and an MPC-based policy in a probabilistic way through trial-and-error interactions. Compared with existing MBRL approaches that model the unobservable disturbances as system noise, FPMPC introduces a Bayesian filter process to implicitly translate the system dynamics to a partially-observed Markov decision process to present those disturbances as hidden states. An adaptive sample selection is proposed to remove the redundant learning samples based on the filter belief. Equipped with bias compensation and parallel computation, an FPMPC system, specific for USV, is developed. Evaluated by both position holding and target reaching tasks in a real USV data-driven simulation, FPMPC shows its significant superiority in control performances, generalization capability, and sample efficiency under large disturbances compared with the baseline approaches.

Understanding and modeling human driver behavior and subsequently applying these patterns in various scenarios is crucial for autonomous vehicle motion planning. However, solution that naturally encodes human driving skills remains challenging due to the difficulty of balancing the variability of human behavior and the robustness in various driving environments. To tackle this issue, a novel motion planning approach is proposed based on Kernelized Movement Primitive (KMP) in this paper to adaptively learn human driving behavior in a stochastic way by employing Gaussian mixture model(GMM) and Gaussian mixture regression(GMR). The Kullback-Leibler(KL) divergence is utilized to minimize the information loss between imitating reference behavior and adapting new tasks and therefore generates robust motion trajectories in a variety of driving situations. The proposed approach is evaluated by a mature urban driving simulator CARLA. The experimental results shows its capability of generating robust driving trajectories by naturally adapting human driving skill into various driving situations.

In this paper, a model predictive control (MPC) approach that combines sparse spectrum Gaussian processes model and unscented Kalman Filter is proposed for path tracking task in autonomous driving. To tackle the difficulty of balancing control performance and computational cost in MPC with Gaussian processes model, the proposed approach employs the sparse spectrum Gaussian processes (SSGP) to efficiently model the vehicle, and utilizes unscented Kalman filter (UKF) to naturally propagate model uncertainties during multiple step prediction of MPC. The proposed approach is evaluated in both a numerical driving simulation and a mature driving simulation CARLA. The results indicate that the proposed method achieves a robust driving performance with a significant reduction of computational complexity.

Gaussian processes (GPs) based Reinforcement Learning (RL) methods with Model Predictive Control (MPC) have demonstrated their excellent sample efficiency. However, since the computational cost of GPs largely depends on the training sample size, learning an accurate dynamics using GPs result in low control frequency in MPC. To alleviate this trade-off and achieve a sample-and-computation-efficient nature, we propose a novel model-based RL method with MPC. Our approach employs a linear Gaussian model with randomized features using the Fastfood as an approximated GP dynamics. Then, we derive an analytic moment-matching scheme in state prediction with the model and uncertain inputs. As a result, the computational cost of the MPC in our RL method does not depend on the training sample size and can improve the control frequency over previous methods. Through experiments with simulated and real robot control tasks, the sample efficiency, as well as the computation efficiency of our model-based RL method, are demonstrated.

Real-world robots have complex strict constraints. Therefore, safe reinforcement learning algorithms that can simultaneously minimize the total cost and the risk of constraint violation are crucial. However, almost no algorithms exist that can scale to high-dimensional systems to the best of our knowledge. In this paper, we propose Dynamic Actor-Advisor Programming (DAAP), as an algorithm for sample-efficient and scalable safe reinforcement learning. DAAP employs two control policies, actor and advisor. They are updated to minimize total cost and risk of constraint violation intertwiningly and smoothly towards each other's direction by using the other as the baseline policy in the Kullback-Leibler divergence of Dynamic Policy Programming framework. We demonstrate the scalability and sample efficiency of DAAP through its application on simulated robot arm control tasks with performance comparisons to baselines.

This paper proposes a probabilistic approach for object search in clutter. Due to heavy occlusions, it is vital for an agent to be able to gradually reduce uncertainty in observations of the objects in its workspace by systematically rearranging them. Probabilistic methodologies present a promising sample-efficient alternative to handle the massively complex state-action space that inherently comes with this problem, avoiding the need for both exhaustive training samples and the accompanying heuristics for traversing a large-scale model during runtime. We approach the object search problem by extending a Gaussian Process active filtering strategy with an additional model for capturing state dynamics as the objects are moved over the course of the activity. This allows viable models to be built upon relatively scarce training data, while the complexity of the action space is also reduced by shifting objects over relatively short distances. Validation in both simulation and with a real Baxter robot with a limited number of training samples demonstrates the efficacy of the proposed approach.

In this research we focus on developing a reinforcement learning system for a challenging task: autonomous control of a real-sized boat, with difficulties arising from large uncertainties in the challenging ocean environment and the extremely high cost of exploring and sampling with a real boat. To this end, we explore a novel Gaussian processes (GP) based reinforcement learning approach that combines sample-efficient model-based reinforcement learning and model predictive control (MPC). Our approach, sample-efficient probabilistic model predictive control (SPMPC), iteratively learns a Gaussian process dynamics model and uses it to efficiently update control signals within the MPC closed control loop. A system using SPMPC is built to efficiently learn an autopilot task. After investigating its performance in a simulation modeled upon real boat driving data, the proposed system successfully learns to drive a real-sized boat equipped with a single engine and sensors measuring GPS, speed, direction, and wind in an autopilot task without human demonstration.

This paper presents an active assistance framework for mobility systems, such as Power Mobility Devices (PMD), with the distinctive goal of being able to operate within a local moving window, as opposed to the common reliance upon persistent global environments and objectives. Demonstration data from able experts driving a simulated mobility aid in a representative indoor setting is used off-line to build behavioral models of navigation postulated separately upon user joystick inputs and on-board sensor data. These models are built respectively via Gaussian Processes for the joystick signals, and a Deep Convolutional Neural Network for the sensor data; in this case a planar LIDAR. Their combined outputs form a continuous distribution of estimated traversal likelihood within the user's immediate space, allowing for real-time stochastic optimal path planning to guide a user to its intended local destination. Moreover, the computational efficiency of the decoupled models permits rapid replanning on-the-fly for a smooth assistive action. On-line and off-line evaluations substantiate the advantages of the framework in generalising intelligent navigational assistance, of particular relevance for users who experience difficulty in safe mobility.

Active assistive mobility systems are largely limited to a-priori mapped environments, whereas their reactive assistive counterparts are in general location independent and focus on the provision of collision avoidance in the immediate space surrounding the platform. This paper presents a framework capable of providing active short-term navigation, combining the intelligence of active assistance with the freedom of location independence. Demonstration data from an able expert while driving the mobility aid in a standard indoor setting is used off-line to learn reference behavioral models of navigation given perceptual information from the platform surroundings and the input controls exerted by the user while navigating. These serve as the foundation for on-line probabilistic short-term destination inference using the instantaneously available data from the user and on-board sensors. This is coupled with a real-time stochastic optimal path generation able to exploit the same short term demonstration paths from the expert with the belief they capture both the driver's awareness of the platform's physical geometry and appropriate behaviors for their surroundings. Experimental results with users of varying proficiency in a setting unvisited in training data show promise in using the framework in assisting users experiencing difficulty in safe power mobility aid use.

Deep reinforcement learning has drawn much attention in robot control since it enables agents to learn control policies from very high dimensional states such as raw images. On the other hand, its dependency upon the availability of a significant quantity of training samples and its fragility in learning makes it difficult to apply for real world robot tasks. To alleviate these issues we propose Deep Dynamic Policy Programming (DDPP), which combines the sample efficiency and smooth policy updates of dynamic policy programming with the contemporary deep reinforcement learning framework. The effectiveness of the proposed method is first demonstrated in a simulation of the robot arm control problem, with comparison to Deep Q-Networks. As validation on a real robot system, DDPP also successfully learned the flipping of a handkerchief with a NEXTAGE humanoid robot using a reduced number of learning samples, whereas Deep Q-Networks failed to learn the task.