Simulation to Real-World Transfer allows affordable and fast training of learning-based robots for manipulation tasks using Deep Reinforcement Learning methods. Currently, Asymmetric Actor-Critic approaches are used for Sim2Real to reduce the rich idealized features in simulation to the accessible ones in the real world. However, the feature reduction from the simulation to the real world is conducted through an empirically defined one-step curtail. Small feature reduction does not sufficiently remove the actor’s features, which may still cause difficulty setting up the physical system, while large feature reduction may cause difficulty and inefficiency in training. To address this issue, we proposed Curriculum-based Sensing Reduction to enable the actor to start with the same rich feature space as the critic and then get rid of the hard-to-extract features step-by-step for higher training performance and better adaptation for real-world feature space. The reduced features are replaced with random signals from a Deep Random Generator to remove the dependency between the output and the removed features and avoid creating new dependencies. The methods are evaluated on the Allegro robot hand in a real-world in-hand manipulation task. The results show that our methods have faster training and higher task performance than baselines and can solve real-world tasks when selected tactile features are reduced.

In-hand manipulation is challenging for a multi-finger robotic hand due to its high degrees of freedom and complex interaction with the object. To enable in-hand manipulation, existing deep reinforcement learning-based approaches mainly focus on training a single robot-structure-specific policy through the centralized learning mechanism, lacking adaptability to changes like robot malfunction. To solve this limitation, this work treats each finger as an individual agent and trains multiple agents to control their assigned fingers to complete the in-hand manipulation task cooperatively. We propose the Multi-Agent Global-Observation Critic and Local-Observation Actor (MAGCLA) method, where the critic can observe all agents' actions globally, and the actor only locally observes its neighbors' actions. Besides, conventional individual experience replay may cause unstable cooperation due to the asynchronous performance increment of each agent, which is critical for in-hand manipulation tasks. To solve this issue, we propose the Synchronized Hindsight Experience Replay (SHER) method to synchronize and efficiently reuse the replayed experience across all agents. The methods are evaluated in two in-hand manipulation tasks on the Shadow dexterous hand. The results show that SHER helps MAGCLA achieve comparable learning efficiency to a single policy, and the MAGCLA approach is more generalizable in different tasks. The trained policies have higher adaptability in the robot malfunction test compared to the baseline multi-agent and single-agent approaches.

The field of autonomous driving has seen increasing proposed use of machine learning methodologies. However, there are still challenges in applying such methods since autonomous driving involves complex and dynamic interactions with the environment. Supervised learning algorithms such as imitation learning can work in environments represented in the training data set, however, it is impractical or cost-prohibitive to collect data for all possible environments. Reinforcement learning methods can train the agent through trial and error, but it may still fail in new environments. To overcome these shortcomings, we thus propose a meta reinforcement learning (MRL) method to improve the agent's generalization capabilities to new environments. The method is applied to automated lane-changing maneuvers at different traffic congestion levels. Specifically, we train the model at light to moderate traffic conditions under a reinforcement learning framework, and then test it at heavy traffic conditions that are never encountered during training. For performance evaluation, we use both collision rate and success rate of the lane-change maneuvers to quantify the safety and effectiveness of the proposed model. A pretrained model is established as a benchmark, which uses the same network structure and training tasks as our proposed model for fair comparison. Simulation results show that the proposed method achieves an overall success rate up to 20% higher than the benchmark model when it is generalized to the new environment of heavy traffic density. The collision rate is also reduced by up to 18% compared with the benchmark model. Furthermore, the proposed model shows more stable and efficient adaptation capabilities and can achieve 100% successful rate and 0% collision rate with only a few steps of gradient update adaptation.

Applying Deep Reinforcement Learning (DRL) to Human-Robot Cooperation (HRC) in dynamic control problems is promising yet challenging as the robot needs to learn the dynamics of the controlled system and dynamics of the human partner. In existing research, the robot powered by DRL adopts coupled observation of the environment and the human partner to learn both dynamics simultaneously. However, such a learning strategy is limited in terms of learning efficiency and team performance. This work proposes a novel task decomposition method with a hierarchical reward mechanism that enables the robot to learn the hierarchical dynamic control task separately from learning the human partner’s behavior. The method is validated with a hierarchical control task in a simulated environment with human subject experiments. Our method also provides insight into the design of the learning strategy for HRC. The results show that the robot should learn the task first to achieve higher team performance and learn the human first to achieve higher learning efficiency.

Lane-change maneuvers are commonly executed by drivers to follow a certain routing plan, overtake a slower vehicle, adapt to a merging lane ahead, etc. However, improper lane change behaviors can be a major cause of traffic flow disruptions and even crashes. While many rule-based methods have been proposed to solve lane change problems for autonomous driving, they tend to exhibit limited performance due to the uncertainty and complexity of the driving environment. Machine learning-based methods offer an alternative approach, as Deep reinforcement learning (DRL) has shown promising success in many application domains including robotic manipulation, navigation, and playing video games. However, applying DRL to autonomous driving still faces many practical challenges in terms of slow learning rates, sample inefficiency, and safety concerns. In this study, we propose an automated lane change strategy using proximal policy optimization-based deep reinforcement learning, which shows great advantages in learning efficiency while still maintaining stable performance. The trained agent is able to learn a smooth, safe, and efficient driving policy to make lane-change decisions (i.e. when and how) in challenging situation such as dense traffic scenarios. The effectiveness of the proposed policy is validated by using metrics of task success rate and collision rate. The simulation results demonstrate the lane change maneuvers can be efficiently learned and executed in a safe, smooth and efficient manner.

Although current research focused on sensor deployment to balance the energy consumption among the sensor nodes for prolonging the lifetime of wireless sensor networks, sensor nodes with larger communication radius have to consume more energy to overcome wireless channel fading and path loss. Cooperative transmission allows nodes with single-antenna to cooperate on information transmission and/or reception to achieve a lower average error probability than its non-cooperative counterpart under the same transmit energy budge. In this work, we introduce the concept of cooperative transmission into sensor deployment to balance the energy consumption among sensor nodes. The key idea is to deploy an optimal number of cooperative nodes into a given area. In this way, batteries of all sensor nodes are discharged at the same rate. Extensive simulations have been conducted to evaluate the performance of the proposed scheme. Compared with non-cooperative cases, the proposed scheme with cooperative transmission improves the battery energy efficiency by 52.1%, and prolongs the lifetime of wireless sensor network by three times.

One of the design challenges of wireless sensor networks is the tradeoff between network operation time and network coverage. Recent studies reveal that the useable battery capacity drops faster at a higher discharge current in a nonlinear fashion. To take advantage of this battery current effect, in this paper we explore a new sensor node deployment scheme to prolong the entire sensor network lifetime as well as each individual sensor node. The key idea of the proposed scheme is to assign a sensor node having higher traffic load to adopt lower transmission power level. In this way, batteries of all nodes in a given area are discharged at the same current, thus they are depleted at the same time. Extensive simulations have conducted to evaluate the performance of the proposed sensor node deployment scheme. Compared with peer work on heterogeneous deployment, the useable battery capacity by using the proposed scheme can be improved by 26.67%, and the operating time per sensor node can be enhanced by 20.95%. Furthermore, the proposed deployment scheme can reduce the number of sensor nodes required to cover the given area, leading to a significant reduction of deployment cost.

Battery-powered embedded networked sensing systems become more and more pervasive with the fast-paced deployment of various remote sensing applications. How to prolong the battery operating time is one of the most challenging areas in the design and development of these sensing systems due to the fact that the battery operation is much more dynamic and complex than considered, which is derived from its application and its internal structure (multiple hardwired series/parallel cells) to produce a specific voltage and capacity. Current research on prolonging the battery operating time is mainly focusing on the low-power hardware and energy-efficient network protocol designs, and simply treats the battery as a passive two-terminal energy source. In this paper, we will propose a novel adaptive, proactive, and reconfigurable multicell battery design for supporting power-aware hardware and energy-efficient network protocols for embedded networked systems, which provides a whole new perspective to look at the energy problems of battery- powered embedded networked sensing systems. A theoretical modeling of the proposed design is provided, and simulation results show that the proposed design can significantly enhance the energy performance, especially for low voltage and low discharge current scenarios.