In the field of robotics, the process of locomotion control policy training is inherently iterative and exploratory. Practitioners often switch between multiple simulation and data analysis tools to observe robot postures and behaviors, track part movements, and compare reward data, which is a tedious process. To better understand their hurdles and requirements, we interviewed five robotics experts and analyzed representative figures from recent robotics publications to identify prevailing challenges and strategies for comparing and communicating locomotion policies. The main challenges include the lack of integrated simulation and visualization, difficulty in comparing multiple policies simultaneously, and the time-intensive process of creating polished visual presentations. Based on the insights, we introduce MRISA (Multi-Robot Interactive Simulation and Analysis Platform), an interactive tool designed to support exploratory analysis on pre-trained locomotion policies. MRISA integrates features including direct observation of one or multiple robots’ behaviors in a simulator, trajectories visualization with customized anchors, key measurement inspection in a timeline view, and key frames capturing. A user evaluation with 14 domain practitioners demonstrated that MRISA provides immediate insights, enabling practitioners to intuitively explore multiple dimensions of locomotion policies.

First-order Policy Gradient (FoPG) algorithms such as Backpropagation through Time and Analytical Policy Gradients leverage local simulation physics to accelerate policy search, significantly improving sample efficiency in robot control compared to standard model-free reinforcement learning. However, FoPG algorithms can exhibit poor learning dynamics in contact-rich tasks like locomotion. Previous approaches address this issue by alleviating contact dynamics via algorithmic or simulation innovations. In contrast, we propose guiding the policy search by learning a residual over a simple baseline policy. For quadruped locomotion, we find that the role of residual policy learning in FoPG-based training (FoPG RPL) is primarily to improve asymptotic rewards, compared to improving sample efficiency for model-free RL. Additionally, we provide insights on applying FoPG's to pixel-based local navigation, training a point-mass robot to convergence within seconds. Finally, we showcase the versatility of FoPG RPL by using it to train locomotion and perceptive navigation end-toend on a quadruped in minutes.

As home robotics gains traction, robots are increasingly integrated into households, offering companionship and assistance. Quadruped robots, particularly those resembling dogs, have emerged as popular alternatives for traditional pets. However, user feedback highlights concerns about the noise these robots generate during walking at home, particularly the loud footstep sound. To address this issue, we propose a sim-to-real based reinforcement learning (RL) approach to minimize the foot contact velocity highly related to the footstep sound. Our framework incorporates three key elements: learning varying PD gains to actively dampen and stiffen each joint, utilizing foot contact sensors, and employing curriculum learning to gradually enforce penalties on foot contact velocity. Experiments demonstrate that our learned policy achieves superior quietness compared to a RL baseline and the carefully handcrafted Sony commercial controllers. Furthermore, the trade-off between robustness and quietness is shown. This research contributes to developing quieter and more user-friendly robotic companions in home environments.

We present a reinforcement learning framework for constructing assemblies composed of rigid parts, which are commonly seen in many historical masonry buildings and bridges. Traditional construction methods for such structures often depend on dense scaffolding to stabilize their intermediate assembly steps, making the process both labor-intensive and time-consuming. This work utilizes multiple robots to collaboratively assemble structures, offering temporary support by holding parts in place without additional scaffolding. Precomputing the robotic assembly process to ensure structural stability involves a time-consuming offline process due to the combinatorial nature of its search space. However, the precomputed assembly plans may get disrupted during real-world execution due to unforeseen changes, such as setup modifications or delays in part delivery. Recomputing these plans using traditional offline methods results in significant project delays. Therefore, we propose a reinforcement learning-based approach in which a neural network is trained to efficiently generate alternative assembly plans for a given structure online, enabling adaptation to external changes. To enable effective and efficient training, we introduce three key innovations: a GPU-based stability simulator for parallelizing simulations, a novel curriculum-based training scheme to address sparse rewards during training, and a new graph neural network architecture for efficiently encoding assembly geometry. We validate our approach by training reinforcement learning agents on various assemblies and evaluating their performance on unseen assembly tasks. Furthermore, we demonstrate the effectiveness of our framework in planning multi-robot assembly processes, effectively handling disruptions in both simulation and physical environments.

We present a GPU-friendly framework for real-time implicit simulation of elastic material in the presence of frictional contacts. The integration of hyperelasticity, non-interpenetration contact, and friction in real-time simulations presents formidable nonlinear and non-smooth problems, which are highly challenging to solve. By incorporating nonlinear complementarity conditions within the local-global framework, we achieve rapid convergence in addressing these challenges. While the structure of local-global methods is not fully GPU-friendly, our proposal of a simple yet efficient solver with sparse presentation of the system inverse enables highly parallel computing while maintaining a fast convergence rate. Moreover, our novel splitting strategy for non-smooth indicators not only amplifies overall performance but also refines the complementarity preconditioner, enhancing the accuracy of frictional behavior modeling. Through extensive experimentation, the robustness of our framework in managing real-time contact scenarios, ranging from large-scale systems and extreme deformations to non-smooth contacts and precise friction interactions, has been validated. Compatible with a wide range of hyperelastic models, our approach maintains efficiency across both low and high stiffness materials. Despite its remarkable efficiency, robustness, and generality, our method is elegantly simple, with its core contributions grounded solely on standard matrix operations.

Transferring human motion skills to humanoid robots remains a significant challenge. In this study, we introduce a Wasserstein adversarial imitation learning system, allowing humanoid robots to replicate natural whole-body locomotion patterns and execute seamless transitions by mimicking human motions. First, we present a unified primitive-skeleton motion retargeting to mitigate morphological differences between arbitrary human demonstrators and humanoid robots. An adversarial critic component is integrated with Reinforcement Learning (RL) to guide the control policy to produce behaviors aligned with the data distribution of mixed reference motions. Additionally, we employ a specific Integral Probabilistic Metric (IPM), namely the Wasserstein-1 distance with a novel soft boundary constraint to stabilize the training process and prevent model collapse. Our system is evaluated on a full-sized humanoid JAXON in the simulator. The resulting control policy demonstrates a wide range of locomotion patterns, including standing, push-recovery, squat walking, humanlike straight-leg walking, and dynamic running. Notably, even in the absence of transition motions in the demonstration dataset, the robot showcases an emerging ability to transit naturally between distinct locomotion patterns as desired speed changes.

Humanoid robots are versatile platforms with the potential for multiple locomotion skills. However, this contact-switched system with only two contact feet is fragile to keep balance in many scenarios. Inspired by birds combining legs and wings, we propose the novel hybrid locomotion behavior for the humanoid robots with the aid of a thruster suit. To fully leverage their agility while guaranteeing efficient computation, we combine the neural controller based on reinforcement learning to handle the complexity of the highly non-linear system and the optimization-based controller to explicitly handle the constraint conditions of the safety-critical thruster module. Our learning framework is demonstrated on several thruster-aided humanoid platforms with hybrid walking and even dynamic locomotion skills. To our best knowledge, it is the first work that, 1. demonstrates agile hybrid whole-body locomotion skills on the thruster-aided humanoid robot; 2. achieves hybrid locomotion under the reinforcement learning settings.

Quadrupedal robots are skillful at locomotion tasks while lacking manipulation skills, not to mention dexterous manipulation abilities. Inspired by the animal behavior and the duality between multi-legged locomotion and multi-fingered manipulation, we showcase a circus ball challenge on a quadrupedal robot, ANYmal. We employ a model-free reinforcement learning approach to train a deep policy that enables the robot to balance and manipulate a light-weight ball robustly using its limbs without any contact measurement sensor. The policy is trained in the simulation, in which we randomize many physical properties with additive noise and inject random disturbance force during manipulation, and achieves zero-shot deployment on the real robot without any adjustment. In the hardware experiments, dynamic performance is achieved with a maximum rotation speed of 15 °/s, and robust recovery is showcased under external poking. To our best knowledge, it is the first work that demonstrates the dexterous dynamic manipulation on a real quadrupedal robot.

Carrying payloads in air is a major mission for multirotor aerial robot. However, the presence of payloads on multirotor aerial robot has a risk of degrading the performance of the flight controller. This concern becomes obvious especially when carrying objects not securely attached to the body or performing aerial manipulation. Therefore, controller with the ability to adapt itself to the effects of payloads on flight stability is needed. This paper proposes a novel nonlinear multiple-input and multiple-output (MIMO) model reference adaptive control (MRAC) system for attitude control of multirotor aerial robots which can dynamically compensate change in the position of center of gravity and inertia caused by payloads. Stability and robustness of the controller are experimentally confirmed in quadrotor and transformable multirotor, and experiments modeling practical applications are conducted for each aerial robot system, proving the utility of the controller.

Stable flight control under ceilings is difficult for multirotor Unmanned Aerial Vehicles (UAVs). The wake interaction between rotors and upper walls, called the "ceiling effect", causes an increase of rotor thrust. As a result of the thrust increase, multi-rotors are drawn upward abruptly and collide with ceilings. In previous work, several thrust models of the ceiling effect have been proposed for stable flight under ceilings, assuming that the airflow around rotors is in steady states. However, the airflow around rotors in vertical flight is not in steady states and each thrust model in previous work is skillfully determined based on large amounts of precise experimental data. In this paper, we introduce an aerodynamics-based thrust model and a stable control method under ceilings. This model is derived from the momentum theory and the relationship between vertical climbing/descending rates of rotors and an induced velocity. To confirm our proposed model, we collect thrust data at various vertical rates in flight. In addition, we use only onboard sensors to estimate selfstate for structural inspections. Consequently, we reveal that the proposed model is consistent with the experimental results. Based on an aerodynamic model, we need not collect large amounts of precise experimental data to realize stable flight. Furthermore, the vertical flight tests under ceilings demonstrate that our in-unsteady-state-model-based controller outperforms the conventional steady-state ones.

Regrasping is one of the most common and important manipulation skills used in our daily life. However, aerial regrasping has not been seriously investigated yet, since most of the aerial manipulator lacks dexterous manipulation abilities except for the basic pick-and-place. In this paper, we focus on pivoting a long box, which is one of the most classical problems among regrasping researches, using a transformable multilink aerial robot. First, we improve our previous controller by compensating for the external wrench. Second, we optimize the joints configuration of our transformable multilink drone for stable grasping form under the constraints of thrust force and joints effort. Third, we sequentially optimize the grasping force in the pivoting process. The optimization goal is to generate continous grasping force whilst maximizing the friction force in case of the downwash, which would influence the grasped object and is difficult to model. Fourth, we develop the impedance controller in joint space and admittance controller in task space. As far as we know, it is the first research to achieve extrinsic contact-aware regrasping task on aerial robots.

External wrench estimation is very helpful for aerial exploration and manipulation tasks. During the exploration, there might be unseen obstacles to cause dangerous collisions. The estimation of the external force and torque is also beneficial in aerial manipulation tasks. In this paper, we present a framework of estimating the external wrench for the aerial multilink robot based on the onboard inertial measurement unit (IMU) sensors, joints state and robot dynamic models. Compared to the conventional multirotor robot, the center of mass (CoM) is always changing when the robot transforms. The sensor could not be attached to CoM to observe the acceleration data. Consequently, we present a novel method by applying a distributed IMU system to estimate the CoM linear and angular accelerations for the external wrench estimation. With the help of the robot model, the position of the contact point could be estimated, which is useful in exploring tasks to safely interact with the physical world. We design the contact-aided navigation strategy and computationally efficient motion primitives library to help our robot react to the unexpected collision. We experimentally validate our framework with a two-dimensional multilink aerial robot to show the results of external wrench estimator and its further applications2.2Experiment video: https://youtu.be/R-WDReLnWWI

We present a novel transformable multilinked aerial robot which consists of link modules with 1 DoF thrust vectoring mechanism. Commonly used UAV is underactuated due to its simplicity and high flight duration, but can not control the position and orientation independently. To overcome this problem, fully actuated multirotor aerial robots have been developed. In our previous work we developed fully actuated multilinked aerial robot which can transform in the air. However, the transformation range was limited because of a singularity problem. In this paper we propose a new design of link module with a tilted rotor and 1 DoF thrust vectoring joint which enables to avoid singularity forms and keep the flight stable during transformation. We describe modeling and control for the fully actuated multilinked multirotor. Then we propose a transformation planning method utilizing the 1 DoF thrust vectoring angle with consideration of guaranteed minimum force/torque. Finally we perform an aerial transformation experiment with a real platform to demonstrate the feasibility of our proposed design and methods.

Transformable aerial robots are favorable in aerial manipulation tasks for their flexible ability to change configuration during the flight. By assuming robot keeping in the mild motion, the previous researches sacrifice aerial agility to simplify the complex non-linear system into a single rigid body with a linear controller. In this paper, we present a framework towards agile swing motion for the transformable multi-links aerial robot. We introduce a computational-efficient non-linear model predictive controller and joints motion primitive frame-work to achieve agile transforming motions and validate with a novel robot named HYRURS-X. Finally, we implement our framework under a table tennis task to validate the online and agile performance.Supplementary MaterialThis paper is accompanied by a experiment video: http://www.jsk.t.u-tokyo.ac.jp/%7eshifan/paper/iros19/video.mp4.

In this paper, we present the achievement of aerial grasping by shape adaptive transformation to the object shape, using a novel transformable aerial robot called HALO: Horizontal Plane Transformable Aerial Robot with Closed-loop Multilinks Structure. Aerial manipulation is an active research area and using multiple aerial robots is an effective solution for the large size object. However the cooperation is considered that there are some difficulties such as the synchronized flight control and collision with each other. Then, we focus on the transformable aerial robot with two-dimensional multilinks proposed in our previous works, which can transform to the suitable form for the target object and grasp it. However the transformable aerial robot with the serial-link structure could not achieve stable flight in terms of horizontal position and yaw control due to the low rigidity and large inertia in the case of more than 4 links. Thus, first we construct a novel type of multilinks with closed-loop structure to avoid the deformation and a new link module with a tilted propeller for fully-actuated control. Second, we describe transformation method with closed-loop multilinks. Third, we present the optimization planning method for the multilinks form to be adaptive to the two-dimensional shape of the target object. Finally, we present experimental results to demonstrate the feasibility of closed-loop aerial transformation and aerial grasping for the large size object.

In this paper, we introduce the achievement of the flight motion to pass through small opening by the multilinked and transformable aerial robot. Previous works about such motion are based on under-actuated multirotors, indicating that aggressive maneuvering is necessary condition. This involves two crucial problems: i) enough free space for deceleration is necessary, otherwise the robot would collide with unknown obstacle after exiting opening; ii) the multirotor can not traverse the openings that are smaller than the robot body. The proposed transformable aerial robot in our work can solve these problems, since the multilinked model can not only guarantee the near-hover condition during the whole motion sequence, but also slowly traverse relative small openings by changing its form like a snake. We first propose an improved dynamics derivation and flight control method for this multilinked aerial robot based on our previous work. Then, we present the path planning method which takes the flight stability in the near-hover condition into account. Finally we demonstrate the experimental results of the motion to pass through a horizontal and small opening which also involves the borders (the floor and the ceiling).

In this paper, we show the achievement of a transformable aerial robot with internal communication system for multiple objects transportation. As it is not easy to make the flight endurance of an aerial robot longer, we study the problem to transport multiple objects at the same time to improve the efficiency of transportation. However, for conventional aerial robots, multiple objects transportation is difficult because the CoG position changes when the number of grasped objects changes, resulting in the instability of the flight. Therefore, to solve this problem, we focus on the multirotor with two-dimensional multilinks proposed in our previous work, which possesses the ability to modify the CoG position actively and can keep the flight stable. First, we introduce the hardware platform including the structure of link module and internal communication system to achieve the extensibility in terms of the link number. We then propose a method to find the optimal form for the multilinks based on the flight stability. Finally, we present experimental results which include aerial transformation and multiple objects transportation.

In recent years, applications of visual tracking algorithms has seen a substantial growth with deployments in intelligent robots such as drones for human tracking. The algorithms for such tasks has to be efficient in terms of computational cost while been robust, accurate and fast. Object tracking algorithms based on handcrafted heuristics and constraints are widely used in uav applications. The handcrafted heuristics are mostly implemented for task-oriented applications which limits the extensions in uav's capability beyond the predefined functions. This paper considers the challenges of tracking and landing an autonomous uav on a speed high moving target, and presents a visual tracking algorithm that integrates correlation filters with deep comparison network for real-time tracking with state-of-the-art accuracy. The method first tracks the target upto translation using an online learnt model via local search technique. The changes in scale is estimated by a deep comparison network (DCN) instead of the commonly used pyramidal approach. In a single network evaluation, DCN can estimate the changes in scale as well as compensate the drifting of the tracker by refining the object region estimated by the correlation filters. The network is end-to-end trained which attempts to learn a powerful matching function for object localization using a known template. Generally, the integrated framework can be viewed as coarse-to-fine level motion estimation. Moreover, the framework can redetect the lost target without a need for a separate detector.