This paper presents a system integrating physical human-robot interaction in a dual-arm object manipulation task using multi-modal robot skin. The skin covers the end-effectors, enabling distance and active force control for aligning and grasping unknown objects, and extends over the entire arms, allowing multi-contact interaction by a user to guide the manipulation motion. Additionally, the skin patches on the non-contact area of the end-effectors function as tactile buttons, with gestures like short taps or long presses designed to switch control objectives. The system integrates two main controllers: a direct wrench controller regulating the grasp force and a Cartesian impedance controller governing the end-effector motion. Interaction wrenches on the arm patches are projected into the end-effector frame or object frame for pre-grasp adjustments or generating manipulation motion. Our method is validated on a dual-arm mobile manipulator equipped with robotic skin, demonstrating smooth motion in response to tactile interaction while consistently maintaining a stable grasp. The system's robustness enables it to handle objects of various geometries without prior knowledge. The system's capability is further highlighted through a practical application where, with human guidance, the robot successfully hangs a toolbox on two nails.

Driven by the need for rapid and reliable heavy payload transport in logistics and manufacturing, researchers are increasingly exploring early applications of humanoid robotics in these domains. Although bipedal locomotion excels on challenging terrain, wheeled modes of transportation remain significantly more energy-efficient on flat surfaces. In this work, we develop a control system that enables a humanoid robot to achieve fast transportation - by riding a two-wheeled hoverboard - and robust heavy payload handling through whole-body grasping, where the robot uses its chest and arms to stabilize bulky objects. Our approach models payload-induced disturbances using a Linear Inverted Pendulum Mode extended with external forces and leverages tactile feedback from an integrated robotic skin to estimate the payload’s weight and center of mass. Feeding these estimates into the hoverboard controller reduces drift and enhances stability. Experimental evaluations on a full-sized real humanoid robot show that our system can withstand strong disturbances and autonomously navigate to deliver payloads of up to 20 kg.

Robots that autonomously operate in human living environments require the ability to adapt to unpredictable changes and flexibly handle a variety of tasks. Particularly, coordinated bimanual motions are essential for enabling tasks that are difficult with just one hand, such as grasping bulky objects, transporting heavy loads, and precision work. Traditional methods of generating robot motions typically involve executing pre-programmed motions, making it challenging to adapt to complex and unpredictable environmental changes. To address this issue, our research focuses on generating diverse motions that can flexibly adapt to environmental changes based on Deep Predictive Learning from a small amount of real-world data. Previous Deep Predictive Learning models have generated the motions of a robot’s left and right arms by a single LSTM, making it difficult to operate them independently. Therefore, we propose a new Hierarchical Deep Predictive Learning model specialized for generating coordinated bimanual motions. This model comprises three components: a Left-LSTM, which learns the body and visual information on the robot’s left side, a Right-LSTM that performs a similar function for the right side, and a Union-LSTM which integrates this information at a higher level. To verify the effectiveness of the proposed model, we conducted bimanual grasping experiments with multiple different objects using two different robots. The experimental results showed that independent of hardware, our model demonstrated a higher success rate compared to the traditional approach, indicating its enhanced capability in coordinating bimanual motions.