Behavioral cloning from human demonstrations has succeeded in programming a robot to generate fine-grained motion, but it is still challenging to learn multimodal trajectories such as with various speeds. This restricts the use of a robot dataset collected by multiusers because the different proficiency of robot operators makes the dataset have diverse distributions of speed. To tackle this issue, we develop Hierarchical Action Chunking Transformer with Vector-quantization (HACT-Vq) to efficiently learn temporal multimodality in addition to fine-grained motion. The proposed hierarchical model consists of a high-level policy to make planning for a latent subgoal and style, and a low-level policy to predict an action chunk conditioned with the latent subgoal and style. The latent subgoal and style are trained as discrete representations so that high-level policy can efficiently learn multimodal distributions of demonstrations and retrieve the mode of fast behavior. In experiments, we set up bimanual robots in both simulation and real-world environments, and collected demonstrations with various speeds. The proposed model with the quantized subgoal and style showed the highest success rates with fast imitation behavior. Our code is available at https://github.com/SamsungLabs/hierarchical-act.

This paper proposes a novel approach to address the technical challenges of stable object grasping, particularly in the context of handling tableware in a home environment. Handling tableware is particularly important, yet challenging, due to the flat nature of most tableware objects and the need to maintain a stable posture to prevent spills. To address these challenges, we present three key contributions: 1) a large-scale tableware dataset, not commonly found in the previous datasets; 2) a novel sampling method for stable grasp pose generation; and 3) a multi-modal fusion grasp network that effectively learns 6- DoF grasp pose, including flat objects. Our dataset contains over 45 million grasp poses and 1 million RGBD images captured in 800 scenes, which include randomly selected 10–18 tableware objects under 4 different lighting conditions. The grasp poses in the dataset are generated using a novel sampling method that incorporates geometric analysis to ensure stable grasping with minimal object movement. Furthermore, we design an RGBD fusion grasp network (RGBD-FGN) that can combine information from RGB and depth images considering each characteristic. Our experimental results demonstrate the superior performance of our approach over existing techniques, which is a significant contribution towards developing a multitasking home robot. Our dataset and source code can be accessed at https://github.com/SamsungLabs/RGBD-FGN.

We develop a novel passivity-based control framework of a manipulator-stage system sitting on a vertical flexible beam. We first model the vertical flexible beam by using Euler-Bernoulli theory and modal approximation while utilizing certain boundary conditions to take into account the effect of the gravity and interaction wrench between the manipulator-stage system and the beam. To separately achieve two control objectives, i.e. the joint tracking and the vibration suppression, we utilize the passive decomposition to split the dynamics into: 1) the stage and beam dynamics; and 2) its orthogonal complement, which converges to the robotic manipulator dynamics as the vibration is subdued. We further show that the linearized stage-beam dynamics is controllable and design an LQR control to stabilize the beam vibration only by using the stage motion. We also design control for the orthogonal complement dynamics in such a way that it can attain the manipulator joint tracking asymptotically as the vibration is suppressed. Experiments are also performed to verify the manipulator joint tracking while the vibration is suppressed internally via the stage motion.

We propose a novel whole-body multi-modal semiautonomous teleoperation framework for mobile manipulator systems, which consists of: 1) Motion capture and whole-body motion mapping to allow the operator to intuitively teleoperate the mobile manipulator without being constrained by the master interface while also fully exploiting whole-body dexterity; 2) Slave robot autonomous control to allow the mobile manipulator to optimally track the operator's whole-body command, while taking into account the user-slave kinematic dissimilarity (slave robot's joint limit, joint velocity limit, and singularity); and 3) Visuo-haptic-vestibular feedback with HMD (Head Mounted Display) for 3D visual information, wearable cutaneous haptic device for manipulation force feedback, and actuated chair for vestibular feedback to reduce HMD-induced motion sickness. Performance of the proposed framework is validated with simulation of a ROV (remotely operated vehicle) manipulator system and some preliminary user studies.

We present a novel vision-based teleoperation control framework for a team of an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV). Our control law allows a remote human user to teleoperate the team with some useful haptic feedback, while also ensuring the UAV-UGV coordination via the camera installed on the UAV (and seeing the UGV) and the velocity limitation of the UGV. For this, we first elucidate a geometric condition for the UAV and UGV velocities to ensure the UAV-UGV coordination by driving the image feature of the UGV to converge to a desired one on the image surface, while also guaranteeing that the UGV's velocity, which is often much slower than that of the UAV, is under a certain specified bound. The UAV is then tele-controlled to track the teleoperation command as close as possible, yet, only to the extent permissible by this geometric condition. Simulation is performed to illustrate the theory.