Shared autonomy is the future of teleoperation as it reduces the teleoperator’s burden, enhances capabilities, and improves embodiment by offering seamless control of the robot. However, it remains rarely used, particularly with humanoid robots, as it faces numerous challenges. In this work, we introduce an innovative shared autonomy framework suitable for a wide range of robots, which we tested on a humanoid robot. This framework leverages Bayesian filtering over a Hidden Markov Model (HMM) to perform goal recognition, employing a landmark-based heuristic that minimizes computational demands while computing observation likelihoods without prior knowledge or a cost function. Once the teleoperator’s goal is identified, the robot assists according to its confidence level in the goal prediction. Assistance is provided by guiding the robot’s end-effector to reach a specified target position and orientation. In experiments with a diverse group of 10 teleoperators, conducted with video transmission delay, we achieved high accuracy in goal prediction and demonstrated significantly faster teleoperation time with shared autonomy.

Imitation learning (IL) aims to enable robots to perform tasks autonomously by observing a few human demonstrations. Recently, a variant of IL, called In-Context IL, utilized off-the-shelf large language models (LLMs) as instant policies that understand the context from a few given demonstrations to perform a new task, rather than explicitly updating network models with large-scale demonstrations. However, its reliability in the robotics domain is undermined by hallucination issues such as LLM-based instant policy, which occasionally generates poor trajectories that deviate from the given demonstrations. To alleviate this problem, we propose a new robust in-context imitation learning algorithm called the robust instant policy (RIP), which utilizes a Student’s t-regression model to be robust against the hallucinated trajectories of instant policies to allow reliable trajectory generation. Specifically, RIP generates several candidate robot trajectories to complete a given task from an LLM and aggregates them using the Student’s t-distribution, which is beneficial for ignoring outliers (i.e., hallucinations); thereby, a robust trajectory against hallucinations is generated. Our experiments, conducted in both simulated and real-world environments, show that RIP significantly outperforms state-of-the-art IL methods, with at least 26% improvement in task success rates, particularly in low-data scenarios for everyday tasks. Video results available at https://sites.google.com/view/robustinstantpolicy

When humans see a scene, they can roughly imagine the forces applied to objects based on their expe-rience and use them to handle the objects properly. This paper considers transferring this “force-visualization” ability to robots. We hypothesize that a rough force distribution (named “force map”) can be utilized for object manipulation strategies even if accurate force estimation is impossible. Based on this hypothesis, we propose a training method to predict the force map from vision. To investigate this hypothesis, we generated scenes where objects were stacked in bulk through simulation and trained a model to predict the contact force from a single image. We further applied domain randomization to make the trained model function on real images. The experimental results showed that the model trained using only synthetic images could predict approximate patterns representing the contact areas of the objects even for real images. Then, we designed a simple algorithm to plan a lifting direction using the predicted force distribution. We confirmed that using the predicted force distribution contributes to finding natural lifting directions for typical real-world scenes. Furthermore, the evaluation through simulations showed that the disturbance caused to surrounding objects was reduced by 26 % (translation displacement) and by 39 % (angular displacement) for scenes where objects were overlapping.

When generating robot motions based on instructions such as cooking recipes, ambiguity of the instructions and lack of necessary information are problematic for the robot. To solve this problem, we propose an efficient motion planning approach for a dual-arm robot by constructing a graph repre-senting a motion sequence based on a recipe consisting of verbal instructions and cooking images. A functional unit is generated based on the linguistic instructions in the recipe. Since most recipes lack the necessary information for executing the motion, we first consider extracting the information about the cooking motion like cutting from the food images of the recipe and supplementing it. In addition, to supplement the actions that humans perform unconsciously, we generate functional units for actions not explicitly mentioned in the recipe based on the current situation of the cooking process, and then connect them to the functional units generated from the recipe. Moreover, during the connection we consider the motion of the robot's arms in parallel for an efficient execution of the recipe, similar to those of a human. Through experiments, we demonstrate that for a given recipe, the proposed method can be used to generate a cooking sequence with the supplementary information needed, and executed by a dual-arm robot. The results show that the proposed method is effective and can simplify robot teaching in cooking tasks.

This article presents a novel software architecture enabling the analysis of assembly actions from fine-grained hand motions. Unlike previous works that compel humans to wear ad-hoc devices or visual markers in the human body, our approach enables users to move without additional burdens. Modules developed are able to: (i) reconstruct the 3D motions of body and hands keypoints using multi-camera systems; (ii) recognize objects manipulated by humans, and (iii) analyze the relationship between the human motions and the manipulated objects. We implement different solutions based on OpenPose and Mediapipe for body and hand keypoint detection. Additionally, we discuss the suitability of these solutions for enabling real-time data processing. We also propose a novel method using Long Short-Term Memory (LSTM) deep neural networks to analyze the relationship between the detected human motions and manipulated objects. Experimental validations show the superiority of the proposed approach against previous works based on Hidden Markov Models (HMMs).

This paper proposes a robot assembly planning method by automatically reading the graphical instruction manuals designed for humans. Essentially, the method generates an Assembly Task Sequence Graph (ATSG) by recognizing a graphical instruction manual. An ATSG is a graph describing the assembly task procedure by detecting types of parts included in the instruction images, completing the missing information automatically, and correcting the detection errors automatically. To build an ATSG, the proposed method first extracts the information of the parts contained in each image of the graphical instruction manual. Then, by using the extracted part information, it estimates the proper work motions and tools for the assembly task. After that, the method builds an ATSG by considering the relationship between the previous and following images, which makes it possible to estimate the undetected parts caused by occlusion using the information of the entire image series. Finally, by collating the total number of each part with the generated ATSG, the excess or deficiency of parts are investigated, and task procedures are removed or added according to those parts. In the experiment section, we build an ATSG using the proposed method to a graphical instruction manual for a chair and demonstrate the action sequences found in the ATSG can be performed by a dual-arm robot execution. The results show the proposed method is effective and simplifies robot teaching in automatic assembly.

Modern manufacturing strategy requires the robotic infrastructure to be able to adapt to new products or to accomplish new tasks quickly. In order to respond to this demand, teaching a robot to realize a task by demonstration has regained popularity in recent years, especially for dual-arm or humanoid robots. One of the main issues using this method is to adapt the captured motion from the human demonstration to the robot's specific kinematics and control. In this paper we present a method where the motion and grasping adaptation is tackled during the capture. We demonstrate the validity of this method with an experiment where a humanoid robot realizes an assembly previously demonstrated by a user wearing a Head Mounted Display (HMD) performing an assembly task in a virtual environment.

A rigid plate end-effector at the tip of a high-speed manipulator can remotely manipulate an object without grasping it. This paper discusses a dynamic nonprehensile manipulation strategy to rotate thin deformable objects on a rigid plate with two degrees of freedom (DOFs). The deformation of the object due to dynamic effects is exploited to produce fast and stable rotation. By varying the frequency of the rotational component of the plate's motion, we show that the dynamic behavior of the object mimics either a sliding, walking, or running gait of a biped. We introduce a model to simulate this type of system in which the object is constructed of multiple nodes that are connected by viscoelastic joint units with three DOFs. The joint's viscoelastic parameters are estimated experimentally in order to model real food. Afterward, simulation analysis is used to investigate how the object's rotational behavior and its angular velocity change with respect to the plate's motion frequency. We show how the object's behavior during rotation is analogous to bipedal sliding, walking, and running gaits and then obtain optimal plate motions leading to the maximal angular velocity of the object. We also reveal that an appropriate angular acceleration of the plate is essential for a dynamically stable and fast object's rotation. We further show that the friction coefficient that maximizes the object's angular velocity depends on its gait.

This paper discusses a nonprehensile dynamic manipulation of a deformable object, where the object is remotely manipulated on a plate attached at the tip of a bar. We have found that the object's deformation generated by dynamic effects can drastically contribute to a fast and stable object rotation. We introduce a new simulation model for a sheet-like object, where the object is constructed of multiple nodes connected by three DOFs viscoelastic joint units. We apply the model to real food after the viscoelastic parameters are estimated. Then, simulation analysis is used to show how the object's rotation behavior changes with respect to the plate's motion frequency, similar to the motion of human legs sliding, walking, and running. Finally we obtain an optimum plate motion leading to the maximal angular velocity of the object. We also reveal that an appropriate angular acceleration of the plate is essential for a dynamically stable and fast object rotation.