Humanoid robots exhibit significant potential in executing diverse human-level skills. However, current research predominantly relies on data-driven approaches that necessitate extensive training datasets to achieve robust multimodal decision-making capabilities and generalizable visuomotor control. These methods raise concerns due to the neglect of geometric reasoning in unseen scenarios and the inefficient modeling of robot-target relationships within the training data, resulting in a significant waste of training resources. To address these limitations, we present the Recurrent Geometric-prior Multimodal Policy (RGMP), an end-to-end framework that unifies geometric-semantic skill reasoning with data-efficient visuomotor control. For perception capabilities, we propose the Geometric-prior Skill Selector, which infuses geometric inductive biases into a vision language model, producing adaptive skill sequences for unseen scenes with minimal spatial common sense tuning. To achieve data-efficient robotic motion synthesis, we introduce the Adaptive Recursive Gaussian Network, which parameterizes robot-object interactions as a compact hierarchy of Gaussian processes that recursively encode multi-scale spatial relationships, yielding dexterous, data-efficient motion synthesis even from sparse demonstrations. Evaluated on both our humanoid robot and desktop robot, the RGMP framework achieves 87% task success in generalization tests and exhibits 5× greater data efficiency than the state-of-the-art model. This performance underscores its superior cross-domain generalization, paving the way for more versatile and data-efficient robotic systems.

Dynamic stability refers to the ability of a magnetic actuation system to maintain equilibrium by damping oscillations. It is typically quantified by physical scalars that serve as critical indicators of both safety and effectiveness in real-world applications. Traditional real-time assessment methods often require measuring and calibrating task-specific parameters. While feasible in principle, these approaches are hindered in practice by procedural complexity, time consumption, measurement difficulty, and uncertainties inherent to the non-contact, non-rigid nature of magnetic actuation. In this work, we eliminate the need for parameter assumptions and direct measurements, enabling a more efficient and generalizable evaluation of dynamic stability. Our approach implicitly incorporates uncertain physical properties into a learning-based framework. Specifically, we propose a temporal difference policy that predicts dynamic stability by comparing multiple time-varying sequences, thereby reducing the influence of task-specific parameters. The robustness and effectiveness are validated through extensive experiments, including baseline comparisons and online implementations across diverse magnetically actuated objects. Our findings highlight its practical advantages and pave the way for innovative control strategies in precise magnetic actuation applications.

Dual-arm humanoid robots are poised to transform industrial manufacturing automation in human-centric environments. However, unlocking this potential requires a unified framework that can simultaneously handle coupled bimanual coordination, versatile physical interaction, and safety. We introduce Passive Model Predictive Cooperative Interaction Control (P-MPCIC), a framework that co-optimizes task performance and interaction safety under a formal passivity guarantee. P-MPCIC integrates model predictive control for the bimanual subsystem within a whole-body architecture and uses a coupling matrix to enforce synchronization objectives across relative motion and force distribution. For interaction prediction, the framework incorporates a composite robot-environment model that combines parallel and series impedance dynamics, yielding a linear state-space predictor. Passivity is enforced as a constraint on the energy balance at the interaction port, preventing destabilizing energy generation from the controller. We verify the framework’s core principles through planar simulations and demonstrate its practical effectiveness on a 7-DoF dual-arm humanoid.

Recent work has demonstrated the potential of diffusion models in robot bimanual skill learning. However, existing methods ignore the learning of posture-dependent task features, which are crucial for adapting dual-arm configurations to meet specific force and velocity requirements in dexterous bimanual manipulation. To address this limitation, we propose Manipulability-Aware Diffusion Policy (ManiDP), a novel imitation learning method that not only generates plausible bimanual trajectories, but also optimizes dual-arm configurations to better satisfy posture-dependent task requirements. ManiDP achieves this by extracting bimanual manipulability from expert demonstrations and encoding the encapsulated posture features using Riemannian-based probabilistic models. These encoded posture features are then incorporated into a conditional diffusion process to guide the generation of task-compatible bimanual motion sequences. We evaluate ManiDP on six real-world bimanual tasks, where the experimental results demonstrate a 39.33% increase in average manipulation success rate and a 0.45 improvement in task compatibility compared to baseline methods. This work highlights the importance of integrating posture-relevant robotic priors into bimanual skill diffusion to enable human-like adaptability and dexterity.

With medical ultrasound becoming one of the most prevalent examination methods, robotic ultrasound systems offer the potential to simplify the scanning process and relieve professional sonographers from repetitive and tedious tasks. Despite recent advances, enabling robots to autonomously perform ultrasound examinations remains a challenge, mainly due to the difficulty in representing and generalizing professional ultrasound skills. In this paper, we present a comprehensive framework for learning autonomous ultrasound skills from freehand demonstrations in clinical settings. Our proposed framework consists of two key stages: offline learning and online adaptation. During the offline learning stage, ultrasound skills are encapsulated into a low-dimensional probabilistic model using a self-supervised architecture. The multimodal signals include ultrasound images, probe orientations, and contact forces. During the online adaptation stage, the model predicts the optimal actions either by direct regression or by using local exploration schemes. We perform clinical demonstrations with 24 volunteers and collect 120 experiences. Our benchmark includes 5 different tasks, including intra-patient, inter-patient, inter-sex, inter-age, and inter-obesity tasks. Both one-step and sequence-based predictions are achieved by using different variants of our framework. Customized and generic representation learning backbones are tested and analyzed. In conclusion, our autonomous ultrasound framework is flexible and robust, and potentially enriches the options for freehand/robotic ultrasound applications. Note to Practitioners—This paper is motivated by the problem of learning multimodal manipulation skills from human demonstrations, with a specific focus on freehand ultrasound skills. Our multimodal fusion framework is effective and compatible with some popular image representation backbones. Our adaptive methods and the variants have satisfactory prediction accuracy, with flexibility achieved by adjusting a few factors. We collect high-quality freehand demonstrations from ultrasound examinations in clinical settings. The data is openly available to facilitate the reproducibility of our work and to support the development of autonomous ultrasound strategies based on imitation learning.

Small-scale robots hold significant promise in the field of minimally invasive surgery (MIS). In this paper, we present a miniature magnetic gripper and develop a data-driven kinematic model. The gripper comprises four fingers, wherein each finger has a maximum size not exceeding 3mm, 4mm and 5.5mm in three dimensions. By integrating permanent magnets and elastic ropes as internal actuation elements into the fingers, the gripper is equipped with the capability to open-close under an external magnetic field, facilitating the manipulation of small objects in confined spaces. Modeling and analysis of the magnetic gripper are undertaken, wherein the relationship between the open angle and the external magnetic field is established. The average error between the experimentally observed open angles and the model-predicted values is 2.31°. Subsequent experiments demonstrated the necessity of the magnetic gripper model for precise manipulation, verified its excellent sensitivity to magnetic fields, and demonstrated its potential for future applications in MIS.

Internet of Medical Things (IoMT) technology has significantly helped surgeons perform complex clinical procedures, including interventional and endoscopic surgeries. However, surgeons face challenges in quickly becoming proficient with some IoMT devices. This difficulty stems from the fact that IoMT devices are not commonly used or familiar in their daily work and lives. To address this issue, we propose a novel interactive framework for IoMT, named freehand interaction, which includes visual control and haptic feedback. Our idea is to allow surgeons to operate surgical instruments, such as needles, capsules, and catheters with their bare hands and regular experience. At the same time, identical instruments in the surgical environment mirror the surgeon’s actions through visual control. The interactive forces encountered in the surgical environment are quantitatively communicated to the surgeon through the hand-held instruments. Our IoMT framework achieves both visual control and haptic feedback using electromagnetic mechanisms, ensuring mid-air freehand manipulation and contactless remote actuation. We perform different tasks in suspended, liquid, and in-vitro environments. The hand-held and mirroring instruments show high similarity and correlation, even within different electromagnetic systems and confined workspaces. Tracking accuracy, response time, and haptic forces quantified by a mechanical gauge are satisfactory. Our algorithm processes raw video streams and maintains efficiency even in scenarios with partial hand occlusion and various types of image noise. In summary, this work contributes to improving visual intuition and haptic immersion in IoMT applications.

For the SLAM system in robotics and autonomous driving, the accuracy of front-end odometry and back-end loop-closure detection determine the whole intelligent system performance. But the LiDAR-SLAM could be disturbed by current scene moving objects, resulting in drift errors and even loop-closure failure. Thus, the ability to detect and segment moving objects is essential for high-precision positioning and building a consistent map. In this paper, we address the problem of moving object segmentation from 3D LiDAR scans to improve the odometry and loop-closure accuracy of SLAM. We propose a novel 3D Sequential Moving-Object-Segmentation (3D-SeqMOS) method that can accurately segment the scene into moving and static objects, such as moving and static cars. Different from the existing projected-image method, we process the raw 3D point cloud and build a 3D convolution neural network for MOS task. In addition, to make full use of the spatio-temporal information of point cloud, we propose a point cloud residual mechanism using the spatial features of current scan and the temporal features of previous residual scans. Besides, we build a complete SLAM framework to verify the effectiveness and accuracy of 3D-SeqMOS. Experiments on SemanticKITTI dataset show that our proposed 3D-SeqMOS method can effectively detect moving objects and improve the accuracy of LiDAR odometry and loop-closure detection. The test results show our 3D-SeqMOS outperforms the existing state-of-the-art methods. We extend the proposed method to the SemanticKITTI: Moving Object Segmentation competition and achieve the 3rd in the leaderboard, showing its effectiveness.

Learning-based methods for robotic grasping have been shown to yield high performance. However, they rely on expensive-to-acquire and well-labeled datasets. In addition, how to generalize the learned grasping ability across different scenarios is still unsolved. In this paper, we present a novel grasp adaptation strategy to transfer the learned grasping ability to new domains based on visual data using a new grasp feature representation. We present a conditional generative model for visual data transformation. By leveraging the deep feature representational capacity from the well-trained grasp synthesis model, our approach utilizes feature-level contrastive representation learning and adopts adversarial learning on output space. This way we bridge the domain gap between the new domain and the training domain while keeping consistency during the adaptation process. Based on transformed input grasp data via the generator, our trained model can generalize to new domains without any fine-tuning. The proposed method is evaluated on benchmark datasets and based on real robot experiments. The results show that our approach leads to high performance in new scenarios.

Accurate estimation of the relative pose between an object and a robot hand is critical for many manipulation tasks. However, most of the existing object-in-hand pose datasets use two-finger grippers and also assume that the object remains fixed in the hand without any relative movements, which is not representative of real-world scenarios. To address this issue, a 6D object-in-hand pose dataset is proposed using a teleoperation method with an anthropomorphic Shadow Dexterous hand. Our dataset comprises RGB-D images, proprioception and tactile data, covering diverse grasping poses, finger contact states, and object occlusions. To overcome the significant hand occlusion and limited tactile sensor contact in real-world scenarios, we propose PoseFusion, a hybrid multi-modal fusion approach that integrates the information from visual and tactile perception channels. PoseFusion generates three candidate object poses from three estimators (tactile only, visual only, and visuo-tactile fusion), which are then filtered by a SelectLSTM network to select the optimal pose, avoiding inferior fusion poses resulting from modality collapse. Extensive experiments demonstrate the robustness and advantages of our framework. All data and codes are available on the project website: https://elevenjiang1.github.io/ObjectlnHand-Dataset/.

Accurate calibration of the robot hand-eye (X) and robot-world (Y) relationships is extremely important for visually-guided robotic systems, and is usually symbolized by the AX=YB equation. The existing methodologies always calibrate the X and Y matrices using the separation of the rotational and translational components, causing the error propagation and accumulation. While the simultaneous calibration solves the derived linear matrix equation by the SVD (Singular Value Decomposition) based approach, which produces unreliable results depending on the smallest singular value of the regression matrix. To this end, the work contained herein proposes a novel and generic calibration methodology for solving the AX=YB problem using the LMI-SDP (Linear Matrix Inequality and Semi-definite Programming) optimization. In this approach, the linear form of the calibration equation is retrieved by means of the Kronecker product, and formulated as an optimization problem involving the unknown variable matrices X and Y with convex constraints, in which the simultaneous solution is obtained via the LMI-SDP techniques. The results procured via the simulation analysis, accounting for the presence of noise levels and different data pairs, as well as the calibration experiments, are compared to those produced using the classical iterative method and DQ (Dual Quaternion)-based approach, thereby verifying the accuracy and efficacy of the proposed method. Note to Practitioners—The motivation behind this work stems from the simultaneous calibration issues pertaining to robot-eye and robot-workpiece coordinate relationships, that are present in vision-guided robotic systems. Considering the inaccuracy and robustness deficiencies of the existing methodologies, due to the separated calibration of the rotational and translational components, this paper proposes a generic and efficient calibration methodology to deal with the AX=YB problem, using the Kronecker product and the LMI-SDP optimization. Simulation analysis reveals that the proposed algorithms exhibit robustness under different noise levels and data pairs. Moreover, the practicability of the algorithm has also been verified via practical experiments. The average errors with 16 sets of calibration data can reach 0.0056rad in the rotational component, and 0.2529mm in the translational component. The proposed methodology can be extended to the practical applications of the coordinate calibration involving the multi-robot systems with visual sensors.

This article provides solutions for a single-leader–dual-follower teleoperation system to collaboratively transport an object. First, to regulate the direct-teleoperated follower robot (DFR), we employ a relative pose transformation algorithm, based on “refixing” the leader and DFR together, to enable that the operator can ergonomically guide DFR without requiring any specific initial position, ensuring a higher teleoperation precision at the same time. Second, to regulate the assistive follower robot (AFR), we provide an efficient technique to acquire the correct orientation to achieve holding. In addition, we devise an adjustable artificial potential field method to autonomously regulate AFR's position to a ready-to-hold position, where the operator's motion is involved. At last, based on the combination of the autoregressive model and the impedance model, we generate a reference trajectory for AFR to follow, which enables the followers to hold a rigid or a deformable object with a desired contact force. Simulations and experimental results verify the feasibility and effectiveness of the proposed method.

The potential diagnostic applications of magnet-actuated capsules have been greatly increased in recent years. For most of these potential applications, accurate position control of the capsule have been highly demanding. However, the friction between the robot and the environment as well as the drag force from the tether play a significant role during the motion control of the capsule. Moreover, these forces especially the friction force are typically hard to model beforehand. In this paper, we first designed a magnet-actuated tethered capsule robot, where the driving magnet is mounted on the end of a robotic arm. Then, we proposed a learning-based approach to model the friction force between the capsule and the environment, with the goal of increasing the control accuracy of the whole system. Finally, several real robot experiments are demonstrated to showcase the effectiveness of our proposed approach.

Different from previous work on single skill learning from human demonstrations, an incremental motor skill learning, generalization and control method based on dynamic movement primitives (DMP) and broad learning system (BLS) is proposed for extracting both ordinary skills and instant reactive skills from demonstrations, the latter of which is usually generated to avoid a sudden danger (e.g., touching a hot cup). The method is completed in three steps. First, the ordinary skills are basically learned from demonstrations in normal cases by using DMP. Then, the incremental learning idea of BLS is combined with DMP to achieve multistylistic reactive skill learning such that the forcing function of the ordinary skills will be reasonably extended into multiple stylistic functions by adding enhancement terms and updating weights of the radial basis function kernels. Finally, electromyography signals are collected from human muscles and processed to achieve stiffness factors. By using fuzzy logic system, the two kinds of skills learned are integrated and generalized in new cases such that not only start, end and scaling factors but also the environmental conditions, robot reactive strategies and impedance control factors will be generalized to lead to various reactions. To verify the effectiveness of the proposed method, an obstacle avoidance experiment that enables robots to approach destinations flexibly in various situations with barriers will be undertaken.

It is necessary to implement real-time grasp detection in robotic grasping tasks. To this end, in this paper we propose a method for effective graspable feature selection and precise classification. In a robotic grasping scene, our method can effectively select graspable rectangles and further extract useful features from them to generate a feature set. A convolutional neural network (CNN) is then developed to score and classify the elements in the feature set. Finally, we compute the desired robotic grasp pose based on the graspable feature that gets the highest score. In the test phase the proposed CNN network achieves an accuracy of 96.5% on the Cornell Grasping Dataset. In real-world grasping experiments 105 frames per second (fps) for the object's grasp detection and a grasp success rate of 89.9% have been achieved with our method.

Abstract Real-time grasp detection plays a key role in manipulation, and it is also a complex task, especially for detecting how to grasp novel objects. This paper proposes a very quick and accurate approach to detect robotic grasps. The main idea is to perform grasping of novel objects in a typical RGB-D scene view. Our goal is not to find the best grasp for every object but to obtain the local optimal grasps in candidate grasp rectangles. There are three main contributions to our detection work. Firstly, an improved graph segmentation approach is used to do objects detection and it can separate objects from the background directly and fast. Secondly, we develop a morphological image processing method to generate candidate grasp rectangles set which avoids us to search grasp rectangles globally. Finally, we train a random forest model to predict grasps and achieve an accuracy of 94.26%. The model is mainly used to score every element in our candidate grasps set and the one gets the highest score will be converted to the final grasp configuration for robots. For real-world experiments, we set up our system on a tabletop scene with multiple objects and when implementing robotic grasps, we control Baxter robot with a different inverse kinematics strategy rather than the built-in one.

Due to the lack of transparent and friendly human-robot interaction (HRI) interface, as well as various uncertainties, it is usually a challenge to remotely manipulate a robot to accomplish a complicated task. To improve the teleoperation performance, we propose a new perception mechanism by integrating a novel learning method to operate the robots in the distance. In order to enhance the perception of the teleoperation system, we utilize a surface electromyogram signal to extract the human operator's muscle activation. As a response to the changes in the external environment, as sensed through haptic and visual feedback, a human operator naturally reacts with various muscle activations. By imitating the human behaviors in task execution, not only motion trajectory but also arm stiffness adjusted by muscle activation, it is expected that the robot would be able to carry out the repetitive tasks autonomously or uncertain tasks with improved intelligence. To this end, we develop a robot learning algorithm based on probability statistics under an integrated framework of the hidden semi-Markov model (HSMM) and the Gaussian mixture method. This method is employed to obtain a generative task model based on the robot's trajectory. Then, Gaussian mixture regression based on HSMM is applied to correct the robot trajectory with the reproduced results from the learned task model. The execution procedures consist of a learning phase and a reproduction phase. To guarantee the stability, immersion, and maneuverability of the teleoperation system, a variable gain control method that involves electromyography (EMG) is introduced. Experimental results have demonstrated the effectiveness of the proposed method.

Efficient and accurate planning of fingertip grasps is essential for dexterous in-hand manipulation. In this work, we present a system for fingertip grasp planning that incrementally learns a heuristic for hand reachability and multi-fingered inverse kinematics. The system consists of an online execution module and an offline optimization module. During execution the system plans and executes fingertip grasps using Canny's grasp quality metric and a learned random forest based hand reachability heuristic. In the offline module, this heuristic is improved based on a grasping manifold that is incrementally learned from the experiences collected during execution. The system is evaluated both in simulation and on a Schunk-SDH dexterous hand mounted on a KUKA-KR5 arm. We show that, as the grasping manifold is adapted to the system's experiences, the heuristic becomes more accurate, which results in an improved performance of the execution module. The improvement is not only observed for experienced objects, but also for previously unknown objects of similar sizes.

We present a unified framework for grasp planning and in-hand grasp adaptation using visual, tactile, and proprioceptive feedback. The main objective of the proposed framework is to enable fingertip grasping by addressing problems of changed weight of the object, slippage, and external disturbances. For this purpose we introduce the Hierarchical Fingertip Space as a representation enabling optimization for both efficient grasp synthesis and online finger gaiting. Grasp synthesis is followed by a grasp adaptation step that consists of both grasp force adaptation through impedance control and regrasping/finger gaiting when the former is not sufficient. Experimental evaluation is conducted on an Allegro hand mounted on a Kuka LWR arm.

Object-level impedance control is of great importance for object-centric tasks, such as robust grasping and dexterous manipulation. Despite the recent progress on this topic, how to specify the desired object impedance for a given task remains an open issue. In this paper, we decompose the object's impedance into two complementary components-the impedance for stable grasping and impedance for object manipulation. Then, we present a method to learn the desired object's manipulation impedance (stiffness) using data obtained from human demonstration. The approach is validated in two tasks, for robust grasping of a wine glass and for inserting a bulb, using the 16 degrees of freedom Allegro Hand mounted with the SynTouch tactile sensors.

Humans have an incredible capacity to learn properties of objects by pure tactile exploration with their two hands. With robots moving into human-centred environment, tactile exploration becomes more and more important as vision may be occluded easily by obstacles or fail because of different illumination conditions. In this paper, we present our first results on bimanual compliant tactile exploration, with the goal to identify objects and grasp them. An exploration strategy is proposed to guide the motion of the two arms and fingers along the object. From this tactile exploration, a point cloud is obtained for each object. As the point cloud is intrinsically noisy and un-uniformly distributed, a filter based on Gaussian Processes is proposed to smooth the data. This data is used at runtime for object identification. Experiments on an iCub humanoid robot have been conducted to validate our approach.

To perform robust grasping, a multi-fingered robotic hand should be able to adapt its grasping configuration, i.e., how the object is grasped, to maintain the stability of the grasp. Such a change of grasp configuration is called grasp adaptation and it depends on the controller, the employed sensory feedback and the type of uncertainties inherit to the problem. This paper proposes a grasp adaptation strategy to deal with uncertainties about physical properties of objects, such as the object weight and the friction at the contact points. Based on an object-level impedance controller, a grasp stability estimator is first learned in the object frame. Once a grasp is predicted to be unstable by the stability estimator, a grasp adaptation strategy is triggered according to the similarity between the new grasp and the training examples. Experimental results demonstrate that our method improves the grasping performance on novel objects with different physical properties from those used for training.

Real time planning strategy is crucial for robots working in dynamic environments. In particular, robot grasping tasks require quick reactions in many applications such as human-robot interaction. In this paper, we propose an approach for grasp learning that enables robots to plan new grasps rapidly according to the object's position and orientation. This is achieved by taking a three-step approach. In the first step, we compute a variety of stable grasps for a given object. In the second step, we propose a strategy that learns a probability distribution of grasps based on the computed grasps. In the third step, we use the model to quickly generate grasps. We have tested the statistical method on the 9 degrees of freedom hand of the iCub humanoid robot and the 4 degrees of freedom Barrett hand. The average computation time for generating one grasp is less than 10 milliseconds. The experiments were run in Matlab on a machine with 2.8GHz processor.

Optimal grasp synthesis has traditionally been solved in two steps: determining optimal grasping points according to a specific quality criterion and then determining how to shape the hand to produce these grasping points. Generating optimal grasps depends on the position of contact points as much as the configuration of the robot hand and it would hence be desirable to solve this in a single step. This paper takes advantage of new development in non-linear optimization and formulates the problem of grasp synthesis as a single constrained optimization problem, generating grasps that are at the same time feasible for the hand's kinematics and optimal according to a force related quality measure. The approach is validated on the 9 degrees of freedom hand of the iCub humanoid robot.