Determining the reasonable grasping order and reducing the interference to surrounding objects are critical for robotic grasping under mutually-stacked scenes. However, existing manipulation relationship reasoning methods generally predict object relationships from detection perspective, and require dense evaluation of all object pairs, leading to restricted reasoning accuracy caused by detection bias and relationship class imbalance. To solve this problem, we propose an EFficient Visual Manipulation Relationship Reasoning Network, called EF-VMR2N, with relationship attention and sparse graph. Specifically, relationship attention constructs two new attention mechanisms to holistically leverage the correlation characteristics between visual features and semantic information among object pairs by highlighting reasoning-specific features. Sparse graph infers the mutual relationships within the scene using fixed number of strongly-correlated triplet sets to improve the efficiency and accuracy of object pair evaluation. Extensive experiments on the VMRD and REGRAD datasets both show that the proposed EF-VMR2N achieves SOTA performance in terms of four evaluation metrics (mAP, OR, OP and IA), and compared with object detection, the improvements on manipulation reasoning are more remarkable. The stacked objects grasping under real-world scenarios further proves the effectiveness of the proposed method. Video: https://github.com/LiMing336/VideoDemonstration.

Most existing robotic fruit harvesting systems rely on mechanically complex end-effectors that lack delicate tactile dexterity and fail to replicate the sophisticated sensory-motor coordination of human pickers. To enable safe, reliable, and efficient automated harvesting of delicate fruits, this paper presents a novel bio-inspired humanoid TacTip gripper for precision strawberry harvesting. Inspired by human thumbfinger opposition, we design an asymmetric TacTip gripper that integrates a Thumb tactile sensor with a built-in fingernail for stem cutting and a supporting Pillow tactile sensor. We further develop a hybrid SSIM-CNN perception framework that fuses real-time structural similarity index measure (SSIM) from both fingertips with convolutional neural network (CNN) features, enabling precise closed-loop grasp-state detection and gentle force adjustment. In addition, a segmented dynamic system (DS) motion planner decomposes the harvesting task into approach, cut, and place phases, generating reactive, smooth, and biologically plausible trajectories. Experimental results on both laboratory setups and live potted strawberry plants demonstrate reliable full-cycle harvesting with high success rates and minimal fruit damage. The proposed system provides a practical and effective solution for automated delicate fruit harvesting.

To meet the multi-dimensional modulation requirements of enabling robots to simultaneously achieve compliant adaptation to human time-varying motions and precise force tracking in unknown environments under physical human-robot-environment interaction (pHREI) scenarios, this paper proposed a fuzzy adaptive admittance control (FAAC) strategy that integrates disturbance observation and unified performance guarantees within a force-position dual-loop structure. In the force control outer loop, an extended state observer (ESO) is constructed to compensate for disturbances arising from uncertain human intentions and unstructured environmental geometry, while a PID-based admittance enhancement scheme is introduced to improve dynamic responsiveness. In the position control inner loop, a novel barrier function is embedded into the backstepping framework to systematically regulate the convergence rate, transient overshoot, steady-state accuracy, and global performance of the trajectory tracking error. Moreover, a fuzzy logic system (FLS) is employed to approximate the lumped model uncertainty, which in turn strengthens the robustness of the control algorithm. The asymptotic stability of the closed-loop system is rigorously established via the Lyapunov analysis criterion. Experiments involving trajectory tracking, planar cutting, and curved-surface carving conducted on an actual robotic platform demonstrate that the proposed method not only realizes superior force-position coordinated control, but also accommodates flexibility in physical human-robot interaction (pHRI) and smoothness in physical robot-environment interaction (pREI).

Reliable 6D pose estimation is crucial for robotic tasks but presents significant challenges in environments with occlusion. Recent approaches tend to directly predict pose parameters of object with deep neural networks, lacking the modeling ability of non-adjacent and complex relationships of surface points in occluded scenarios. To solve this problem, we propose a novel occlusion-aware 6D pose estimation framework, which uses depth-guided graph neural network (GNN) to model potential relationships from RGBD input. Two semantic information, which are mask and binary code of object, are adaptively fused to extract 2D-3D correspondence related features in an effective manner. Both enhanced graph features and fused semantic information contribute to the performance improvement of pose estimation with occlusion. Extensive experiments indicate that our approach outperforms comparative methods by 1.2% and 1.9% on LMO and YCBV datasets (up to 30% for certain objects) and its validity is also verified under real-world pose estimation test.

Due to non-destructive testing (NDT) techniques being both expensive and inconvenient in dynamic detection scenarios, innovative alternatives are urgently needed to address cost-efficiency and deployment challenges. We first design TacRoller, a tactile sensor roller for automated characterization of surface defects in composite materials, to address the dilemma. It collects tactile images of defects on the composite’s plies by capturing changes caused by deformation of the outer elastomer through the internal camera. It reduces the cost of inspection by 80% to 90% compared to NDT equipment like radiographic testing while ensuring detection efficiency. It takes 58.86 seconds to complete a 35 cm×18 cm × 0.5 mm dry-woven fabric. Moreover, we collect a total of 2,744 images of samples of dry-woven fabric unidirectional prepreg through TacRoller to form a dataset, including wrinkles, foreign objects and debris (FODs), broken fibre, voids and healthy textures. Subsequently, we propose a multi-order gated aggregation (MOGA)-U-Net to tackle critical challenges of noise sensitivity and multi-scale defect recognition in tactile images, enabling robust segmentation and multi-category classification tasks. The results show that the MOGA-U-Net achieves a test dice coefficient of 76.0% and classification accuracy of 98.9%, outperforming DeepLabV3 and other benchmarks. By providing a scalable and effective NDT substitute, our system realises autonomous defect identification and classification on composites surface, thus improving quality control in the production of composites.

This paper presents a framework for learning and transferring robot tool-use skills based on Dynamic Movement Primitives (DMPs) for robot fine manipulation. DMPs and their enhanced methods are employed to acquire a specific tool-use skill applicable to tools with similar sizes, shapes, and uses. However, the acquired skills may not be transferable to other scenarios and tools with variations. The new framework introduces two new types of skills based on DMPs: Object Operating (O2) skill and Tool Flipping (TF) skill. The O2 skill enables robots to handle tools for manipulating objects to achieve desired effects. The learning process for the O2 skill considers limitations imposed by tools and the environment during human demonstrations. Distinguishing between whether constraints can be modelled or not, we propose both a model-based and a constraint-based method to separate a constraint-irrelevant (CI) skill and the constrained conditions. The CI skill is generalized using a novel method called constrained -DMP lite, enabling adaptation to new tasks with special tools. The TF skill addresses situations where tools must generate an action to alter contacting positions on both objects and tools while avoiding conflicts during movement. Finally, the TF and O2 skills are generalized to be applied in creating a continuous action chain. We conduct several experiments to compare and analyze the advantages and disadvantages of the proposed methods with other approaches in terms of generalizability and calculation complexity.Note to Practitioners—Strengthening robot tool-use ability has been a hot research topic in recent years because these tools can extend the reachability and enhance the flexibility of robots. The previous research on DMPs has been utilized for learning tool-use skills. However, the learned skills few considered the tools’ special use regulations, therefore the skill of using a tool is hard to transfer to another tool-use case. This paper explores tool-use skill learning and transfer between different tools by developing a framework based on the DMPs for this problem. The framework consists of two kinds of skills: O2 skill and TF skill with different purposes as well as a series of newly developed algorithms, such as constrained -DMP lite, a model-based and a constraint-based CI skill learning methods. These methods can separate the constraints from human demonstrations of using tools to achieve a CI skill and generalize the CI skill according to the constraints generated from a new tool-use manipulation task. We verify the effectiveness of the proposed framework through some typical tool-use experiments, including pushing objects, cutting and obstacle avoidance in actuality. The development of this framework can be used in industrial and house working scenarios.

Dynamic motion primitives (DMPs) have been widely used in robotics and automation systems because of their rapid deployment capability. Previous research has concentrated on extending coupled dynamic movement primitives (CDMPs) to manipulate rigid objects using a dual-arm robot. However, manipulating deformable objects may fail due to issues with the workspace, manipulability caused by the robot’s layout, and the uncertain dynamics of deformable objects. This research proposes a unique system that combines Learning from Demonstration (LfD) to optimise robot layouts based on human experience and robot manipulability. It also employs a modified CDMPs (MCDMPs) method to manipulate deformable objects. The MCDMPs include a new term to ensure that the deformed object can change its configuration during manipulation. Furthermore, another term is introduced to track the desired trajectory of the deformed object, which is crucial for transporting tasks. We conducted simulations using a mass-spring-damper system for cooperative manipulation to validate the proposed approach. We also employed a dual-arm robot platform to transport a deformed ball with disturbance. The simulation and experimental findings indicate that our method performs well in trajectory tracking and configuration change. Note to Practitioners—This research paper addresses the need for faster deployment of dual-arm robots in various applications, such as industrial and services. We must consider the robot’s installation layout and task programming for rapid deployment. To optimise layouts for tasks quickly, we introduce an optimisation framework which considers the comprehensive performance of dual arms and uses demonstration sampling and learning generalisation. To address the manipulation of deformable objects by dual-arm robots, we use an MCDMPs method that employs the barrier Lyapunov function (BLF) to track changed configurations. Additionally, we introduce a term for reference trajectory tracking component for the manipulated object.

Teleoperation and Learning from Demonstration (LfD) are complementary paradigms for robotic control, yet their synergistic potential remains underexplored. This work proposes a unified framework that dynamically adjusts teleoperation precision using task-specific priors from LfD while optimizing demonstration learning through teleoperation scaling. By analyzing human operator signals (e.g., muscle activity), our method autonomously scales robot movements to balance the precision requirements and execution efficiency during teleoperation. Conversely, teleoperation data informs the adaptive arrangement of motion primitives in LfD, improving trajectory accuracy in critical task phases. Experiments show that the proposed method can improve the efficiency of teleoperation tasks and the accuracy of task learning. This bidirectional synergy offers a practical pathway to enhance both human-robot interaction and skill learning, particularly in applications demanding variable precision, such as assembly and surgical robotics.

In this paper, we present the TacShade: a newly designed 3D-printed soft optical tactile sensor. The sensor is developed for shape reconstruction under the inspiration of sketch drawing that uses the density of sketch lines to draw light and shadow, resulting in the creation of a 3D-view effect. TacShade, building upon the strengths of the TacTip, a single-camera tactile sensor of large in-depth deformation and being sensitive to edge and surface following, improves the structure in that the markers are distributed within the gap of papillae pins. Variations in light, dark and grey effects can be generated inside the sensor under the external contact interactions. The contours of the contacting objects are outlined by white markers, while the contact depth characteristics can be indirectly obtained from the distribution of black pins and white markers, creating a 2.5D visualization. Based on the imaging effect, we improve the Shape from Shading (SFS) algorithm to process tactile images, enabling a coarse but fast reconstruction for the contact objects. Two experiments are performed. The first verifies TacShade’s ability to reconstruct the shape of the contact objects through one image for object distinction. The second experiment shows the shape reconstruction capability of TacShade for a large panel with ridged patterns based on the location of robots and image splicing technology.

Neuroscience research has demonstrated the sig-nificance of modulating stiffness during human task performance. Similarly, endowing robots with such capability is expected. However, existing methods for robot teleoperation require operators to simultaneously control position and stiffness, resulting in high workload and task inefficiency. On the other hand, learning from demonstration (LfD) offers a feasible approach for autonomously generating stiffness. Therefore, this paper proposes a robot teleoperation system that combines the advantages of teleoperation and LfD. Teleoperation enables precise positioning guided by human operators, while LfD can transfer human stiffness skills to robots. A teleoperation-oriented stiffness-adaptive Gaussian Mixture Model/Gaussian Mixture Regression method is proposed to learn human multi-dimensional stiffness and reproduce robot stiffness on a Riemannian manifold. To enhance generalization and cooperate with teleoperation, reference points and position-driven output are introduced. Furthermore, a teleoperation strategy for both the single-leader-single-follower configuration and the single-leader-dual-follower configuration are designed, which allows operators to control either one or two robot arms with a single leader device. Finally, the effectiveness of our method is verified through a plugging-in task and a continuous flipping task, demonstrating that the proposed system is capable of performing tasks that demand high positioning accuracy and stiffness adjustment. A supplementary video for this paper is available in GitHub**https://github.com/setowenGit/TOSA-GMM-GMR-Video.

In this paper, a distributed observer-based prescribed performance control method is proposed for using a multi-robot teleoperation system to manipulate a common deformable object. To achieve a stable position-tracking effect and realize the desired cooperative operational performance, we first define a new hybrid error matrix for both the relative distances and absolute positions of robots and then decompose the matrix into two new error terms for cooperative and independent robot control. Then, we improve the Kelvin-Voigt (K-V) contact model based on the new error terms. Because the center position and deformation of the object cannot be measured, the object dynamics are then expressed by the relative distances of robots and an equivalent impedance term. Each robot incorporates an observer to estimate contact force and object dynamics based on its own measurements. To address the position errors caused by biases in force estimation and realize the position-tracking effect of each robot, we improve the barrier Lyapunov functions (BLFs) by incorporating the errors into system control. which allows us to achieve a predefined position-tracking effect. We conduct an experiment to verify the proposed controller’s ability in a dual-telerobot cooperative manipulation task, even when the object is subjected to unknown disturbances.Note to Practitioners—This article is inspired by the limitations of multi-telerobot manipulation with a deformable object, where the deformation of the object cannot be measured directly. Meanwhile, force sensors, especially 6-axis force sensors, are very expensive. To realize the purpose that objects manipulated by multiple robots match the same state as operated on the leader side, we propose an object-centric teleoperation framework based on the estimates of contact forces and object dynamics and the improved barrier Lyapunov functions (BLFs). This framework contributes to two aspects in practice: 1) propose a control diagram for deformable object co-teleoperation of multi-robots for unmeasurable object’s centre position and deformation; 2) propose an improved BLFs controller based on the estimation of contact force and robot dynamics. The estimation errors are considered and transferred using an equivalent impedance to be integrated into the Lyapunov function to minimize both force and motion-tracking errors. The experimental results verify the effectiveness of the proposed method. The developed framework can be used in industrial applications with a similar scenario.

Tactile sensors can be used for motion detection and object perception in robot manipulation. The contact detection within the camera's visual inspection area has been well-developed, but perception outside the field of view of the camera is overlooked. In this paper, we present a new tendon-linked tactile sensor, MechTac, to achieve perceptions inside and outside the field of view. The MechTac is an evolution of the typical TacTip sensor with the following two advantages. 1) The ability to provide perception outside the field of view. This is achieved by using a network of braided tendons to transfer deformation from the blind perception regions (TacSide) to the visual areas (TacTip). 2) The tactility of the TacSide and TacTip is reflected by the movements of multiple papillae pins and visible markers on the pin tips on the inner surface of the TacTip. The pins and markers are differentially sensitive to various touch features, which is similar to the differentiated perceptual ability of humans. TacTip is more sensitive to small touches, corresponding to the fingertip, while the TacSide is less sensitive but has a larger perceptual area, corresponding to the middle part of the finger. Moreover, we propose a new deep learning method to decompose the mixed information affected by the TacSide and the TacTip. A modified DenseNet121 was specifically designed for object perception at the TacTip and localization at the TacSide. The experimental results show that prediction accuracy reaches about 98% for object perception or localization and over 99% for the case requiring two functions.

Due to changes in the environment and errors that occurred during skill initialization, the robot's operational skills should be modified to adapt to new tasks. As such, skills learned by the methods with fixed features, such as the classical Dynamical Movement Primitive (DMP), are difficult to use when the using cases are significantly different from the demonstrations. In this work, we propose an incremental robot skill learning and generalization framework including an incremental DMP (IDMP) for robot trajectory learning and an adaptive neural network (NN) control method, which are incrementally updated to enable robots to adapt to new cases. IDMP uses multi-mapping feature vectors to rebuild the forcing function of DMP, which are extended based on the original feature vector. In order to maintain the original skills and represent skill changes in a new task, the new feature vector consists of three parts with different usages. Therefore, the trajectories are gradually changed by expanding the feature and weight vectors, and all transition states are also easily recovered. Then, an adaptive NN controller with performance constraints is proposed to compensate dynamics errors and changed trajectories after using the IDMP. The new controller is also incrementally updated and can accumulate and reuse the learned knowledge to improve the learning efficiency. Compared with other methods, the proposed framework achieves higher tracking accuracy, realizes incremental skill learning and modification, achieves multiple stylistic skills, and is used for obstacle avoidance with different heights, which are verified in three comparative experiments.

Cellularization is a promising architecture for spacecraft construction and on-oribt service. Functional adjustment of a cellularized system can be easily achieved through structural reconfiguration. Cellular space robot is one of the cellularized systems for on-orbit service of heterogeneous targets. As an example of on-orbit service, the attitude takeover control problem of target spacecraft by cellular space robot with redundant cells is studied in this paper. A distributed impedance control strategy is proposed for cooperative control of redundant cells in cellular space robot. Each cell is driven by its local impedance controller. Simultaneously, impedance parameters of the cells are adjusted though distributed data interaction. The attitude of the target spacecraft can be settled under the joint control of the cells and energy balance of the cells can be realized through distributed impedance parameters adjustment.

In this paper, we create a new tendon-connected multi-functional optical tactile sensor, MechTac, for object perception in the field of view (TacTip) and location of touching points in the blind area of vision (TacSide). In a multi-point touch task, the information of the TacSide and the TacTip are overlapped to commonly affect the distribution of papillae pins on the TacTip. Since the effects of TacSide are much less obvious to those affected on the TacTip, a perceiving out-of-view neural network (O2VNet) is created to separate the mixed information with unequal affection. To reduce the dependence of the O2VNet on the grayscale information of the image, we create one new binarized convolutional (BConv) layer in front of the backbone of the O2VNet. The O2VNet can not only achieve real-time temporal sequence prediction (34 ms per image), but also attain the average classification accuracy of 99.06%. The experimental results show that the O2VNet can hold a high classification accuracy even facing the image contrast changes.

At present, there are about 70 million deaf people using sign language in the world, but for most normal people, it is difficult to understand the meaning of the sign language expression. Therefore, it is of great importance to explore the ways of recognising the sign language. In this paper, we propose a dynamic sign language recognition method based on the modified long short-term memory (LSTM) model. Firstly, we use Leap Motion to collect the features of Chinese Sign Language (CSL). LSTM has a good effect in processing time series data, but the parameters of its hidden layer are shared, making it important information lost when dealing with long time series. The attention mechanism can give different attention weights to different features according to the correlation between the input data and output data, so as to enhance the model’s attention to key information. Therefore, we combine LSTM with attention mechanism for dynamic sign language recognition. Experimental results show that the recognition accuracy of the modified LSTM model is 99.55%, which is higher than that of LSTM model. Finally, we developed a sign language human-computer interaction system, which verifies the real-time performance and effectiveness of the method proposed in this paper.

Different from previous work on single skill learning from human demonstrations, an incremental motor skill learning, generalization and control method based on dynamic movement primitives (DMP) and broad learning system (BLS) is proposed for extracting both ordinary skills and instant reactive skills from demonstrations, the latter of which is usually generated to avoid a sudden danger (e.g., touching a hot cup). The method is completed in three steps. First, the ordinary skills are basically learned from demonstrations in normal cases by using DMP. Then, the incremental learning idea of BLS is combined with DMP to achieve multistylistic reactive skill learning such that the forcing function of the ordinary skills will be reasonably extended into multiple stylistic functions by adding enhancement terms and updating weights of the radial basis function kernels. Finally, electromyography signals are collected from human muscles and processed to achieve stiffness factors. By using fuzzy logic system, the two kinds of skills learned are integrated and generalized in new cases such that not only start, end and scaling factors but also the environmental conditions, robot reactive strategies and impedance control factors will be generalized to lead to various reactions. To verify the effectiveness of the proposed method, an obstacle avoidance experiment that enables robots to approach destinations flexibly in various situations with barriers will be undertaken.

In this paper, a novel fuzzy-observer-based hybrid force/position control method is investigated for a bimanual teleoperation system in the presence of dynamics uncertainties, random network-induced time delays, and multiple sampling rates of remote control signals and local measured data. The system structure consists of two pairs of position observers and contact force/torque estimators. The position observers are designed based on Takagi-Sugeno fuzzy inference rules to estimate the delayed remote state with low sampling rates. The force/torque estimators are designed for estimating the coupled item of uncertain dynamics and contact forces without acceleration information. By adding a compensatory item based on an auxiliary model, the force estimation and motion-tracking errors caused by varying dynamics uncertainties decrease, which is certified by two comparative force estimation techniques. The stability condition for the closed-loop system is also proved by the linear matrix inequality method based on the Lyapunov function. Finally, two simulations verify the effectiveness of the proposed method. The results indicate that the proposed method enables a better motion synchronization effect in soft-handling environment.

Facing the new challenges of the spacecraft developing, the concept of cellular space robot (CSR) for both space-craft system construction and on-orbit operation is presented in this paper. The system description and design principles are introduced to ensure the flexibility of the system. And dynamics model for takeover control is developed. After that, the regression models for the parameter identification are deduced based on the dynamics model. An interaction model identification algorithm is presented to solve the parameter identification problem for the distributed cells. Besides, the interactive model identification is validated by simulations. The simulations show that the interactive model identification method can achieve the consensus and convergence.