Robotic soft tissue manipulation in surgery presents significant challenges due to the tissue’s high deformability and the spatial constraints of the surgical environment. While data-driven methods for affordance estimation are common, they often face challenges with data requirements and generalization in surgical scenarios. To address these challenges, a novel framework is proposed that combines a deformation model-based shape controller with an analytical affordance estimation approach for multi-contact scenarios by employing a manipulability metric. The method leverages the deformation Jacobian matrix derived from a linearized deformation model to evaluate the manipulability of candidate multi-contact points, providing a robust and data-efficient solution. For tissue manipulation, a differentiable deformation model is employed to efficiently compute forward and backward deformation processes in real-time. Point-based visual features are constructed to represent and track the tissue deformation, enabling precise control through visual feedback. The proposed framework has been validated through simulations and physical experiments. These results demonstrate the real-time (∼ 60Hz) ability to achieve targeted configuration with high accuracy (< 1mm RMSE for each marker position) and a strong correlation (near-zero p-value) between the predicted affordance and the observed manipulation efficiency, confirming the effectiveness of our approach in soft tissue manipulation and affordance estimation.

The inherent flexibility and real-time deformation of breast tissue pose significant challenges for achieving full coverage and accurate lesion localization in autonomous breast ultrasound scanning. This paper introduces a robust finite state machine-based framework that mimics the decision-making process of an experienced physician, dynamically transitioning between the global breast scan and the fine lesion scan. An autonomous radial and anti-radial global scan pattern ensures comprehensive breast coverage. To avoid lesion misidentification caused by soft tissue movement, a real-time lesion fine scan method is proposed for lesion detection and localization. Experimental results demonstrate that the system in full coverage tests achieves 7 identified lesions out of 7 existing lesions and maintains a robust localization accuracy of$\mathbf{3. 2 3 ~ m m}$across phantoms with varying stiffnesses.

Deformable object manipulation (DOM) which is a common subtask in various surgical procedures represents an inevitable challenge in robot-assisted surgery (RAS) due to complex nonlinear deformation. This paper proposes a deep reinforcement learning guided adaptive control (RLAC) modelfree framework, which combines learning-based and Jacobianbased methods. To complement each other for optimized performance, we harness the sampling of deep reinforcement learning (DRL) policy explored in simulations to solve a reasonable estimation of the initial deformation Jacobian. In early control iterations, the actions suggested by the DRL agent are adopted until the estimated real-time Jacobian approximates the actual deformation model. Subsequently, the independent Jacobianbased adaptive control (AC) with sufficient initial deformation awareness begins execution to achieve precise internal feature manipulation on deformable objects. Experimental results demonstrate that our method enables more efficient positioning and exhibits near-optimal positioning paths. RLAC with robust sim-to-real performance provides a feasible approach for the complex autonomous DOM in the real world.

Circular electrical connectors (CECs) have a wide range of applications in scenarios that require reliable connections. However, sockets are often located in narrow scenes with random spatial orientations, complex lighting conditions, and obstructions from cables, making it difficult to accurately locate them through cameras. Besides, due to the complex geometric structure of CECs and the presence of electrode protection slots, the existing research on the assembly of cylindrical or polygonal pegs and holes may not be applicable to the assembly of such components. To this end, this article proposes a novel robotic assembly strategy for CECs with small relative initial deviations, whose core is to design a search trajectory and heuristic force strategy to perceive force/pose (F/P) discontinuity characteristics under different geometric constraints. This assembly strategy is independent of the CEC's size and is not affected by the socket's spatial orientation. The experiments with two different sizes of CECs on a robot equipped with a 6-dimensional force/torque ($\mathbf{F} / \mathbf{T}$) sensor are conducted, and the effectiveness and robustness of the proposed assembly strategy for CECs are demonstrated.

Surgical procedures are inherently complex and dynamic, with intricate dependencies and various execution paths. Accurate identification of the intentions behind critical actions, referred to as Primary Intentions (PIs), is crucial to understanding and planning the procedure. This paper presents a novel framework that advances PI recognition in instructional videos by combining top-down grammatical structure with bottom-up visual cues. The grammatical structure is based on a rich corpus of surgical procedures, offering a hierarchical perspective on surgical activities. A grammar parser, utilizing the surgical activity grammar, processes visual data obtained from laparoscopic images through surgical action detectors, ensuring a more precise interpretation of the visual information. Experimental results on the benchmark dataset demonstrate that our method outperforms existing surgical activity detectors that rely solely on visual features. Our research provides a promising foundation for developing advanced robotic surgical systems with enhanced planning and automation capabilities.

Dual-arm robots, with their high flexibility and broad operational range, are widely used in industrial and household transportation tasks. However, their high degrees of motion freedom and the closed-chain constraints formed during transportation pose challenges for path planning and force control. This study proposes a dual-agent control framework based on the Soft-Actor-Critic (SAC) algorithm of Deep Reinforcement Learning (DRL), where one agent is responsible for path planning and the other handles force control. This framework enables dual-arm robots to achieve dynamic obstacle avoidance, avoid unsolvable configurations and singularities, and generate efficient and smooth paths, while also fulfilling internal force tracking requirements based on task demands. Additionally, it addresses the gap in transferring the force control agent from simulation to real-world applications through a state mapping network, and the force control agent does not require retraining for different objects. Finally, the effectiveness of the proposed framework is validated through two scenes, including multiple bookshelves stacking and dynamic obstacle avoidance during the box transportation. Validation was also carried out with objects of different geometries and weights.

The classical image-based visual servoing (IBVS) methods exhibit strong robustness to robot modeling and camera calibration errors, but suffer from uncontrollable spatial trajectories and local convergence issues. Although current IBVS variants employ specialized visual features or information to improve trajectory controllability, these approaches impose restrictive geometric constraints, such as requiring coplanar feature points or configurations approximately orthogonal to the optical axis of the camera. Furthermore, despite conclusive evidence of local minima (LM) in the IBVS scheme, there is currently no solution to address this challenge. For the reason, this article first introduces a novel virtual image construction method and proposes a decoupling visual servoing control law based on virtual image, achieving explicit separation of translational and rotational error dynamics. Then, through Monte Carlo simulations, the spatial distribution patterns of local minima in IBVS are systematically studied, and an escape algorithm leveraging virtual image is further designed. To the best of our knowledge, this may be the first attempt to address the issue of local minima only through IBVS to a certain extent. Comparative simulations and experiments validate the effectiveness and superiority of the proposed decoupling control law based on virtual image, and the success rate of the local minima escape algorithm is 100% under different configurations.

Electrical connectors (ECs) are extensively employed in industrial scenarios, and their assembly quality is crucial. However, these connectors are often located in confined spaces, which poses challenges of complex lighting conditions and visual occlusions during the execution of robotic assembly tasks. Hence, guiding robots to assemble solely through force/torque (F/T) feedback is an alternative way. However, there is currently limited research on achieving assembly tasks solely through F/T information, especially when the spatial pose of the socket is uncertain, and how to achieve the robotic assembly of ECs remains a difficult problem. To this end, this paper proposes a novel strategy for assembling ECs under large initial pose deviations. Specifically, inspired by observing the assembly process of humans without visual assistance, the robotic assembly is first divided into two stages: by arbitrary surface tracking, the relative pose of the current EC is confirmed and adjusted to a dual-point contact state, i.e. the conversion from non-contact to dual-point contact; by setting heuristic F/T, the alignment of the edge, plane and slots of EC is driven, i.e. the alignment of the plug and socket. Next, we analyse the geometric constraint states at these stages and formulate the corresponding desired contact F/T strategies. To our knowledge, this may be the first attempt to achieve the robotic assembly of ECs under large initial deviations solely using F/T information. Finally, experiments on a UR5 robot indicate that the proposed strategy exhibits robustness to large initial pose deviations and can overcome obstacles caused by friction and jamming to ensure the robotic assembly of ECs. Note to Practitioners—The automatic assembly of ECs in confined spaces with visual occlusions remains an unsolved challenge, especially in aerospace scenarios where manual assembly may not be appropriate. The proposed assembly strategy is based on F/T perception without visual assistance, which can effectively address the perception and assembly processes of plugs under large initial pose deviations, demonstrating robust performance. With the requirement of only a F/T sensor and no need for precise dimensions of the assembly object, this strategy exhibits favourable deployment characteristics for automatic assembly platforms. Furthermore, the proposed strategy can be optimized by the integration of learning-based methods into the perception process, thereby further improving assembly efficiency.

Chatter will affect machining accuracy, production efficiency, tool wear and workers' health. In order to avoid chatter early, a grinding chatter online monitoring model based on multisensor fusion information and hybrid deep neural network is proposed. First, the grinding experiment of acoustic emission (AE), force and displacement multichannel signal acquisition are carried out. Then, the grinding process is divided into five stages: air cut; stable; slight chatter; severe chatter; and severe chatter with beat effect, the correlation between sensor signals and classic evaluation indicators is analyzed. Next, a hybrid deep neural network model is established, and the feature classification ability, testing accuracy, sensitivity and generalization ability of the model are studied. Finally, the proposed model is applied to microstructured grinding wheel to further verify the generalization ability and chatter prediction ability of the model. The results indicate that our approach can predict the occurrence of flutter 0.15–0.45 s in advance.

Accurate and reliable annotation of anatomical landmarks in laparoscopic surgery remains a challenge due to varying degrees of landmark visibility and changing shapes of human tissues during a surgical procedure in videos. In this paper, we propose a knowledge-driven framework that integrates prior surgical expertise with visual data to address this problem. Inspired by visual reasoning knowledge of tool-anatomy interactions, our framework models a spatio-temporal graph to represent the static topology of tool and tissue and dynamic transitions of landmarks' temporal behavior. By assigning explainable features of the surgical scene as node attributes in the graph, the surgical context is incorporated into the knowledge space. An attention-guided message passing mechanism across the graph dynamically adjusts the focus in different scenarios, enabling robust tracking of landmark states throughout the surgical process. Evaluations on the clinical dataset demonstrate the framework's ability to effectively use the inductive bias of explainable features to label landmarks, showing its potential in tackling intricate surgical tasks with improved stability and reliability.

Harmonious human-robot collaboration requires the robot to behave like a human partner, which raises the critical question of what factors make the robot do so. This paper proposes a series of policies based on empathetic and non-empathetic intent inference, proactive and reactive action planning, and ego and non-ego action styles to examine which modules enable robots to exhibit human-like behaviors. Two series of experiments are conducted with human subjects to test the performance of the proposed controllers. In Experiment 1, the participant must identify whether the collaborating partner is a human, similar to a Turing test. The classification results empirically verify that the designed empathetic proactive policies enable the robot to exhibit human-like behaviors. Experiment 2 indicates that the proposed policy can be applied to complex collaborative tasks, and this result is consistent with the findings of Experiment 1. From empirical evidence from the experiments, we believe that empathy and proactive policies are essential elements to enable robots to perform human-like actions.

In minimally invasive endovascular interventional surgery, guidewire navigation is an indispensable process. However, even experienced physicians often encounter difficulties in manually manipulating the guidewire for branch selection, while also facing the risk of radiation exposure. In this study, we investigated robotic autonomous guidewire navigation methods. An electromagnetic system was used to track the real-time position and orientation of the guidewire tip, and a state space representing the guidewire within the vascular environment was constructed to guide the robot in precise guidewire manipulation. Experimental results demonstrated that the proposed trial-and-error and centerline-guided methods successfully completed navigation tasks in a static environment, outperforming human navigation performance in terms of trajectory smoothness, trajectory length, and incorrect branch entry counts. For dynamic environment navigation, dynamic time warping (DTW), a technique for measuring the similarity between two temporal sequences, was integrated into the centerline-guided method. The proposed approaches eliminate the need for visual feedback and thereby minimizing the risk of radiation exposure for both patients and medical staff present in the operating room during the procedure.

Calibrating unknown transformation relationships is an essential task for multirobot cooperative systems. Traditional linear methods are inadequate to decouple and simultaneously solve the unknown matrices due to their intercoupling. This article proposes a novel dual-robot accurate calibration method that uses convex optimization and Lie derivative to solve the dual-robot calibration problem simultaneously. The key idea is that a convex optimization model based on dual-robot transformation chain is established using Lie representation of special Euclidean group in 3 dimensions [SE(3)]. The Jacobian matrix of the established optimization model is explicitly derived using the corresponding Lie derivative ofSE(3). To balance the influence of the magnitudes of the rotational and translational optimization variables, a weight coefficient is defined. Due to the closure and smoothness of Lie group, the optimization model can be solved simultaneously using Newton-like iterative methods without additional orthogonalization processing. The performance of the proposed method is verified through simulation and actual calibration experiments. The results show that the proposed method outperforms the previous calibration methods in terms of accuracy and stability. The actual experiments are used to compare the proposed method with two existing calibration methods, and the mean measurement error of a certified ceramic sphere is reduced from 0.9205 and 0.5363 to 0.4381 mm, respectively.

Surgical workflow analysis integrates perception, comprehension, and prediction of the surgical workflow, which helps real-time surgical support systems provide proper guidance and assistance for surgeons. This article promotes the idea of critical actions, which refer to the essential surgical actions that progress towards the fulfillment of the operation. Fine-grained workflow analysis involves recognizing current critical actions and previewing the moving tendency of instruments in the early stage of critical actions. Aiming at this, we propose a framework that incorporates operational experience to improve the robustness and interpretability of action recognition in in-vivo situations. High-dimensional images are mapped into an experience-based explainable feature space with low dimensions to achieve critical action recognition through a hierarchical classification structure. To forecast the instrument's motion tendency, we model the motion primitives in the polar coordinate system (PCS) to represent patterns of complex trajectories. Given the laparoscopy variance, the adaptive pattern recognition (APR) method, which adapts to uncertain trajectories by modifying model parameters, is designed to improve prediction accuracy. The in-vivo dataset validations show that our framework fulfilled the surgical awareness tasks with exceptional accuracy and real-time performance.

Robot-Assisted Minimally Invasive Surgery (RAMIS), which introduced robot-actuated invasive tools to increase the dexterity and efficiency of traditional MIS, has become popular. Investigations on how to achieve autonomy in RAMIS have drawn vast intention recently, which urges further insights into the process of the surgical procedures. In this paper, the definition of critical actions, which discriminates the essential stages from regular surgical actions, is proposed to help decompose the complicated surgical processes. A critical intra-operative moment of the surgical workflow, which is called the keyframe, is introduced to indicate the beginning or ending moments of the critical actions. A keyframe detection method is proposed for critical action identification based on a new in-vivo dataset labeled by expert surgeons. Surgeons' criteria for critical actions are captured by the explainable features, which can be extracted from the raw laparoscopic images with a two-stage network. Motivated by the surgeon's decision process of keyframes, a hierarchical structure is designed for keyframe identification by checking the spatial-temporal characteristics of the explainable features. Experimental results show that the reliability of the proposed method for keyframe detection achieves unanimous agreement by expert surgeons.

Due to the robustness to robot modeling and camera calibration errors and avoidance of complete target geometry, image-based visual servoing has always been an important topic in the fields such as robotics, computer vision and so forth. When the image information obtained by the camera is mapped to the robotic task space to design the servoing control law, the resulting interaction matrix, which links the spatial velocity of the camera to the temporal variation of the selected image features, depends on the unknown feature depths. The use of inaccurate feature depths may influence the stability and robustness of the controller, and even cause the failure of the task. In this article, based on the perspective camera model, by employing the principle of reduced order observer, a novel logarithmic observer is presented for on-line recovery of feature depth. Compared with the typical observers now available, the presented observer offers several advantages: global convergence, a faster convergence rate of error structure than exponential error structure, a less restricted observability condition and greater robustness against measurements with noise. The comparison results of numerical simulations indicate the superiority of the presented observer, and real experiments with Kinect v2 sensor further validate the effectiveness of the presented observer in practical situation. Note to Practitioners—This article was motivated by the depth problem in the image-based visual servoing scheme, but it can also be used in other situations where the image depth information is needed, such as 3D reconstruction, robot navigation, etc. The existing depth acquisition methods include TOF sensors, stereo vision, depth observers and so on. However, TOF sensors are sensitive to light conditions, and the mounting space of stereo vision is slightly large, and there is contradiction between observation performance and computational complexity in most existing observers. In this article, a novel structure of logarithmic reduced order observer is described in detail, which can be utilized to estimate the depth information of images easily. The simulations and experiments verify the good performance of the observer. The limitations of the given observer are that the estimation accuracy is not very good under weak excitation, and the camera needs to be calibrated in advance. Future work will focus on overcoming these two limitations.

Industrial robots are characterized by good flexibility and a large working space, and offer a new approach for the machining of large and complex parts with small machining allowances (extra material allowed for subsequent machining). Parts of this type (such as aircraft skin parts, wind turbine blades, etc.) are easily deformed due to their large scale and low stiffness. Therefore, these parts cannot be directly machined according to the designed model. A feasible method is to plan a robotic machining path by using the point clouds of parts after clamping from onsite measurement which contains inherent defects of measurement such as noise points and abrupt points. In this article, a novel method is proposed to plan and optimize a robotic machining path that meets the requirements of smoothness, dexterity, and stiffness based on the point cloud from onsite measurement. The dual nonuniform rational B-spline curves of the machining path points and tool axis points are generated at first. Next, an objective function of smoothness optimization is established to filter out the local mutation of the path by considering the constraints of both the deformation energy and the deviation. Then, the objective function of robot postures optimization is established to optimize dexterity and Cartesian stiffness of a robot during the machining process. To show the feasibility of the proposed method, simulation and experiments are carried out. It is proved that the proposed method can generate a smooth machining trajectory. The stability of joint rotation and the rigidity and dexterity of the robot are improved during the machining process.

This paper presents a novel eddy current damper design for chatter suppression in robotic milling process. The designed eddy current dampers are installed on a milling spindle to damp the tool tip vibrations. The structural design of the eddy current dampers and the working principle of the proposed vibration attenuation method are explained. Finite element method is used to analyze the magnetic flux density and the magnetic force generated by the designed eddy current. The dynamics of the robotic milling system without and with eddy current dampers are modeled, and the damping performance of the proposed method is verified through simulations in both frequency and time domains. The results show that the peaks of the tool tip frequency response function caused by the spindle and milling tool modes are damped by 3.2 dB and 5.3 dB, respectively, and the chatter stability is improved by about 43% in the high spindle speed zone, compared to the case without eddy current dampers.

Without the 3-D geometry of the target and robust to camera calibration error, image-based visual servoing schemes have gained a lot of attention. However, the depth of the selected feature, which is involved in the interaction matrix relating the time variation of the feature to the velocity twist of the camera, must be estimated correctly to guarantee the stability of the controller. To this end, this paper proposes a new nonlinear reduced-order observer structure to recover the feature depth in real time. Compared with the existing works, the proposed observer has a global asymptotic convergence property and fast convergence rate, and the convergence rate can be easily adjusted only using a single gain parameter. In addition, the proposed observer has a less restrictive observability condition and stronger robustness to noisy measurements. Extensive comparative numerical simulations are carried out to validate the effectiveness of the proposed depth observer.