Non-rigid registration of intraoperative tissue is essential for surgical navigation and scene reconstruction in minimally invasive surgery. However, accurate registration remains challenging due to significant tissue deformation and partial overlaps caused by laparoscope movement. We propose an Overlap-Aware Online-Adaptive Non-Rigid Registration Method (OANRM) to address these challenges. The framework introduces a Hierarchical Matching Network (HMNet) that simultaneously predicts overlapping regions and their correspondences through a novel similarity-based approach. Our method uniquely incorporates an online adaptation mechanism that continuously fine-tunes the network parameters using unsupervised losses, enabling robust performance across varying surgical scenarios without requiring additional training data. A Transform Displacement Deformation Prediction (TDDP) module further enhances the framework by handling non-overlapping regions through integrating Random Sample Consensus with distance-based interpolation. The method is validated on both artificial datasets with controlled deformations and clinical datasets from real surgical procedures. Experimental results demonstrate that OANRM achieves state-of-the-art performance, significantly outperforming existing methods in handling complex tissue deformations and varying overlap ratios. https://github.com/AIGCer0807/OANRM.

Resection of pathological tissue is a common procedure in surgical oncology for treating tumors. In robot-assisted electrosurgery, the use of predefined markers to guide autonomous robotic resection is gaining traction. Accurate tracking of these markers and minimizing electrocautery damage are critical for the safe and effective autonomous resection of tumors. This paper introduces a safety enhanced autonomous resection method for laparoscopic surgery, designed to mitigate the risks posed by tissue deformation during the resection process. Initially, we pre-plan the cutting path and design a switching strategy for navigation waypoints based on a preview tracking mechanism. Then, we develop a depth-fused navigation controller and a safe withdrawal motion controller. Next, an inertial tracking mechanism is established to evaluate tissue deformation over short periods. Finally, we develop a confidence generator to fuse the two controllers, ensuring that tissue deformation during the resection process does not cause additional electrocautery damage. Simulation and phantom experiments were conducted, demonstrating the effectiveness of our proposed method. This work represents a significant step toward achieving autonomous robotic resection.

The magnetic micromanipulation of swarm microparticles has attracted considerable attention because of its advantages of non-invasiveness, high drug-carrying capacity, and easy observation in the targeted delivery in in-vivo environments. This paper presents an automated control scheme for the magnetic micromanipulation of microswarms in a rotating gradient-based field. Different from the rotating uniform magnetic field generated by Helmholtz coils, the rotating gradient-based field is a type of convergent field established by sequentially powering each coil of the electromagnetic coil system. By changing the coil currents, the field can rotate while driving the microswarm to a pre-determined position, facilitating the swarm localization and tracking. According to the preliminary motion characterization of the swarm in the rotating gradient-based field, an intuitive trapping dynamic model which can simplify the analysis of swarm dynamics is established to facilitate controller design. Based on this model, a super-twisting sliding mode estimator is first designed to estimate the position of the microswarm as well as the disturbances caused by parameter variations and unmodeled dynamics. A robust controller is then developed based on the estimator. In this way, closed-loop manipulation of the microswarm to follow a desired trajectory in the rotating gradient-based field is realized, and the system’s behavior has been significantly improved due to the capability to estimate disturbances. The proposed control scheme for the rotating gradient-based field has the potential to avoid volume loss and unexpected drug diffusion of the swarm when facing complex in-vivo environments. The stability of the control scheme is proved by the Lyapunov approach. Experiments are finally performed to demonstrate the effectiveness of the proposed control approach in a collision-free environment and in a simulated channel.Note to Practitioners—The motivation of this study is to realize automated feedback control of the microparticle swarm in a rotating gradient-based magnetic field. The rotating gradient-based magnetic field is a type of convergent field that can rotate while driving the microswarm to a pre-determined position. This magnetic actuation method facilitates microswarm tracking and enables microagents to overcome static friction with the bottom substrate by rotating them, thereby inducing movement. However, existing research on the rotating gradient-based magnetic field concentrates on moving the microswarm to the targeted position without real-time visual guidance in an open-loop manner, which provides less reliability and convenience compared to real-time closed-loop control. Moreover, it will result in unavoidable volume loss when collisions happen in the complicated environments. In addition, the existing swarm dynamics in such a field need to be simplified to apply in controller design. To solve the above problems, an automated point-to-point navigation control scheme is proposed in this study. An intuitive trapping model is first established to simplify the swarm dynamics. Then, a robust controller with a super-twisting algorithm-based estimator is designed based on the model to manipulate the microswarm. Experimental results have shown that the proposed method can successfully control the microparticle swarm in collision-free environments and a simulated channel with considerable accuracy in the rotating gradient-based magnetic field. The proposed control scheme has the potential to avoid volume loss and unexpected drug diffusion of the swarm when facing complex in-vivo vascular environments, particularly those involve fluids with completely different directions in a bifurcation, such as the vascular network of lymphatic vessels and blood vessels.

This research investigates learning-based control of continuum robots in constrained environments without relying on analytical models. We propose a data-efficient stochastic control strategy incorporating online model updates to achieve precise manipulation even when arbitrary robot deformations occur due to environmental interactions. A localized Gaussian process regression approach accounting for state stochasticity is first presented to approximate the forward kinematics. The learned model enables uncertainty-aware stochastic predictions via the proposed scaled unscented transform (SUT)-based method for efficient exploration. Leveraging new data, online model updates are performed in a highly sample-efficient manner. Furthermore, a probabilistic model predictive control approach integrating the learned models and chance constraints based on Chebyshev’s inequality is developed for searching an optimal control sequence. Simulations and experiments are performed to demonstrate the effectiveness of the proposed approach for controlling continuum robots in constrained environments using limited observational data.Note to Practitioners—The motivation of this research is to solve the problem of controlling continuum robots in constraint environment. The flexibility of continuum robots significantly affects the manipulation accuracy, and the interaction between the continuum robot and environmental constraints can also lead to unpredictable behavior. Learning control methods that rely only on sensory data, provide a feasible solution to the aforementioned problem. However, current methods lack sample efficiency and the capability to handle unknown environmental constraints. This research proposes a learning control method which can control a flexible continuum robot in constrained environments with high data-efficiency and robustness even when the robot shape undergoes sudden deformations due to contact with obstacles.

In minimally invasive surgery (MIS), tissue retraction is critical for exposing the surgical site and facilitating pathological tissue excision. However, tissue retraction in MIS is subject to multiple constraints, including restricted field of view, non-damaging tissue retraction force, and limited instrument operating range, which hinder the safe and continuous retraction of tissue during robotic-assisted surgery. This paper introduces a novel autonomous retraction method for MIS, capable of safely retracting tissue within the aforementioned multiple constraints, assisting the surgeon in tissue excision. The method takes into account information from three different modalities: vision, force, and position. Based on these, three distinct control objectives are defined: retraction angle, retraction force, and safety space constraints, with an individual controller and modelpredictive evaluation function designed for each objective. We propose a novel multi-objective motion fusion strategy designed to balance three distinct control objectives. This strategy evaluates the sensitivity of each objective to changes in the system state by comparing the gradients of their respective prediction evaluation functions and fuses the control inputs of the individual controllers. The proposed method allows for rapid addition or removal of objectives without altering the algorithmic framework. Experiments with phantoms andex vivoanimal tissue are conducted on robotic platform to validate the effectiveness of the proposed method in various configurations.

Accurate estimation of absolute depth from a monocular endoscope is a fundamental task for automatic navigation systems in robotic surgery. Previous works solely rely on uni-modal data (i.e., monocular images), which can only estimate depth values arbitrarily scaled with the real world. In this paper, we present a novel framework, SADER, which explores vision and robot kinematics to estimate the high-quality absolute depth for monocular surgical scenes. To jointly learn the multi-modal data, we introduce a self-distillation based two-stage training policy in the framework. In the first stage, a boosting depth module based on vision transformer is proposed to improve the relative depth estimation network that is trained in a self-supervised method. Then, we develop an algorithm to automatically compute the scale from robot kinematics. By coupling the scale and relative depth data, pseudo absolute depth labels for all images are yielded. In the second stage, we re-train the network with 3D loss supervised by pseudo labels. To make our method generalize to different endoscopes, the learning of endoscopic intrinsics is integrated into the network. In addition, we did cadaver experiments to collect new surgical depth estimation data about robotic laparoscopy for evaluation. Experimental results on public SCARED and cadaver data demonstrate that the SADER outperforms previous state-of-art even stereo-based methods with an accuracy error under 1.90 mm, proving the feasibility of our approach to recover the absolute depth with monocular inputs. Note to Practitioners—This paper aims to solve the problem of absolute monocular depth estimation in automatic surgical navigation by leveraging the multi-modal data from the robot-based endoscopic system. Accurate depth perception with real scales of the monocular scene is essential for the control of surgical robots in automatic navigation. However, current methods can only predict the relative depth of the surgical scene using monocular images. In this article, we propose a self-supervised learning-based method to achieve high-quality absolute depth estimation of monocular endoscopic images. It neither needs manual data annotation, nor other imaging modalities. The experiments extensively validate the feasibility and high performance of our framework for absolute depth estimation on monocular endoscopes. This absolute depth perception framework can be potentially encapsulated into the automatic navigation system in the near future.

The development of autonomous laparoscope control in robot-assisted surgery has emerged as a significant research area, particularly due to its potential to reduce assistant fatigue and minimize miscommunication between the surgeon and assistant. A notable challenge, however, is the tendency of autonomous control strategies to override the surgeon’s direct command occasionally. To address this issue, we propose a novel haptic feedback-based cooperative control strategy that enhances the surgeon’s command of laparoscopic field of view (FOV) movement in robot-assisted laparoscopic surgery. Specifically, we first established a dynamic model of the laparoscope-holding robot, which serves as a link between the movement of the laparoscopic FOV and the surgical instruments to deliver haptic feedback to the surgeon. Next, a motion observer was developed to transform 30 Hz visual feedback into 1 kHz haptic feedback by integrating visual tracking data with kinematic information, ensuring smoother and more continuous haptic feedback. Finally, we propose two distinct collaboration modes: the plane tracking mode (PTM) ensures instruments remain within the laparoscopic image, and the space tracking mode (STM) synchronizes the laparoscope with instrument movement. The laboratory experiments validated the effectiveness of the proposed method in enhancing the cooperative performance of robot-assisted laparoscope systems while animal experiments demonstrated the feasibility of the PTM design.

Laparoscope-holding robots significantly enhance the stability and precision of visualization in minimally invasive surgeries. Most existing robots of this kind depend on visual servo systems and struggle with efficient, rapid adjustments in the field-of-view (FOV), especially when identifying organs and needles outside the FOV. This paper presents a laparoscope-holding robot system capable of employing both vision-driven and force-driven mechanisms for continuous and large-scale FOV adjustments, respectively. The system features an integrated tactile handle, enabling the reception of human-robot interaction forces during surgical navigation. We propose a hybrid control method that leverages both force and vision inputs for laparoscopic FOV adjustments. This approach integrates a virtual wrench, generated from visual information, and an interaction wrench, obtained from the tactile handle, into the robot's dynamic model, which complies with remote center of motion constraints. The interaction wrench's gain is adjusted with the gripping force on the integrated tactile handle, ensuring that unintended movements caused by accidental contacts are prevented, thus safeguarding operational safety. The proposed method eliminates the need to switch control modes, enabling simultaneous visual tracking and tactile interaction guidance. Experimental results demonstrate that the proposed method not only allows for FOV adjustments with surgical instrument guiding but also adapts well to large-scale FOV adjustment tasks.

Numerous studies have emphasized the application of autonomous intelligence in human-robot shared control to enhance surgical convenience and efficiency. However, the neglect of human dominance may reduce surgical safety. This paper developed a safety-enhanced human-robot shared control method by intelligently allocating control authority, with the surgeon remaining the leader during the surgical procedure. Three controllers are designed initially, including a master hand position (MP) controller and a master hand velocity (MV) controller related to the surgeon's manipulation, and a planned trajectory tracking (PT) controller related to the robot. In precision surgical manipulation scenarios, precise tracking of the human's operation is achieved by combining MP and MV controllers, while a combination of MV and PT controllers is developed in high-efficiency surgical scenarios, which relaxes the requirement for precise tracking of hand position and enables precise robot assistance guided by the velocity of human hand. The autonomous scenarios and controllers switching are accomplished through a motion fusion mechanism, which is achieved via optimizing evaluation functions that are reliant on future states. Furthermore, a force feedback mechanism is proposed to help human understand the intent of autonomous control to improve safety. The feasibility and effectiveness of this method have been validated through simulations and experiments.

Estimating precise metric depth and scene reconstruction from monocular endoscopy is a fundamental task for surgical navigation in robotic surgery. However, traditional stereo matching adopts binocular images to perceive the depth information, which is difficult to transfer to the soft robotics-based surgical systems due to the use of monocular endoscopy. In this paper, we present a novel framework that combines robot kinematics and monocular endoscope images with deep unsupervised learning into a single network for metric depth estimation and then achieve 3D reconstruction of complex anatomy. Specifically, we first obtain the relative depth maps of surgical scenes by leveraging a brightness-aware monocular depth estimation method. Then, the corresponding endoscope poses are computed based on non-linear optimization of geo-metric and photometric reprojection residuals. Afterwards, we develop a Depth-driven Sliding Optimization (DDSO) algorithm to extract the scaling coefficient from kinematics and calculated poses offline. By coupling the metric scale and relative depth data, we form a robust ensemble that represents the metric and consistent depth. Next, we treat the ensemble as supervisory labels to train a metric depth estimation network for surgeries (i.e., MetricDepthS-Net) that distills the embeddings from the robot kinematics, endoscopic videos, and poses. With accurate metric depth estimation, we utilize a dense visual reconstruction method to recover the 3D structure of the whole surgical site. We have extensively evaluated the proposed framework on public SCARED and achieved comparable performance with stereo-based depth estimation methods. Our results demon-strate the feasibility of the proposed approach to recover the metric depth and 3D structure with monocular inputs.

This study investigates the use of a tendon-driven continuum robot to deform a soft object, whereas the robot body is deformed into an arbitrary shape to adapt to a constrained environment. A dynamic estimator (DE) is developed to approximate the Jacobian matrix that associates the actuator input with the deformed output of the soft object. This helps solve the singularity problem and reduce the effects of noise. Then a visual predictive controller (VPC) with a reference trajectory is developed to ensure a smooth operation. A linear extended-state observer (ESO) is further designed to measure the robot states, such that the controller can compensate for the estimation error. Simulations and experiments are performed to verify the proposed control approach.Note to Practitioners—The motivation of this article is to solve the problem of automatic deformation control of soft objects in restricted environments. The existing soft object deformation control is achieved using rigid robots in an open environment, but rigid robots are difficult to use in specific applications where the environment is restricted (e.g., natural orifice surgery). Flexible continuum robots with mechanical compliance can manipulate soft objects in narrow spaces. However, due to environmental constraints, the robot body may be deformed into any shape regardless of the input of the actuator. To solve the problem, this research provides a new visual servo control strategy that deforms soft objects using a continuum robot in a restricted environment. The proposed method can control a flexible robot to manipulate soft objects while taking into account the change in the robot configuration in a restricted environment.

Flexible continuum robots have exhibited unique advantages in working in an unstructured environment. Many applications require robots to actively control the deformation of soft objects, such as soft tissues in surgery. Thus, this study presents a robust model-predictive deformation control of a soft object using a flexible continuum robot. A linear approximation model for mapping from actuation space of a continuum robot to deformation space of a soft object is established. Jacobian matrix is estimated online by using a robust Geman-McClure estimator. Then, the deformation of the soft object is regulated by using a prediction horizon-based controller with exponential weighting for model uncertainty. The proposed control approach is effective in manipulating a soft object with a flexible continuum robot that is in contact with obstacles.