Monocular endoscope-based reconstruction of dynamic 3D surgical fields is beneficial for both intraoperative manual/robotic manipulation and post-operative surgical skills training. However, the natural characteristics of tissue deformations with instruments and blood obscuration bring great challenges to 3D scene awareness, especially under sparse viewpoints limited by laparoscopic movements. In this work, we propose Endo-4SRF, an effective Neural Radiance Field (NeRF)-based method that can reconstruct deformable tissues with instruments stealth by solely relying on monocular endoscopic image flows from a single viewpoint. Specifically, to enhance the 3D reconstruction accuracy under the deficiency of depth ground truth, we devised a dynamic Gaussian-based neural sampling strategy, leveraging the depth inherently obtained from NeRF and the conjunction information inferred by a prior learning-based depth estimation network. Besides, we integrated the Signed Distance Function (SDF) and resolved its singularity problem by furnishing additional geometric constraints for the neural radiance field, thereby achieving precise reconstruction of dynamic scenes devoid of depth ground truth supervision. Furthermore, adopting spherical harmonic functions for color fitting has significantly improved our model's computational efficiency and rendering quality. We extensively performed cross validation experiments to verify the performance using public and in-house datasets. Our quantitative and qualitative results demonstrate remarkable superiority over the state-of-the-art (SOTA) approaches concerning depth prediction accuracy, image rendering quality, model training efficiency, and 3D reconstruction outcomes.

Robotic endoluminal surgery has gained tremendous attention for its enhanced treatments in gastrointestinal intervention, where navigating surgeons with monocular camera-based metric depth estimation is a vital sector. However, existing methods either rely on external sensors or perform poorly in terms of visual navigation. In this work, we present our M$^{3}$-Degrees Net, a novel monocular vision-guided and graph learning-based network tailored for accurate metric marching depth (MD) estimation. We first leverage a generative model to output a scale-free depth map, providing a depth basis in a coarse granularity. To achieve an optimized and metric MD prediction, a relational graph convolutional network with multi-modal visual knowledge fusion is devised. It utilizes shared salient features between keyframes and encodes their pixel differences on the depth basis as the main node, while a projection length-based node that predicts the MD on a proportional relationship basis is introduced, aiming to enable the network with explicit depth awareness. Moreover, to compensate for rotation-induced MD estimation bias, we model the endoscope's orientation changes as image-level feature shifts, formulating an ego-motion correction node for MD optimization. Lastly, a multi-layer regression network for the metric MD estimation with finer granularity is devised. We validate our network on both public and in-house datasets, and the quantitative results reveal that it can limit the overall MD error under 27.3%, which vastly outperforms the existing methods. Besides, our M$^{3}$-Degrees Net is qualitatively tested on the in-house clinical gastrointestinal endoscopy data, demonstrating its satisfactory performance even under cavity mucus with varying reflections, indicating promising clinical potentials.

Robotic-assisted percutaneous coronary intervention (PCI) holds considerable promise for elevating precision and safety in cardiovascular procedures. Nevertheless, current systems heavily depend on human operators, resulting in variability and the potential for human error. To tackle these challenges, Sim4EndoR, an innovative reinforcement learning (RL) based simulation environment, is first introduced to bolster task-level autonomy in PCI. This platform offers a comprehensive and risk-free environment for the development, evaluation, and refinement of potential autonomous systems, enhancing data collection efficiency and minimizing the need for costly hardware trials. A notable aspect of the groundbreaking Sim4EndoR is its reward function, which takes into account the anatomical constraints of the vascular environment, utilizing the geometric characteristics of vessels to steer the learning process. By seamlessly integrating advanced physical simulations with neural network-driven policy learning, Sim4EndoR fosters efficient sim-to-real translation, paving the way for safer, more consistent robotic interventions in clinical practice, ultimately improving patient outcomes.

Accurate three-dimensional (3D) reconstruction of guidewire shapes is crucial for precise navigation in robot-assisted endovascular interventions. Conventional 2D Digital Subtraction Angiography (DSA) is limited by the absence of depth information, leading to spatial ambiguities that hinder reliable guidewire shape sensing. This paper introduces a novel multimodal framework for real-time 3D guidewire reconstruction, combining preoperative 3D Computed Tomography Angiography (CTA) with intraoperative 2D DSA images. The method utilizes robust feature extraction to address noise and distortion in 2D DSA data, followed by deformable image registration to align the 2D projections with the 3D CTA model. Subsequently, the inverse projection algorithm reconstructs the 3D guidewire shape, providing real-time, accurate spatial information. This framework significantly enhances spatial awareness for robotic-assisted endovascular procedures, effectively bridging the gap between preoperative planning and intraoperative execution. The system demonstrates notable improvements in real-time processing speed, reconstruction accuracy, and computational efficiency. The proposed method achieves a projection error of 1.76±0.08 pixels and a length deviation of 2.93±0.15%, with a frame rate of 39.3 1.5 frames per second (FPS). These advancements have the ±potential to optimize robotic performance and increase the precision of complex endovascular interventions, ultimately contributing to better clinical outcomes.

Over the past decade, significant advancements have been made in the research and industrialization of robotic systems for endovascular procedures, yet their clinical application remains relatively limited. Physicians commonly report that these robots lack certain intelligent assistive capabilities during procedures. There has been increasing interest and attempts to apply learning-centered algorithms to the training and enhancement of surgical robot skills. This paper proposes an autonomous navigation algorithm for interventional guidewires that is initially trained solely in a virtual simulation environment and subsequently deployed to a real-world robot. Experimental results demonstrate the feasibility of this approach for real-world applications. The proposed approach can help physicians reduce the learning curve for guidewire manipulation and elevate the robot to a higher level of autonomous operation, thereby breaking through the current bottleneck in the level of intelligence for clinical applications of interventional robots. It also holds promise for bringing intelligent transformation to future interventional procedures. Note to Practitioners—This work is motivated by the emerging need to increase the level of autonomy in robotic-assisted endovascular procedures, which has the potential to improve procedural efficiency, standardize procedures, and broaden the adoption of robotic systems in clinical practice. The proposed simulation-based reinforcement learning provides a safe and efficient method for training robotic systems, enabling them to master complex tasks in simulation environments prior to real-world application. The successful deployment of models trained in simulation onto physical robotic platforms demonstrates the feasibility of this method for real-world applications. The proposed simulation-based reinforcement learning method offers a promising and viable pathway for enhancing skill acquisition in endovascular interventional robots.

Shape control of deformable objects is a challenging and important robotic problem. This article proposes a model-free controller using novel 3-D global deformation features based on modal analysis. Unlike most existing controllers using geometric features, our controller employs physically based deformation features designed by decoupling global deformation into low-frequency modes. Although modal analysis is widely adopted in computer vision and simulation, its usage in robotic deformation control is still an open topic. We develop a new model-free framework for the modal-based deformation control. Physical interpretation of the modes enables us to formulate an analytical deformation Jacobian matrix mapping the robot manipulation onto changes of the modal features. In the Jacobian matrix, unknown geometric and physical models of the object are treated as low-dimensional modal parameters, which can be used to linearly parameterize the closed-loop system. Thus, an adaptive controller with proven stability can be designed to deform the object while online estimating the modal parameters. Simulations and experiments are conducted using linear, planar, and volumetric objects under different settings. The results not only confirm the superior performance of our controller, but also demonstrate its advantages over the baseline method.

Automatic laparoscope motion control is fundamentally important for surgeons to efficiently perform operations. However, its traditional control methods based on tool tracking without considering information hidden in surgical scenes are not intelligent enough, while the latest supervised imitation learning (IL)-based methods require expensive sensor data and suffer from distribution mismatch issues caused by limited demonstrations. In this paper, we propose a novel Imitation Learning framework for Laparoscope Control (ILLC) with reinforcement learning (RL), which can efficiently learn the control policy from limited surgical video clips. Specially, we first extract surgical laparoscope trajectories from unlabeled videos as the demonstrations and reconstruct the corresponding surgical scenes. To fully learn from limited motion trajectory demonstrations, we propose Shape Preserving Trajectory Augmentation (SPTA) to augment these data, and build a simulation environment that supports parallel RGB-D rendering to reinforce the RL policy for interacting with the environment efficiently. With adversarial training for IL, we obtain the laparoscope control policy based on the generated rollouts and surgical demonstrations. Extensive experiments are conducted in unseen reconstructed surgical scenes, and our method outperforms the previous IL methods, which proves the feasibility of our unified learning-based framework for laparoscope control.

In this study, a system of automatic microparticle patterning that could enable the separation, trapping, and translation of single microbeads in liquid suspension using negative dielectrophoresis (DEP) tweezers was presented to form a single-bead pattern. A microchip with integrated electrodes was flipped and placed above the substrate through a micromanipulator. Microparticles laying on the substrate could be displaced to different positions relative to the electrodes on the microchip, and only the selected particles would be trapped by the electric fields generated from electrodes. Vision-based approaches were used to evaluate the necessary information, such as the gap distance and the positions of electrodes and microparticles in the image. A strategy for separating nearby particles was proposed to achieve single-bead patterning with high accuracy. A controller was used to guide the microparticles toward the position for trapping while avoiding flow disturbance. Different strategies were simulated to decrease the patterning time and find the minimum traveling distance and the best route of movement. The optimization problem is NP-hard. Hence, global optimization algorithms, such as genetic algorithm, particle swarm optimization, and ant colony optimization (ACO), were simulated, and the results were compared with those of the local optimization method. The comparison results showed that ACO obtained the best performance among the methods. The strategy for constructing high-quality microparticle patterns was also examined through experiments. Orange fluorescent polystyrene beads suspended in 6-aminohexanoic acid solution were considered and successfully patterned on a glass substrate by using the proposed system.Note to Practitioners—Micropatterning is an effective tool for pharmaceutical research and drug discovery. However, the reliability of results depends on the quality of patterns. Existing approaches, such as microfluidic devices, are limited to create a pattern from one chip for single use, and the entire process is sealed and isolated from the environment. In this study, a multielectrode microchip combined with a vision-based micromanipulator is introduced to create a novel noncontact approach for microparticle patterning, which offers high flexibility and guarantees the quality of the constructed patterns. The electrodes on the chip can be selectively energized to determine the shape of the final pattern. A real-time screening is performed so that the micromanipulator will only guide the particles in good condition for selection. An optimization algorithm is implemented to aid the particle selection with the electrodes, allowing high-quality microparticle patterns to be constructed in a short time for various applications.

To realize a higher-level autonomy of surgical knot tying in minimally invasive surgery (MIS), automated suture grasping, which bridges the suture stitching and looping procedures, is an important yet challenging task needs to be achieved. This paper presents a holistic framework with image-guided and automation techniques to robotize this operation even under complex environments. The whole task is initialized by suture segmentation, in which we propose a novel semi-supervised learning architecture featured with a suture-aware loss to pertinently learn its slender information using both annotated and unannotated data. With successful segmentation in stereo-camera, we develop a Sampling-based Sliding Pairing (SSP) algorithm to online optimize the suture’s 3D shape. By jointly studying the robotic configuration and the suture’s spatial characteristics, a target function is introduced to find the optimal grasping pose of the surgical tool with Remote Center of Motion (RCM) constraints. To compensate for inherent errors and practical uncertainties, a unified grasping strategy with a novel vision-based mechanism is introduced to autonomously accomplish this grasping task. Our framework is extensively evaluated from learning-based segmentation, 3D reconstruction, and image-guided grasping on the da Vinci Research Kit (dVRK) platform, where we achieve high performances and successful rates in perceptions and robotic manipulations. These results prove the feasibility of our approach in automating the suture grasping task, and this work fills the gap between automated surgical stitching and looping, stepping towards a higher-level of task autonomy in surgical knot tying. Note to Practitioners—This paper aims to automate the suture grasping task in surgical knot tying by leveraging stereo visual guidance. To effectively robotize this procedure, it requires multidisciplinary knowledge to achieve suture segmentation, 3D shape reconstruction, and reliable automated grasping, while there are no existing works tackling this procedure especially using robots with RCM kinematics constraints and under complex environments. In this article, we propose a learning-driven method along with a 3D shape optimizer, which can conduct the suture segmentation and output its accurate spatial coordinates, serving as guidance for automated grasping operation. Apart from this, we introduce a unified function to optimize the grasping pose, and a vision-based grasping strategy is also proposed to intelligently complete this task. The experiments extensively validate the feasibility of our framework for automated suture grasp, and its successful completion can serve as a basis for the following looping manipulation, hence filling a step gap in robot-assisted knot tying. This framework can be also encapsulated into the medical robotic system, and by simply indicating (e.g. mouse click) the rough position of the suture’s tip in one camera frame, the overall framework can be initialized and further accomplish the suture grasping task, which further prompts a full autonomy of surgical knot tying in the near future.

Laparoscopic Field of View (FOV) control is one of the most fundamental and important components in Minimally Invasive Surgery (MIS), nevertheless the traditional manual holding paradigm may easily bring fatigue to surgical assistants, and misunderstanding between surgeons also hinders assistants to provide a high-quality FOV. Targeting this problem, we here present a data-driven framework to realize an automated laparoscopic optimal FOV control. To achieve this goal, we offline learn a motion strategy of laparoscope relative to the surgeon’s hand-held surgical tool from our in-house surgical videos, developing our control domain knowledge and an optimal view generator. To adjust the laparoscope online, we first adopt a learning-based method to segment the two-dimensional (2D) position of the surgical tool, and further leverage this outcome to obtain its scale-aware depth from dense depth estimation results calculated by our novel unsupervised RoboDepth model only with the monocular camera feedback, hence in return fusing the above real-time 3D position into our control loop. To eliminate the misorientation of FOV caused by Remote Center of Motion (RCM) constraints when moving the laparoscope, we propose a novel rotation constraint using an affine map to minimize the visual warping problem, and a null-space controller is also embedded into the framework to optimize all types of errors in a unified and decoupled manner. Experiments are conducted using Universal Robot (UR) and Karl Storz Laparoscope/Instruments, which prove the feasibility of our domain knowledge and learning enabled framework for automated camera control.

Automatic surgical gesture recognition is fundamentally important to enable intelligent cognitive assistance in robotic surgery. With recent advancement in robot-assisted minimally invasive surgery, rich information including surgical videos and robotic kinematics can be recorded, which provide complementary knowledge for understanding surgical gestures. However, existing methods either solely adopt uni-modal data or directly concatenate multi-modal representations, which can not sufficiently exploit the informative correlations inherent in visual and kinematics data to boost gesture recognition accuracies. In this regard, we propose a novel online approach of multi-modal relational graph network (i.e., MRG-Net) to dynamically integrate visual and kinematics information through interactive message propagation in the latent feature space. In specific, we first extract embeddings from video and kinematics sequences with temporal convolutional networks and LSTM units. Next, we identify multi-relations in these multi-modal embeddings and leverage them through a hierarchical relational graph learning module. The effectiveness of our method is demonstrated with state-of-the-art results on the public JIGSAWS dataset, outperforming current uni-modal and multi-modal methods on both suturing and knot typing tasks. Furthermore, we validated our method on in-house visual-kinematics datasets collected with da Vinci Research Kit (dVRK) platforms in two centers, with consistent promising performance achieved. Our code and data are released at: https://www.cse.cuhk.edu.hk/~yhlong/mrgnet.html.

Surgical instrument segmentation in robot-assisted surgery (RAS) - especially that using learning-based models - relies on the assumption that training and testing videos are sampled from the same domain. However, it is impractical and expensive to collect and annotate sufficient data from every new domain. To greatly increase the label efficiency, we explore a new problem, i.e., adaptive instrument segmentation, which is to effectively adapt one source model to new robotic surgical videos from multiple target domains, only given the annotated instruments in the first frame. We propose MDAL, a meta-learning based dynamic online adaptive learning scheme with a two-stage framework to fast adapt the model parameters on the first frame and partial subsequent frames while predicting the results. MDAL learns the general knowledge of instruments and the fast adaptation ability through the video-specific meta-learning paradigm. The added gradient gate excludes the noisy supervision from pseudo masks for dynamic online adaptation on target videos. We demonstrate empirically that MDAL outperforms other state-of-the-art methods on two datasets (including a real-world RAS dataset). The promising performance on ex-vivo scenes also benefits the downstream tasks such as robot-assisted suturing and camera control.

Autonomous surgical execution relieves tedious routines and surgeon’s fatigue. Recent learning-based methods, especially reinforcement learning (RL) based methods, achieve promising performance for dexterous manipulation, which usually requires the simulation to collect data efficiently and reduce the hardware cost. The existing learning-based simulation platforms for medical robots suffer from limited scenarios and simplified physical interactions, which degrades the real-world performance of learned policies. In this work, we designed SurRoL, an RL-centered simulation platform for surgical robot learning compatible with the da Vinci Research Kit (dVRK). The designed SurRoL integrates a user-friendly RL library for algorithm development and a real-time physics engine, which is able to support more PSM/ECM scenarios and more realistic physical interactions. Ten learning-based surgical tasks are built in the platform, which are common in the real autonomous surgical execution. We evaluate SurRoL using RL algorithms in simulation, provide in-depth analysis, deploy the trained policies on the real dVRK, and show that our SurRoL achieves better transferability in the real world.

This paper presents a model-free approach to automate folding of a deformable object with robot manipulators, where its surface was labelled with markers to facilitate vision-based control and alignment. While performing the task involves solving nonconvex or nonlinear terms, in this paper, linearization was first performed to approximate the problem. By using the Levenberg-Marquardt algorithm, the task of folding a deformable thin object can be reformulated as a convex optimization problem. The mapping relationship between the motions of markers on the image and the joint inputs of the robot manipulator was evaluated through a Jacobian matrix. To account for the uncertainty in the matrix due to the deformable object, a two-stage evaluation scheme, which consists of approximate-rigidity rule and Broyden-update rule, was performed. Proper constraints were also added to avoid causing damage to the object. The performance and the robustness of the proposed approach were examined through simulation using Bullet simulator. The video of the simulation can be retrieved from the attachment. The results confirm that the thin object can be precisely folded together based on different markers labelled on the surface.

Surgical knot tying is one of the most fundamental and important procedures in surgery, and a high-quality knot can significantly benefit the postoperative recovery of the patient. However, a longtime operation may easily cause fatigue to surgeons, especially during the tedious wound closure task. In this paper, we present a vision-based method to automate the suture thread grasping, which is a sub-task in surgical knot tying and an intermediate step between the stitching and looping manipulations. To achieve this goal, the acquisition of a suture's three-dimensional (3D) information is critical. Towards this objective, we adopt a transfer-learning strategy first to fine-tune a pre-trained model by learning the information from large legacy surgical data and images obtained by the onsite equipment. Thus, a robust suture segmentation can be achieved regardless of inherent environment noises. We further leverage a searching strategy with termination policies for a suture's sequence inference based on the analysis of multiple topologies. Exact results of the pixel-level sequence along a suture can be obtained, and they can be further applied for a 3D shape reconstruction using our optimized shortest path approach. The grasping point considering the suturing criterion can be ultimately acquired. Experiments regarding the suture 2D segmentation and ordering sequence inference under environmental noises were extensively evaluated. Results related to the automated grasping operation were demonstrated by simulations in V-REP and by robot experiments using Universal Robot (UR) together with the da Vinci Research Kit (dVRK) adopting our learning-driven framework.

A vision-based system presents one of the most reliable methods for achieving an automated robot-assisted manipulation associated with surgical knot tying. However, some challenges in suture thread detection and automated suture thread grasping significantly hinder the realization of a fully automated surgical knot tying. In this article, we propose a novel algorithm that can be used for computing the 3-D coordinates of a suture thread in knot tying. After proper training with our data set, we built a deep-learning model for accurately locating the suture's tip. By applying a Hessian-based filter with multiscale parameters, the environmental noises can be eliminated while preserving the suture thread information. A multistencils fast marching method was then employed to segment the suture thread, and a precise stereomatching algorithm was implemented to compute the 3-D coordinates of this thread. Experiments associated with the precision of the deep-learning model, the robustness of the 2-D segmentation approach, and the overall accuracy of 3-D coordinate computation of the suture thread were conducted in various scenarios, and the results quantitatively validate the feasibility and reliability of the entire scheme for automated 3-D shape reconstruction.

The ability to patterning cells is an important technique to facilitate cell-based assay and characterization. In this paper, an automated cell patterning system was developed for the fabrication of large-scale cell patterns. To resolve the challenge of the limited printable area, the cell-printing microchip and the substrate were mounted on the movable stages of the system, and large-scale cell patterns were realized through coordination between the stages. An autofocusing technique was integrated in the system to evaluate the gap between the microchip and the substrate. In order to enhance the performance of the patterning system, different experimental parameters, including the velocity of the moving stage, were examined. Yeast cells suspending in 6-aminohexanoic acid (AHA) solution were considered in this study, and a sequence of characters was successfully printed using the proposed system. The results confirm that this system offers an automatic method with high flexibility to construct large-scale cell patterns for various applications.

Robot-assisted surgery has revolutionized the field of surgery over the past few decades. Despite many successes, achieving full automation in these surgeries remains a challenging task. In this paper, a dynamic approach is proposed to automate knot tying with an in-house robot vision system. Through efficient path planning and coordination between two grippers, the workspace required for constructing the suture loop can be reduced while issues such as suture slippage and collisions between instruments can be eliminated. Visual images were employed to monitor the two grippers in real time and their positions were evaluated using transformation matrices obtained experimentally. A linear quadratic control scheme was applied to optimize the tracking performance of the two grippers. From the experiments, this visual evaluation method can achieve a position accuracy of 1 mm in the workspace. The proposed algorithm was evaluated and automatic suture looping operation was successfully performed in all six trials. Different parameters in the control scheme were also examined by introducing external impulse disturbances during the knot-tying process. This proposed knot-tying approach demonstrates a simple and efficient way to construct a suture knot in a minimal workspace. Note to Practitioners-A surgical operation usually takes several hours to complete, which is a test of surgeon's endurance. To better assist surgeons, several repetitive surgical tasks can be performed automatically with a robotic system. This paper presents a new method for robot-assisted surgical knot tying in a confined environment. Through simultaneous manipulation and trajectory planning of the two surgical grippers, suture loops were constructed while keeping the suture in tension. Images were employed to guide the gripers in completing the knot-tying process with high precision. Experiments were conducted to demonstrate the automated suture looping operation with the system and the proposed method. The robustness of the method was also examined by introducing impulse disturbances and occlusions during the experiments. This proposed method does not require complicated hardware setup, allowing easy implementations on various surgical systems.

Robot assisted surgery has become increasingly popular with the rapid development of more sophisticated robotic systems. Nevertheless, there are still many tedious surgical tasks that remain to be performed manually by surgeons with high supervisions. One example is to tie a surgical knot during the surgery, which includes the processes of needle threading, suture looping, suture tail grasping and suture pulling. In this research, we present a new method for robotic suture looping process. This method is based on a spatial trajectory planning for the surgical instruments, where challenges such as suture slippage and potential collisions between the instruments could be eliminated. In contrast to conventional looping process, a dynamic control scheme is proposed to optimize the time and workspace required to complete the process. Vision information is incorporated in order to enable closed loop control of the robotic system with Linear Quadratic (LQ) controller. A series of experiments were conducted in order to examine the performance of the proposed control scheme and the results confirmed that surgical knots can be successfully tied robotically, offering new insights to the field of automated robotassisted surgery.