Continuum robots offer high flexibility and multiple degrees of freedom, making them ideal for navigating narrow lumens. However, accurately modeling their behavior under large deformations and frequent environmental contacts remains challenging. Current methods for solving the deformation of these robots, such as the Model Order Reduction and Gauss-Seidel (GS) methods, suffer from significant drawbacks. They experience reduced computational speed as the number of contact points increases and struggle to balance speed with model accuracy. To overcome these limitations, we introduce a novel finite element method (FEM) named Acc-FEM. Acc-FEM employs a large deformation quasi-static finite element model and integrates an accelerated solver scheme to handle multi-contact simulations efficiently. Additionally, it utilizes parallel computing with Graphics Processing Units (GPU) for real-time updates of the finite element models and collision detection. Extensive numerical experiments demonstrate that Acc-Fem significantly improves computational efficiency in modeling continuum robots with multiple contacts while achieving satisfactory accuracy, addressing the deficiencies of existing methods.

Localizing the bronchoscope in real time is essential for ensuring intervention quality. However, most existing vision-based methods struggle to balance between speed and generalization. To address these challenges, we present BronchoTrack, an innovative real-time framework for accurate branch-level localization, encompassing lumen detection, tracking, and airway association. To achieve real-time performance, we employ benchmark light weight detector for efficient lumen detection. We firstly introduce multi-object tracking to bronchoscopic localization, mitigating temporal confusion in lumen identification caused by rapid bronchoscope movement and complex airway structures. To ensure generalization across patient cases, we propose a training-free detection-airway association method based on a semantic airway graph that encodes the hierarchy of bronchial tree structures. Experiments on 11 patient datasets demonstrate BronchoTrack's localization accuracy of 81.72%, while accessing up to the 6th generation of airways. Furthermore, we tested BronchoTrack in an in-vivo animal study using a porcine model, where it localized the bronchoscope into the 8th generation airway successfully. Experimental evaluation underscores BronchoTrack's real-time performance in both satisfying accuracy and generalization, demonstrating its potential for clinical applications.

Traditional rigid endoscopes have challenges in flexibly treating tumors located deep in the brain, and low operability and fixed viewing angles limit its development. This study introduces a novel dual-segment flexible robotic endoscope MicroNeuro, designed to perform biopsies with dexterous surgical manipulation deep in the brain. Taking into account the uncertainty of the control model, an image-based visual servoing with online robot Jacobian estimation has been implemented to enhance motion accuracy. Furthermore, the application of model predictive control with constraints significantly bolsters the flexible robot’s ability to adaptively track mobile objects and resist external interference. Experimental results underscore that the proposed control system enhances motion stability and precision. Phantom testing substantiates its considerable potential for deployment in neurosurgery.

Transorbital Neuro Surgery (TNS) offers a novel treatment towards the lesion inside skull pursuing minimal invasiveness. Most conventional TNS tools are rigid and straight, limiting the dexterity and accessibility in passing a small port. Bendable and steerable surgical tools provides an alternative for this issue. In this work, we proposed a dual-segment slender surgical robot arm for TNS, which is a Co-Axial Slender Tubular robot (CAST), and modelled it using novel approaches. Another contribution is tendon-mortise shaped slits along the axial direction, enhancing the overall stiffness. The bending of CAST is actuated by pushing/pulling distance, and the maximum diameter is only 1.7mm with high dexterity after mounting on a rigid robot arm. Experiments demonstrates that the proposed the slit design doubles the stiffness properties compared to traditional rectangle slit designs. The path-following task shows that the position error was maximally 3mm in open-looped control. Test on a skull model demonstrates that the whole system could successfully perform electrocoagulation procedure inside the depth of skull in a robotized manner effectively.

Real-time 6 DOF localization of bronchoscopes is crucial for enhancing intervention quality. However, current vision-based technologies struggle to balance between generalization to unseen data and computational speed. In this study, we propose a Depth-based Dual-Loop framework for real-time Visually Navigated Bronchoscopy (DD-VNB) that can generalize across patient cases without the need of re-training. The DD-VNB framework integrates two key modules: depth estimation and dual-loop localization. To address the domain gap among patients, we propose a knowledge-embedded depth estimation network that maps endoscope frames to depth, ensuring generalization by eliminating patient-specific textures. The network embeds view synthesis knowledge into a cycle adversarial architecture for scale-constrained monocular depth estimation. For real-time performance, our localization module embeds a fast ego-motion estimation network into the loop of depth registration. The ego-motion inference network estimates the pose change of the bronchoscope in high frequency while depth registration against the pre-operative 3D model provides absolute pose periodically. Specifically, the relative pose changes are fed into the registration process as the initial guess to boost its accuracy and speed. Experiments on phantom and in-vivo data from patients demonstrate the effectiveness of our framework: 1) monocular depth estimation outperforms SOTA, 2) localization achieves an accuracy of Absolute Tracking Error (ATE) of 4.7 ± 3.17 mm in phantom and 6.49 ± 3.88 mm in patient data, 3) with a frame-rate approaching video capture speed, 4) without the necessity of case-wise network retraining. The framework’s superior speed and accuracy demonstrate its promising clinical potential for real-time bronchoscopic navigation.

Bronchoscopy plays a significant role in the early diagnosis and treatment of lung diseases. This process demands physicians to maneuver the flexible endoscope for reaching distal lesions, particularly requiring substantial expertise when examining the airways of the upper lung lobe. With the development of artificial intelligence and robotics, reinforcement learning (RL) method has been applied to the manipulation of interventional surgical robots. However, unlike human physicians who utilize multimodal information, most of the current RL methods rely on a single modality, limiting their performance. In this paper, we propose BronchoCopilot, a multimodal RL agent designed to acquire manipulation skills for autonomous bronchoscopy. BronchoCopilot specifically integrates images from the bronchoscope camera and estimated robot poses, aiming for a higher success rate within challenging airway environment. We employ auxiliary reconstruction tasks to compress multimodal data and utilize attention mechanisms to achieve an efficient latent representation of this data, serving as input for the RL module. This framework adopts a stepwise training and fine-tuning approach to mitigate the challenges of training difficulty. Our evaluation in the realistic simulation environment reveals that BronchoCopilot, by effectively harnessing multimodal information, attains a success rate of approximately 90% in fifth generation airways with consistent movements. Additionally, it demonstrates a robust capacity to adapt to diverse cases.

Current imaging of the artery relies primarily on computed tomography angiography (CTA), which requires contrast injections and exposure to radiation. In this paper, we present a method for fully autonomous artery 3D image acquisition using a linear ultrasound (US) probe and a 6 DoFs robot arm with a 3D camera. Robotic vessel acquisition can minimize tissue deformation and permit the reproduction of scans. Additionally, the robotic-based acquisition can provide more precise vessel position data that can be utilized for 3D reconstruction as a preoperative image. The first scanning point is determined by the 3D camera using a neural network for leg area estimation. A visual servo algorithm adjusts the in-plane motions using a cross-sectional vessel segmentation produced by a neural network with a UNet structure, while a US confidence map regulates the in-plane rotation. The robot is equipped with impedance control to maintain a constant and safe scan. Experiments on a leg phantom and a volunteer indicate that the robot can follow the vessel and modify its position to provide a sharper US image. The average error of phantom scanning in y-axis and z-axis are 0.2536mm and 0.2928mm, respectively, while the root means square error (RMSE) of contact force in the volunteer experiment is 0.2664N. In addition, a 3D vessel reconstruction demonstrates the possibility of robotic US acquisition as a preoperative image.