Continuum robots, owing to their inherent compliance, have become essential in endoscopic surgical procedures, such as mucosal ablation. However, the prevalent design of endoscopic manipulators, which typically features only a single active bending segment, often results in limited dexterity and accessibility. Additionally, the incorporation of variable stiffness in these robots has attracted significant interest, with the aim to improve manipulation capabilities in confined spaces. In the paper, we propose a novel variable-length continuum robot with variable stiffness for endoscopic surgery. The robot’s stiffness can be altered either by modifying the catheter’s length or solid-liquid transition of low-melting-point alloy (LMPA). The design and fabrication methods of the robot are meticulously detailed. Additionally, a quasi-static stiffness model along with a learning-based stiffness compensation approach for accurate stiffness estimation are proposed. Leveraging this model, a contact force controller is designed for ablation procedure. The experimental results show that our robot possesses good flexibility and accessibility, making it highly adept at manipulating in confined spaces. Its variable stiffness feature significantly enhances its ability to counteract external disturbance and prevent tip deformation (with a average position change of 1.1mm). Finally, through force control experiments and a surgical demonstration in a gastrointestinal model, we have further validated the robot’s applicability in surgical contexts. Note to Practitioners—This paper proposed a variable-length continuum robot with variable stiffness for endoscopic surgery. The robot can achieve axial elongation and omnidirectional bending motion, having better dexterity and accessibility than traditional medical continuum robots with one active bending segment. The robot’s stiffness can be adjusted by the length changes or solid-liquid transition of low-melting-point alloy (LMPA). Besides, an accurate stiffness model and a contact force controller are proposed for endoscopic ablation surgery. By experimental results, the robot shows high flexibility and accessibility, allowing access to confined spaces for manipulation, and good control accuracy and variable stiffness capability for endoscopic surgery.

X-ray to CT registration is an essential technique to provide on-site guidance for clinicians and medical robots by aligning preoperative information with intraoperative images. Current methods focus on local registration with small capture ranges and necessitate a manual initial alignment before precise registration. Some existing global methods are likely to fail in thoracic surgeries because of the respiratory motion and the nearly colinear nature of vertebrae landmarks. In this study, we propose a vertebrae-based global X-ray to CT registration method with the assistance of clinical setups for thoracic surgeries. Firstly, vertebrae centroids are automatically localized by CNN-based networks in CT and X-ray for establishing 2D/3-D correspondences. Then, inspired by clinical setup, we address the degradation of colinear landmarks of 6-DoF pose estimation by introducing a 4-DoF solver. Considering the inaccurate priori and landmark mislocalization, the solver is embedded into the Adaptive Error-Aware Estimator (AE2) to simultaneously estimate weights and aggregate candidate poses. Finally, the whole method is trained in an end-to-end manner for better performance. Evaluations on both the public LIDC-IDRI dataset and clinical dataset demonstrate that our method outperforms existing optimization-based and learningbased approaches in terms of registration accuracy and success rate. Our code: https://github.com/LiuLiluZJU/2P-AE2

Common medical image segmentation tasks require large training datasets with pixel-level annotations which are very expensive and time-consuming to prepare. To overcome such limitation and achieve the desired segmentation accuracy, a novel Weakly-Interactive-Mixed Learning (WIML) framework is proposed by efficiently using weak labels. On one hand, utilize weak labels to reduce annotation time for high-quality strong labels by designing a Weakly-Interactive Annotation (WIA) part of the WIML which prudently introduces interactive learning into the weakly-supervised segmentation strategy. On the other hand, utilize weak labels and very few strong labels to achieve desired segmentation accuracy by designing a Mixed-Supervised Learning (MSL) part of the WIML which can boost the segmentation accuracy by providing strong prior knowledge during training. Besides, a multi-task Full-Parameter-Sharing Network (FPSNet) is proposed to better implement this framework. Specifically, to further reduce annotation time, attention modules (scSE) are integrated into FPSNet to improve the class activation map (CAM) performance for the first time. To further improve segmentation accuracy, a Full-Parameter-Sharing (FPS) strategy is designed in FPSNet to alleviate the overfitting of the segmentation task supervised by very few strong labels. The proposed method is validated on the BraTS 2019 and LiTS 2017 datasets, and experiments demonstrate that the proposed method WIML-FPSNet outperforms several state-of-the-art segmentation methods with minimal annotation efforts.

Current RGB-based 6D object pose estimation methods have achieved noticeable performance on datasets and real world applications. However, predicting 6D pose from single 2D image features is susceptible to disturbance from changing of environment and textureless or resemblant object surfaces. Hence, RGB-based methods generally achieve less competitive results than RGBD-based methods, which deploy both image features and 3D structure features. To narrow down this performance gap, this paper proposes a framework for 6D object pose estimation that learns implicit 3D information from 2 RGB images. Combining the learned 3D information and 2D image features, we establish more stable correspondence between the scene and the object models. To seek for the methods best utilizing 3D information from RGB inputs, we conduct an investigation on three different approaches, including Early-Fusion, Mid-Fusion, and Late-Fusion. We ascertain the Mid-Fusion approach is the best approach to restore the most precise 3D keypoints useful for object pose estimation. The experiments show that our method outperforms state-of-the-art RGB-based methods, and achieves comparable results with RGBD-based methods.

Visual localization for planar moving robot is important to various indoor service robotic applications. To handle the textureless areas and frequent human activities in indoor environments, a novel robust visual localization algorithm which leverages dense correspondence and sparse depth for planar moving robot is proposed. The key component is a minimal solution which computes the absolute camera pose with one 3D-2D correspondence and one 2D-2D correspondence. The advantages are obvious in two aspects. First, the robustness is enhanced as the sample set for pose estimation is maximal by utilizing all correspondences with or without depth. Second, no extra effort for dense map construction is required to exploit dense correspondences for handling textureless and repetitive texture scenes. That is meaningful as building a dense map is computational expensive especially in large scale. Moreover, a probabilistic analysis among different solutions is presented and an automatic solution selection mechanism is designed to maximize the success rate by selecting appropriate solutions in different environmental characteristics. Finally, a complete visual localization pipeline considering situations from the perspective of correspondence and depth density is summarized and validated on both simulation and public real-world indoor localization dataset.