Registration of intraoperative ultrasound (iUS) with preoperative CT represents a significant yet challenging task in the context of radiation-free spinal surgical navigation. The presence of thickness response artifacts in US images poses a considerable obstacle to the accurate extraction of bone boundaries. Furthermore, US-CT registration typically necessitates the detection and correspondence of high-quality landmarks at the initial stage. This can be accomplished by surgeons who have undergone extensive training in the localization of standard spinal US views, enabling them to identify key vertebral landmarks for subsequent precise registration. In this paper, we propose a fully automated iUS registration method that employs a limited number of spinal US views as observation objects. Specifically, three-dimensional vertebral meshes segmented from the preoperative CT images are superimposed on the US images and then fed to the reinforcement learning (RL) agent for sequential decision-making. The proposed method achieves fully automatic US-CT registration without relying on prespecified initialization. This is achieved by training the agent to approach bone surfaces on several randomly selected 2D US views. The instability of RL-based iUS registration is primarily attributable to the difficulty of correlating long-range information within the neural network. To address this issue, we propose a Field of View Separation (FoVS) module. The proposed approach employs separate encoders for US and mesh images, followed by cross-attention aggregation, which facilitates information flow between non-adjacent pixels. This approach enables pretraining of feature extraction on distinct encoders and the application of supplementary loss for enhanced feature matching precision, thereby significantly improving the learning capability and stability of the network. Furthermore, a refinement module is introduced to correct the results of the RL registration, which improves the stability of the registration process. To ascertain the efficacy of each module, action, and auxiliary task, comprehensive experiments are conducted. The results demonstrate that the performance of the RL agent is enhanced by the associated modules and auxiliary tasks. The registration exhibited an angular error of$8.83 \; \pm \; 4.69$degrees and a translational error of$3.34 \; \pm \; 1.42$mm, achieving the state-of-the-art (SOTA) results. It is noteworthy that fine-tuning the model prior to the surgical phase can significantly reduce the registration error, which is a promising outcome for its clinical translation.Note to Practitioners—The objective of this study is to address the issue of image registration using iUS in conjunction with preoperative CT scans in the context of spine surgery. The current 2D/3D image registration methods are constrained by several limitations. Firstly, they often exhibit reduced accuracy, and require high-quality images in substantial quantities. Secondly, there is a lack of effective mechanisms to rectify errors identified after the registration process. This paper proposes a fully automated registration framework based on RL, which incorporates image rendering and mesh clipping to enable continuous adjustment of the pose of 3D data, thereby facilitating 2D/3D registration. The framework employs the distinctive attributes of iUS images and incorporates a refinement module to evaluate registration accuracy, thereby facilitating the rectification of any registration issues. The proposed framework was tested on both sheep lumbar subjects and human lumbar phantoms, demonstrating the highest level of performance to date and indicating its potential for integration into surgical navigation systems.

The fiber-guided laser scanner has been widely adopted in endoscopic microsurgeries due to its outstanding precision, swiftness, and dexterity for incision, ablation, and photocoagulation of some difficult-to-reach tissues, e.g., the laryngeal, respiratory, and alimentary sites. However, fabricating and actuating such precision surgical instruments are challenging, considering the strictly limited workspace and the high-performance requirements. This article proposes a compact fiber-guided laser scanner with two rotational degrees of freedom (DOFs), one of which is actuated with the stepper motor and the other controlled by electromagnetism. The novel design of the electromagnetically driven system allows the laser scanner for a more compact radial dimension while maintaining high precision in trajectory tracking tasks. In addition, we develop a robust control strategy based on visual feedback to achieve autonomous detecting and tracking of the trajectory targets. In vitro tests are subsequently performed to validate the efficacy and accuracy of our proposed system. The experimental results reveal that the average tracking error of our laser steering system is as low as$ {279.29~{\mu~{\mathrm {m}}}}$, which is deemed satisfactory in practical endoscopic microsurgeries. Note to Practitioners—This article is motivated by the unmet needs of in vivo laser endoscopic microsurgery in an automated fashion, and simultaneously the proposed methodology can be applied to other laser scanning-related scenarios. Existing solutions to in vivo laser scanning generally are associated with complex micro-electromechanical systems (MEMS), and hence high costs and relatively bulky footprint. Moreover, existing manual control schemes cannot enable high-precision laser scanning in an automated fashion. This article demonstrates a novel approach incorporating mechanism and electromagnetism to actuate the laser scanner, whereby achieving a highly compact design at low costs. Further, a closed-loop control strategy is developed for such an electromagnetically driven laser scanner, enhancing its scanning accuracy and reliability. This article theoretically analyzes the mechanics and electromagnetics of the laser scanner. Subsequently, it proposes a closed-loop control scheme through endoscopic visual feedback, allowing the laser scanner for autonomous target tracking. The pilot in vitro experiments demonstrate the feasibility and reliability of this holistic approach. Ultimately, the design and control of the electromagnetically driven laser scanner would be further optimized and validated for its laser ablation performance on ex vivo tissues or animal models in the future.

In medical robotics and image-guided surgery (IGS), registration is needed in order to align together the coordinate frames of robots, medical imaging modalities, surgical tools, and patients. Existing registration algorithms often assume one point set to be a noise-free model while the other to contain noise and outliers. However, in real scenarios, noise and outliers can exist in both point sets to be registered. To eliminate the above-mentioned challenge, in this paper, we formally formulate the Bi-directional Generalised Rigid Point Set Registration (Bi-GRPSR) problem where normal vectors are adopted, bi-directional probability density function (PDFs) and Hybrid Mixture Models (HMMs) are constructed to derive the objective function. Bi-GRPSR considering anisotropic positional noise is thus cast as a maximum likelihood estimation (MLE) problem, which is solved by the proposed Bi-directional Generalised Anisotropic Coherent Point Drift (Bi-AGCPD) where spatially nearby points are considered to move coherently and iterative expectation maximization (EM) steps are involved. Experimental results on two human bone point sets, under different settings of noise, outliers, and overlapping ratios, validate the effectiveness and improvements of Bi-AGCPD over existing probabilistic and learning-based methods.Note to Practitioners—This paper presents a novel rigid point set registration method that explicitly takes the anisotropic noise in both point sets into account. The proposed framework first formulates the probability density functions of generalised points in a bi-directional way, with which the bi-directional hybrid mixture model is built. The resulting objective function is minimised with the expectation maximisation technique. The algorithms are essential for real-world applications in that noise usually exists in both spaces to be registered and is generally anisotropic. The proposed method has demonstrated promising results on two human femur bone models, which indicates the great potential for it to be readily applied to related applications, especially medical scenarios, given that the two point sets are coarsely aligned. Future work will extend the presented method into scenarios of global registration.

This paper proposes a new approach to achieve direct visual servoing (DVS) based on discrete orthogonal moments (DOMs). DVS is performed in such a way that the extraction of geometric primitives, matching, and tracking steps in the conventional feature-based visual servoing pipeline can be bypassed. Although DVS enables highly precise positioning, it suffers from a limited convergence domain and poor robustness due to the extreme nonlinearity of the cost function to be minimized and the presence of redundant data between visual features. To tackle these issues, we propose a generic and augmented framework that considers DOMs as visual features. By using the Tchebichef, Krawtchouk, and Hahn moments as examples, we not only present the strategies for adaptively tuning the parameters and order of the visual features but also exhibit an analytical formulation of the associated interaction matrix. Simulations demonstrate the robustness and accuracy of our approach, as well as its advantages over the state-of-the-art. Real-world experiments have also been performed to validate the effectiveness of our approach.

In medical robotics and image-guided surgery (IGS), registration is needed in order to align together the coordinate frames of robots, medical imaging modalities, surgical tools, and patients. Existing registration algorithms often assume one point set to be a noise-free model while the other to contain noise and outliers. However, in real scenarios, noise and outliers can exist in both point sets to be registered. To eliminate the above-mentioned challenge, in this paper, we formally formulate the Bi-directional Generalised Rigid Point Set Registration (Bi-GRPSR) problem where normal vectors are adopted, bi-directional probability density function (PDFs) and Hybrid Mixture Models (HMMs) are constructed to derive the objective function. Bi-GRPSR considering anisotropic positional noise is thus cast as a maximum likelihood estimation (MLE) problem, which is solved by the proposed Bi-directional Generalised Anisotropic Coherent Point Drift (Bi-AGCPD) where spatially nearby points are considered to move coherently and iterative expectation maximization (EM) steps are involved. Experimental results on two human bone point sets, under different settings of noise, outliers, and overlapping ratios, validate the effectiveness and improvements of Bi-AGCPD over existing probabilistic and learning-based methods.

Autonomous ultrasound (US) acquisition is an important yet challenging task, as it involves interpretation of the highly complex and variable images and their spatial relationships. In this work, we propose a deep reinforcement learning framework to autonomously control the 6-D pose of a virtual US probe based on real-time image feedback to navigate towards the standard scan planes under the restrictions in real-world US scans. Furthermore, we propose a confidence-based approach to encode the optimization of image quality in the learning process. We validate our method in a simulation environment built with real-world data collected in the US imaging of the spine. Experimental results demonstrate that our method can perform reproducible US probe navigation towards the standard scan plane with an accuracy of 4.91mm/4.65° in the intra-patient setting, and accomplish the task in the intra- and inter-patient settings with a success rate of 92% and 46%, respectively. The results also show that the introduction of image quality optimization in our method can effectively improve the navigation performance.

Active wireless capsule endoscopy (WCE) under magnetic actuation is a promising technology to reduce the inspection time and relieve the burden of physicians. In this paper, we propose a reciprocally rotating magnetic actuation method for trajectory following of a capsule and develop its dynamic model. For the trajectory following task, we investigate the closed-loop tracking control strategies based on different controllers to actuate the capsule in the complex environments. The effectiveness of our method is validated in extensive experiments in a simulation environment as well as in an ex-vivo pig colon. The results demonstrate that the proposed method can accurately and efficiently actuate the capsule to follow the desired trajectory in the complex environments, achieving tracking errors on the order of millimeter. Moreover, the experiments on the ex-vivo pig colon show that the proposed reciprocally rotating magnetic actuation method has the potential to reduce the clinical risks and improve the safety and clinical acceptability of this technology.

Metastatic involvement of lymph nodes is one of the most important prognostic variables for many cancers. Several deep learning based algorithms have been developed to segment metastatic regions in pathological images to help predict prognosis. However, the training of these methods requires a large amount of annotated data, and the labeling task is an extremely time-consuming process for human annotators. In order to reduce the annotation burden, we for the first time propose a semi-automatic annotation method (PiPo-Net) for the labeling of pathological images. The method is comprised of two subnetworks, a pixel-wise segmentation network (Pi-Net) and a polygon-based annotation network (Po-Net). The Pi-Net adopts an improved encoder-decoder architecture and can effectively aggregate multi-scale image features. The Po-Net is built on the Pi-Net and leverages a two-layer recurrent neural network to generate tight-bounded polygons for the metastatic regions. Corresponding to the proposed network architecture, a loss function called PiPo-loss is introduced to help optimize the whole network. The main advantage of our method is that it integrates human annotators into the prediction loop, allowing to iteratively refine the predictions according to the suggestions from human annotators. We evaluate our method on Camelyon16 database and achieve a Dice score of 91% in the initial annotation attempt. We also demonstrate the effectiveness of the human-network collaborative annotation, which achieves promising labeling results, verifying the advantages of our proposed method.

Flexible manipulator enables curvilinear accessibility through small incisions or natural orifices for minimally invasive surgery and diagnosis, which makes it a good choice for minimally invasive surgery. In order to control the robot precisely and safely, the real-time position and shape information of the robot need to be measured well. In this paper, we propose a magnetic tracking based tip pose and shape detection method for wire driven flexible robots. A permanent magnet is mounted at the distal end of the robot. Its magnetic field can be sensed with a sensor array. Therefore, position and orientation of the tip can be estimated utilizing the tracking method. A shape sensing algorithm is then carried out to estimate the real-time shape based on the tip pose. With the tip pose and shape display in the reconstructed visual environment, navigation can be achieved. This method provides the advantages that no sensors are needed to mount on the robot and has no line-of-sight problem. Experimental results verified the feasibility of the proposed method. A navigation error of 1.9mm is achieved.