Capsule endoscopes, predominantly serving diagnostic functions, provide lucid internal imagery but are devoid of surgical or therapeutic capabilities. Consequently, despite lesion detection, physicians frequently resort to traditional endoscopic or open surgical procedures for treatment, resulting in more complex, potentially risky interventions. To surmount these limitations, this study introduces a chained flexible capsule endoscope (FCE) design concept, specifically conceived to navigate the inherent volume constraints of capsule endoscopes whilst augmenting their therapeutic functionalities. The FCE’s distinctive flexibility originates from a conventional rotating joint design and the incision pattern in the flexible material. In vitro experiments validated the passive navigation ability of the FCE in rugged intestinal tracts. Further, the FCE demonstrates consistent reptile-like peristalsis under the influence of an external magnetic field, and possesses the capability for film expansion and disintegration under high-frequency electromagnetic stimulation. These findings illuminate a promising path toward amplifying the therapeutic capacities of capsule endoscopes without necessitating a size compromise.

Human-aware path planner is essential for achieving harmonious coexistence between humans and robots in highly dynamic environments. In this paper, we propose an integrated framework to find the optimal path in the complex environment with considering collision risk, social norms, and crowded areas. In the proposed framework, a general dynamic group model (g-space) based on the Gaussian Mixed Model (GMM) is proposed as the social norms of dynamic groups, which not only considers the factors of humans (e.g., pose, quantity, distribution, psychology) but also establishes the proximity and human interacting constraints of dynamic groups. An integrated Collision Risk and Human Space (CR&HS) model is applied to achieve human-acceptable behaviors, in which both collision avoidance, human comfort, and interference-free constraints have been involved. Moreover, an Improved Virtual Doppler Method (IVDM) has been used to realize safety navigation to avoid the robot falling into the crowded area. Finally, the proposed framework has been utilized with the sampling-based rapidly-exploring random tree. Experimental results demonstrate that the proposed method can generate the optimal human-aware collision-free path in complex environments. Note to Practitioners—This paper aims to plan an optimal trajectory for the robot in highly dynamic environments. In this field, it is still a challenging task to plan a trajectory with collision-free, human-aware, and crowd-aware. To do that, we present an integrated framework to generate the optimal trajectory by involving the collision risk, social norms, and human density. First, the g-space model is adopted as interference-free constraints of dynamic groups. The integrated knowledge fusion model (CR&HS) then penalizes the manners which have higher collision risk and adverse effects on human interaction or human comfortable. Besides, human motion and density are provided to a robot by IVDM. The proposed framework is utilized in the sampling-based rapidly-exploring random tree as the evaluation module. Finally, the feasibility and reliability of the proposed method have been verified by experiments in different simulated environments. The proposed framework can be applied in most mobile service robots to achieve human-friendly manners.

This paper proposes a network called points-guided sampling net (PGSN) to guide the sampling process in sampling-based motion planner by utilizing the geometric information of obstacles. The geometric information is extracted from the point cloud of obstacles. By analyzing the properties of the point cloud, we propose a VAE feature extraction net that incorporates the variational autoencoder (VAE) framework with unique architectures designed for point clouds. Furthermore, we design a multi-modal sampling net to model the probability distribution of the states based on training trajectories taken from different environments. Based on PGSN, we propose a sampling-based motion planning algorithm called the point-guided rapidly-exploring random tree (PG-RRT). Three experiments are conducted to verify the proposed PGSN: Exp I shows the proposed VAE feature extraction net can successfully extract geometric features from the inputted point cloud; Exp II verifies the multi-modal sampling net successfully chooses corresponding mode with respect to extracted features; Exp III demonstrates the efficacy of our PG-RRT algorithm by showing PG-RRT outperforms other algorithms. Moreover, we provide theoretical analysis and insights towards understanding our model. Note to Practitioners—Obstacles cause lots of the sampling space invalid, thus the traditional sampling-based motion planning (SBMP) algorithm is usually unable to generate a trajectory within a reasonable short period of time. To improve the success rate and efficiency of SBMP, this paper proposes a novel deep neural network called points-guided sampling net (PGSN). PGSN is designed to exploit: (1) environmental point clouds and (2) training trajectories from multiple environments with different obstacles. In the first step, the point clouds include important geometric information. To utilize this information, we adopt a variational autoencoder approach which combines an encoder and a decoder together to extract geometric features more accurately from point clouds. In the second step, trajectories from multiple environments have a multi-modal property which can be represented by a truncated multivariate Gaussian mixture model. We propose a multi-modal sampling net to learn optimal parameters of this model from the training trajectories, and to select corresponding mode based on the extracted features. Experiments demonstrate that the proposed algorithm is feasible and can achieve higher success rate than the state-of-the-art methods. Our method uses a single frame of point cloud to improve efficiency, therefore multiple point clouds from different perspective maybe needed when objects occlude with each other.

High-precision pose adjustment for the self-charging of mobile robots remains a significant challenge. Permanent magnet (PM)-based magnetic tracking technique is a promising technical solution, with occlusion-free and simultaneous positioning and orientation tracking. However, the superposition of the geomagnetic vector and the magnetic field vector generated by the PM leads to the degrading of magnetic tracking performance. Thus, a magnetic tracking technique with real-time geomagnetic vector separation is investigated in this study. Firstly, the environmental magnetic field is accurately modeled, consisting of the PM field, uniform disturbance field, and non-uniform disturbance field. For the uniform disturbance field, we combine it with the PM pose as unknown parameters to be estimated. For the non-uniform disturbance field, a robust kernel function is employed to diminish its influence on positioning performance. Finally, the PM pose and geomagnetic vector are simultaneously estimated by optimization algorithms. A docking experiment for self-charging mobile robots was carried out based on the proposed tracking technique. The robot can successfully recharge its battery with only one alignment operation, where the repeat parking accuracy at the anchor point is 1.38 mm and ±1.27°, respectively.

This paper proposes a new network for reconstructing multi-object point cloud. Different from previous networks which reconstruct multi-object point cloud as a whole, our network iteratively reconstructs each individual object point cloud from a frame of multi-object point cloud. To achieve this goal, we have designed MO-Transformer, a transformer-based autoregressive network. During training, MO-Transformer takes a frame of multi-object point cloud and individual object point clouds as input. During testing, MO-Transformer iteratively reconstructs individual object point clouds only based on the input multi-object point cloud. To train the proposed MO-Transformer, we design a new loss function called separate Chamfer distance (SCD). In addition, we prove that SCD is an upper bound of the traditional Chamfer distance calculated based on the entire multi-object point cloud. The reconstruction experiment verifies the efficacy of our network in multi-object point cloud reconstruction. Furthermore, the reconstruction experiment also investigates the effect of different dimensions using a series of datasets. The ablation study experiment verifies the necessity of SCD in training MO-Transformer.

Magnetically-actuated flexible endoscopes (MAFE) have been well used in minimally-invasive surgery because they can be steered by a magnetic field thus more flexible than traditional endoscopes. Model-free and uncalibrated visual-feedback control makes it possible to manipulate MAFE with a magnetic field without external tracking systems. Because no extra sensor is required to obtain position and posture information, the size of MAFE can be made smaller. However, the traditional control method focuses on 2DoF control, which lacks control over the posture of the end of MAFE. This may result in unnecessary contact between MAFE and tissue and cause injury during the advancement of the endoscope. In this letter, we propose algorithms to enhance the pose control of MAFE to 4DoF and 5DoF based on model-free and uncalibrated visual-feedback control. Experiments in structured environments verify that the control algorithms are able to realize 4DoF manual navigation and 5DoF automatic navigation.

Small-scale robots have great potential in minimally invasive surgery (MIS). In this paper, we propose an untethered magnetic gripper with small scale and build a double-magnet model for it. The gripper is 4.3mm long and its maximum width is 4mm. It contains a spindle and two magnets, which can achieve precise control of orientation, position and open angle with external magnetic driven field. As a result, it can perform operations such as transporting medicines in confined and constrained environments. Modeling and analysis of the magnetic gripper have been carried out. Relationship between the open angle and external magnetic field has been established. Kinematics model of the gripper has been built. A 3-axis Helmholtz-Maxwell coil system has been established to generate the magnetic field, in which orientation and open angle can be controlled with uniform magnetic field while position can be controlled with gradient field. The proposed gripper have been validated with phantom experiments. An opened angle control error of 0.63° and direction control error of 1.1° have been obtained.

Accurate position feedback in a wide range is critical for medical microrobotics and robot-assisted examinations, such as colonoscopy, bronchoscopy and capsule endoscopy examination. Among the many modalities of positioning feedback, magnetic tracking is a preferable method due to the unique advantages of free line of sight, free energy storage and untethered connection. However, the field strength of the magnetic source decreases with the third power of the distance, limiting the effectiveness of position feedback at long distances. In order to maintain a consistently high tracking accuracy in a broad area, this paper presents a new dynamic tracking solution by applying a movable sensor array. In this new solution, the tracking accuracy of the magnet is first determined and optimized within a short range. When the target microrobot carrying the magnet exceeds this optimized range, the sensor array is relocated by an external robotic arm to keep the target in the effective tracking range. Moreover, we also propose a multi-point locating algorithm to minimize the varying background noise. Experimental results show that the proposed method increases the range of magnetic tracking and achieves a satisfactory level of tracking accuracy, which demonstrates significant potentials to improve the position feedback of microrobots in medical applications.

The optical tracking system (OTS) plays a vital role in the computer-assisted surgical navigation process, whereas the performance of the commonly used binocular stereo vision is affected by the line-of-sight problem and limited workspace. Thus, this article proposed a prior knowledge-based multicamera reconstruction model (PKRM) to both expand the tracking workspace and improve the tracking robust and computational efficiency of OTS when working in unstructured clinical conditions. This reconstruction model inherits the advantages of the geometrical method, data-driven method, and gating technique (GT). First, we added the geometric principle as the prior knowledge to optimize the training of the multicamera OTS reconstruction model through the Lagrange multiplier method; hence, the prior knowledge feedforward NN (PKFNN) was built. Second, besides the training features, the state of camera (SOC) was extracted in advance to determine the NN structure using GT. According to the SOC feature, the OTS can be self-adaptive to the changing field of view (FOV) caused by optical occlusion, which is frequently occurred in surgery. Furthermore, experiments were carried out to verify the performance of the proposed model, whose accuracy and runtime performed 0.4627 mm and 0.0016 ms, respectively. Results demonstrate that the proposed reconstruction model can achieve higher accuracy and computational efficiency than both the geometrical model and the data-driven model. Especially, by considering SOC as the state prior knowledge, the tracking robustness is enhanced when one or two of the four cameras are not working properly. Note to Practitioners-The original motivation for this article derives from both the line-of-sight limitation and robust demand for optical tracking of surgical instruments. The performance of the multicamera optical tracking system (OTS) depends on its reconstruction model. However, the geometric reconstruction model requires more calculation to obtain high accuracy, which will enlarge the latency and reduce the update rate. In our previous work, the reconstruction model based on the neural network (NN) has achieved accurate tracking in real-time, while the training of the model tends into local optimal values. Hence, we proposed the prior knowledge feedforward NN model to improve the accuracy and computational efficiency. Moreover, to guarantee the line-of-sight in the optical occlusion, the state of camera combining with the gating technique enables the OTS to be self-adaptive for changing the field of view, which greatly ensures the robust tracking process with larger workspace in case of line-of-sight obstructions.

This paper presents a sensor fusion-based surgical instrument tracking system which uses multiple monocular modules. The system is an optical tracking system, which has been widely utilized in the image-guide surgery because of its high accuracy and precision. However, the line-of-sight occlusion problem which remains unresolved in current systems frustrates surgeons during the operation. To address this challenge, we propose a surgical instrument tracking system based on multiple monocular modules. The rationale is to enable the system to track the surgical instruments inside the surgical site from different views. Three sensor fusion algorithms are proposed to integrate all sensor data from the multimodule system. In order to show the feasibility of the tracking system, simulations and comparison experiments have been carried out. The intensive investigation results give a practical instruction to the real implementation of the proposed system in image-guided interventions. Moreover, an image-guided surgical trial by using a cadaver head has been carried out to validate the feasibility of the proposed system and the tracking algorithms. The results from both the simulation and the cadaver trial have shown the effectiveness of the proposed robust fusion algorithm.

Alignment of two point clouds is an essential problem in medical robotics and computer-assisted surgery. In this paper, we first formally formulate the generalized point cloud registration problem in a probabilistic manner. Specifically, not only positional but also the orientational information are incorporated into registration. Notably, the positional error is assumed to obey a multivariate Gaussian distribution to accommodate anisotropic cases. Expectation conditional maximization framework is utilized to solve the problem. In E-step, the correspondence probabilities between points in two generalized point clouds are computed. In M -step, the constrained optimization problem with respect to the transformation matrix is re-formulated as an unconstrained one. Extensive experiments are conducted to compare the proposed algorithm with the state-of-the-art registration methods. The experimental results demonstrate the algorithm's robustness to noise and outliers, fast convergence speed.

Flexible manipulator enables curvilinear accessibility through small incisions or natural orifices for minimally invasive surgery and diagnosis, which makes it a good choice for minimally invasive surgery. In order to control the robot precisely and safely, the real-time position and shape information of the robot need to be measured well. In this paper, we propose a magnetic tracking based tip pose and shape detection method for wire driven flexible robots. A permanent magnet is mounted at the distal end of the robot. Its magnetic field can be sensed with a sensor array. Therefore, position and orientation of the tip can be estimated utilizing the tracking method. A shape sensing algorithm is then carried out to estimate the real-time shape based on the tip pose. With the tip pose and shape display in the reconstructed visual environment, navigation can be achieved. This method provides the advantages that no sensors are needed to mount on the robot and has no line-of-sight problem. Experimental results verified the feasibility of the proposed method. A navigation error of 1.9mm is achieved.