3D point clouds are data points defining the morphology of environments, and completion refers to the reconstruction of missing points. 6G Internet of Things (6G-IoT) connected with 3D mapping devices needs reliable, consistent, high-fidelity real-time point cloud completion for accurate environment registration. Trustworthy AI, modeled with dependable Deep Learning (DL), enables reliable and robust point completion with spatial-geometrical consistency for deployment with 6G-IoT devices. Although several DL-based completion techniques are integrated with 6G-IoT devices, they have reliability issues, limiting key trustworthy AI characteristics. This research focuses on the reliability and robustness aspects of trustworthy AI to propose Point-KAN, a dependable real-time 3D point cloud completion model for 6G IoT-connected 3D mapping devices. Point-KAN integrates multi-head attention and Kolmogorov-Arnold Networks (KAN) within the modules of Attention Enhanced-Embedded Feature Collector (AEFC) and KAN-Enhanced Feature Mapper (KEFM) for trustworthy point cloud completion. Empirical evaluations on the ShapeNet demonstrate the superiority of Point-KAN against state-of-the-art (SOTA). Results concrete Point-KAN’s evolution as a trustworthy AI framework ensures reliability and robustness for real-time deployment in 6G-IoT-connected devices, facilitating 3D environment mapping.

In intraocular microsurgery with minute operational scales, instruments pass through non-visible regions of the anterior segment, where robot-assisted surgery, which heavily relies on visual perception, fails to determine the instrument’s attitude relative to the eyeball. This compromises surgical flexibility, increases risks, and hinders autonomous surgery development. Therefore, a framework for predicting instrument trajectories in non-visible regions during robot-assisted microsurgery has been proposed to mitigate the risks of retinal and lens injuries caused by blind operations and enhance surgical procedures’ intelligence and autonomy. First, a lightweight reconstruction of the anterior segment environment is performed under controlled knowledge guidance to construct a global map. Second, the tip position of the surgical instrument is detected through multi-sensor fusion, enabling the perception of instrument-environment interactions under visual constraints. Based on this, a long short-term spatiotemporal aggregation algorithm for instrument trajectory prediction is proposed, which enhances surgical safety by providing high-precision predictions of the instrument tip’s motion trajectory. Experiments show that the framework achieved a 0.0435 mm average prediction error in non-visible regions, corresponding to 0.03% of the region in a single dimension and 7.25% of the surgical instrument’s diameter. This significantly enhances the precision of robot-assisted surgery under visual constraints and provides robust technical support for safe, intelligent, and autonomous intraocular robotic surgery.

Creating an intelligent surgical environment requires not only advanced robotic systems but also optimized microscopic imaging. However, autofocus remains a fundamental challenge, with current methods suffering from slow iterative processes or directional ambiguity, which compromises real-time performance. This paper presents an implicit disparity-blur alignment approach for robotic microsurgical autofocus, integrating stereo geometry’s monotonic depth cues with de-focus characteristics for rapid convergence. A novel physics-guided dual-stream network is developed to encode implicit depth representations through hierarchical cross-pathway feature fusion, enabling reliable focus prediction without explicit stereo matching in blur-degraded regions. An ROI-aware attention module is proposed to dynamically optimize focus-critical regions, coupled with learnable physics-guided kernel learning for precise Z-offset estimation. The approach achieves a top directional accuracy of 94.85% and a single-pass focus error of 0.20 mm with an inference time of 53 ms on a surgical dataset, which outperforms state-of-the-art methods in reducing iteration count by 22.8% and inference time by 51.8%. An intelligent robotic microscope prototype is developed, with validation through ex vivo tests demonstrating its ability to enable fast and precise multi-region focusing for microsurgeries.

Robot-assisted surgery has significantly advanced surgical precision, yet the development of autonomous surgical robots remains hindered by their limited understanding of complex surgical actions. Current systems lack the ability to effectively perceive and interpret intricate surgical relationships, which restricts their capability to assist surgeons in dynamic surgical environments. To overcome these challenges, a novel self-supervised learning method for surgical action recognition has been proposed, aimed at enhancing the understanding of surgical actions. The method has introduced a dynamic masking with attention-based action localization module to focus the model on critical spatial regions where actions occur, enabling surgical view guidance for intelligent surgical robot while extracting key features. Moreover, a graph-enhanced adaptive feature selection module is employed to assign relevance to features and capture the temporal relationships between adjacent frames. Long Short-Term Memory has been utilized to model long-term dependencies across video sequences, while multi-view contrastive learning facilitates the extraction of discriminative features from both masked and unmasked sequences. Experimental results demonstrate a 3.4% improvement in Average Precision and an Area Under Receiver Operating Characteristic Curve of 92.9% on Neuro67 dataset for surgical action recognition. The method enables dynamic adjustments to the surgical view, achieving surgical visual navigation. These advancements contribute to the development of intelligent and autonomous surgical robots capable of assisting surgeons in complex and dynamic surgical settings.

This research addresses the challenge of achieving high-precision tracking of time-varying trajectories under nonlinear disturbances and motion constraints in microsurgical robots. A hybrid control framework integrating fuzzy adaptive sliding mode control with radial basis function neural networks is proposed. This framework dynamically adjusts the sliding mode gain to suppress high-frequency jitter and compensate for unmodeled disturbances such as joint friction and tissue contact forces. Experiments conducted on a self-developed microscopic ophthalmic robot platform demonstrated that the trajectory tracking error was reduced to 1.1 μm, representing improvements of 85.9%, 76.1%, and 66.7% compared to PID control, sliding mode control and non-singular fast terminal sliding mode control respectively. The tracking delay was 19 milliseconds. In experiments on living pigs with central retinal artery occlusion, the system successfully performed intravascular injection, with a maximum error of 3.97 μm. This solution, through optimization via fuzzy logic and neural networks, achieves micron-level precision and robustness, effectively solving high-frequency control noise and low-frequency environmental disturbances, ensuring both the accuracy and safety of the microsurgical robot.

Autonomous robotic surgery offers enhanced effectiveness, precision, and reliability, regardless of the surgeons’ expertise. Prior neurosurgery robot studies involved surgeons manually assisting the robot in skull-milling tasks by holding the milling cutter shank, constraining the robot’s autonomy. A model-free adaptive nonlinear force control algorithm is designed to accomplish automatic cranial-milling tasks. Furthermore, a skull-milling breakthrough detection algorithm by monitoring the change of feed force is proposed to determine the completion of the milling task autonomously. A robotic system is developed for automatic cranium-milling and 72 in vitro skull-milling experiments indicate that when using the proposed control algorithm, the maximum root mean square error percentage of the vertical force is 0.99$\%$, while the control error percentages of other mainstream methods are all above 5.5$\%$. Moreover, the success rate of breakthrough detection is 98.61$\%$and the robot autonomously performs the skull milling task with minimal human intervention during the whole experiment. The results demonstrate that the proposed method provides the potential to improve the intelligence of neurosurgery.Note to Practitioners— The purpose is to propose a model-free adaptive nonlinear force control method for automatic skull-milling tasks. In previous studies, the involvement of surgeons manually holding the milling cutter shank to assist neurosurgery robots in skull-milling tasks has been observed. However, the autonomy of the robot is restricted and its potential for precise control tasks is failed to leverage. Therefore, a model-free adaptive nonlinear force control algorithm is proposed and a robotic system is built to enable the robot to autonomously perform cranial-milling tasks in this work. This application aims to enhance the autonomy of robot-assisted neurosurgery, making it a potential solution for remote surgery and addressing the shortage of medical resources in rural areas.

Digital transformation is changing the entire landscape of the financial industry. The increasing customer demand to address, or at least have a positive impact on social problems, fuel the rapid growth of social fintech companies. However, as these companies scale significantly, they face critical challenges in managing employee stress, which can lead to decreased performance and high turnover rates. Following the rise of ChatGPT, large language models (LLMs) have been increasingly utilized in mental health-related applications and offering a promising solution for social fintech companies to support their employee’s mental health. Nevertheless, existing LLMs are predominantly English-focused, limiting their effectiveness in addressing mental health support across diverse linguistic groups. To address this gap, we propose a novel multilingual adaptation of widely used mental health datasets, translated from English into the two most widely spoken languages globally—Mandarin and Spanish. This adaptation enables a comprehensive evaluation of LLMs, such as GPT and Llama, in detecting and assessing mental health conditions across different languages. Initially, we used ChatGPT-4o-Mini to translate the original English dataset into Spanish and Mandarin. We then evaluate the performance of these translated datasets using various state-of-the-art LLMs. Additionally, we analyze the relationship between sentence length and prediction performance. Our experiments reveal significant variability in model performance, with language-specific nuances and disparities in mental health data coverage posing challenges to achieving consistent accuracy.

Redundantly actuated parallel mechanisms (RAPMs) have been widely adopted in advanced robotic systems and precision machine tools. It has been demonstrated that redundant actuation can improve the performance of mechatronic systems but introduce challenges with respect to control. One main difficulty is in establishing an accurate dynamic model of the RAPM system. With an inaccurate dynamic model, the torque applied by the actuators will be incorrect, leading to increased antagonistic forces in the system. To solve this problem, a novel coordinated adaptive impedance control approach based on a new adaptive impedance control law is presented here, along with proof of the stability of the closed-loop system. The control algorithm has been validated experimentally by a prototype cable-driven parallel manipulator. It can be seen from the experimental results that the proposed control method is an effective way to correct the antagonistic forces of the system, thus facilitating the improvement of its dynamic performance and its efficacy in different applications.

The transbronchial interventional surgery presents challenges with winding and convoluted pathways, prone to compression and friction. Current autonomous planning struggles to reach deeper bronchial positions, and hard to consider multiple conflicting goals simultaneously. This article introduces an innovative planning scheme with preference weights to achieve smooth, frictionless, and collision-free autonomous transbronchial intervention with continuum robot (CR). A few-human-interaction twin-delayed deep deterministic policy gradient (FHITD3) generated from surgeon preference guidance is proposed, which determines the optimal strategy for the motion of CR. Preference knowledge is generated through interaction between human and few diversity samples. An abstract actuator space description is proposed for the posture and position representation of CR during movement within bronchus. A contact motion analysis strategy is proposed to calculate real-time attitude of CR in contact with bronchus. In addition, an oscillation suppression approach to address CR's unsmooth distal end trajectory is proposed. Simulated experiments show that the CR autonomously completes intervention tasks with a smooth and stable trajectory, reducing distal end oscillation by over 45%. It achieves a target endpoint within the fourth level bronchus (approximately 5 mm diameter) with over 90% probability.

Internal limiting membrane (ILM) peeling is a vital vitreoretinal surgery procedure. However, due to the thickness of just 1-2 micrometers and the intricacies associated with its varying density and adhesion, the difficulty of manipulation exceeds the physiological limits of human perception and operation. Surgical robot is characterized by high precision and stability. However, navigating intricate intraocular environments and handling minuscule high-precision areas remain enormous challenges. These include issues of uneven lighting, field-of-view loss, and motion blur. This paper proposed a perception method named ‘Multimodal Surgical Process Recognition based on Domain Knowledge and Segmentation (MSPR-DKS),’ designed to address these challenges and provide input for the precise control of robots. Moreover, a comprehensive dataset focused on ILM peeling during macular hole surgeries was established. Experimental results underscore the efficacy of this approach, with segmentation accuracies exceeding 99.27% for instruments and macular holes and an average accuracy of 98.97% in recognizing surgical processes. This study paves the way for leveraging domain knowledge and image segmentation to improve robot-assisted manipulation of soft tissues in ophthalmology.

Prior robot-assisted cranium-milling studies only considered controlling the force in the skull’s vertical direction and neglected the milling cutter’s feed force. Additionally, achieving stable force control in multiple directions is challenging for robots due to the uneven skull surface. Here a hybrid admittance control algorithm incorporating a model-free adaptive nonlinear force control and fuzzy control algorithms is proposed to accomplish effective automatic cranial-milling tasks. First, a pure data-driven model-free adaptive control method based on partial form dynamic linearization is used to control the feed force. Second, fuzzy control minimizes the total error of both the vertical and feed force by adaptively adjusting the milling cutter’s velocity and position. 42 ex vivo animal skull-milling experiments conducted by the automatic robotic cranium-milling system indicate that when using the proposed control algorithm, the force error percentage can be maintained below 5.0% within 3 s and the maximal root mean square error percentages for vertical and feed force are 1.85% and 1.94%, respectively. Moreover, no instances of dura mater damage are observed and the robotic system exhibits a high level of autonomy as it performs the skull milling task with minimal human involvement throughout the entire experiment. The results suggest the potential for advancing the intelligence level of neurosurgery in the future.

Cable-driven continuum robots in bronchoscopic procedures hold immense potential to revolutionize the diagnosis and treatment of lung cancer. However, robotic bronchoscopes in current studies are typically large in size and inflexible. Therefore, this article introduces a novel cable-driven continuum robot bronchoscopy system that achieves modular design between the actuation and operation ends. A continuum structure with a dual-segment notched flexible skeleton, featuring a wall thickness of 0.45 mm, has been designed to perform bending movements exceeding 190°. This enhances flexibility and increases the spatial capacity of the working channels. A kinematic model was developed, integrating the actuation force and the mechanical characteristics of the driving cables for error compensation, estimating the correlation between the displacement of the driving cables and the position of the continuum robot’s end-effector. The verification showed that the root mean square error (RMSE) of the end-effector position is 2.57 mm, which accounts for 4.8% of the continuum’s length. A prototype of the robotic bronchoscopy system was created, and its performance and potential applications in bronchoscopic intervention surgeries were validated through vivo pig intervention experiments.

Emotion is a complex physiological and psychological activity, accompanied by subjective physiological sensations and objective physiological changes. The body sensation map describes the changes in body sensation associated with emotion in a topographic manner, but it relies on subjective evaluations from participants. Physiological signals are a more reliable measure of emotion, but most research focuses on the central nervous system, neglecting the importance of the peripheral nervous system. In this study, a body surface potential mapping (BSPM) system was constructed, and an experiment was designed to induce emotions and obtain high-density body surface potential information under negative and non-negative emotions. Then, by constructing and analyzing the functional connectivity network of BSPs, the high-density electrophysiological characteristics are obtained and visualized as bodily emotion maps. The results showed that the functional connectivity network of BSPs under negative emotions had denser connections, and emotion maps based on local clustering coefficient (LCC) are consistent with BSMs under negative emotions. in addition, our features can classify negative and non-negative emotions with the highest classification accuracy of 80.77%. In conclusion, this study constructs an emotion map based on high-density BSPs, which offers a novel approach to psychophysiological computing.

Minimally invasive surgery, which relies on surgical robots and microscopes, demands precise image segmentation to ensure safe and efficient procedures. Nevertheless, achieving accurate segmentation of surgical instruments remains challenging due to the complexity of the surgical environment. To tackle this issue, this paper introduces a novel multiscale dual-encoding segmentation network, termed MSDE-Net, designed to automatically and precisely segment surgical instruments. The proposed MSDE-Net leverages a dual-branch encoder comprising a convolutional neural network (CNN) branch and a transformer branch to effectively extract both local and global features. Moreover, an attention fusion block (AFB) is introduced to ensure effective information complementarity between the dual-branch encoding paths. Additionally, a multilayer context fusion block (MCF) is proposed to enhance the network's capacity to simultaneously extract global and local features. Finally, to extend the scope of global feature information under larger receptive fields, a multi-receptive field fusion (MRF) block is incorporated. Through comprehensive experimental evaluations on two publicly available datasets for surgical instrument segmentation, the proposed MSDE-Net demonstrates superior performance compared to existing methods.

Pelvic fractures are one of the most serious traumas in orthopedics, and the technical proficiency and expertise of the surgical team strongly influence the quality of reduction results. With the advancement of information technology and robotics, robot-assisted pelvic fracture reduction surgery is expected to reduce the impact caused by inexperienced doctors and improve the accuracy and stability of pelvic reduction. However, this requires the robot to detect key surgeon actions from time-series data, enabling the robot to independently perceive the surgical status, predict the surgeon's intentions, assess the demonstrated level of professional competence, and assess the progress of the surgery. Therefore, a multi-task deep learning neural network architecture is proposed, which incorporates Convolutional Neural Network-Bidirectional Long Short-Term Memory (CNN-BiLSTM) along with tri-modality fusion and feature extraction techniques. The proposed framework aims to achieve key action detection in closed reduction operations for pelvic fractures. Subsequently, a trimodal fine-grained dataset was constructed, wherein 29, 32, and 14 labels were marked on flexion, position, and pressure data for 14 key closed reduction actions. The experimental results show that the correct detection rate of closed reduction actions is 92.3 %, significantly higher than the commonly used recognition algorithms. This work provides a method for the robot to learn the surgeon's professional knowledge, provides the basis for the operation's motion perception, and contributes to the autonomy of the robot-assisted closed reduction surgery of pelvic fractures.

Pelvic fractures are one of the most serious traumas in orthopedic care, and reduction during routine surgery is a significant challenge. Because there are so many vital organs, blood vessels, and nerves around the pelvis, and the reduction force is large, the operational requirements for the surgeon are extremely strict and require extensive experience and surgical skills. This article proposes a method for collecting and digitizing doctors’ reduction movements, which aims to help intelligent devices recognize surgeons’ reduction actions and provides a means to learn from expert experience to improve the accuracy of surgery. First, the convolutional bidirectional long short-term memory algorithm with multilayer cross-fused features is proposed. It extracts time and spatial correlations between multimodal data in a hierarchical manner. Second, discrete dynamic motion primitives are adopted for mapping the surgeon's palm movement trajectory. Finally, this article constructs a data acquisition platform and collects data from surgeons with varying proficiency in closed reduction. Experiment results show that the closed reduction action recognition accuracy is 99% and posture recognition accuracy is 95.5%. The recognition algorithm proposed by this article is significantly higher than the commonly used algorithms in terms of Accuracy, Precision, Recall, and F1-Score. This article provides methods and means for the digitization of surgical expertise and transfers learning for robot-assisted surgery.

In recent intelligent-robot-assisted surgery studies, an urgent issue is how to detect the motion of instruments and soft tissue accurately from intra-operative images. Although optical flow technology from computer vision is a powerful solution to the motion-tracking problem, it has difficulty obtaining the pixel-wise optical flow ground truth of real surgery videos for supervised learning. Thus, unsupervised learning methods are critical. However, current unsupervised methods face the challenge of heavy occlusion in the surgical scene. This paper proposes a novel unsupervised learning framework to estimate the motion from surgical images under occlusion. The framework consists of a Motion Decoupling Network to estimate the tissue and the instrument motion with different constraints. Notably, the network integrates a segmentation subnet that estimates the segmentation map of instruments in an unsupervised manner to obtain the occlusion region and improve the dual motion estimation. Additionally, a hybrid self-supervised strategy with occlusion completion is introduced to recover realistic vision clues. Extensive experiments on two surgical datasets show that the proposed method achieves accurate motion estimation for intra-operative scenes and outperforms other unsupervised methods, with a margin of 15% in accuracy. The average estimation error for tissue is less than 2.2 pixels on average for both surgical datasets.

Percutaneous coronary intervention (PCI) has increasingly become the main treatment for coronary artery disease. The procedure requires high experienced skills and dexterous manipulations. However, there are few techniques to model PCI skill so far. In this study, a learning framework with local and ensemble learning is proposed to learn skill characteristics of different skill-level subjects from their PCI manipulations. Ten interventional cardiologists (four experts and six novices) were recruited to deliver a medical guidewire to two target arteries on a porcine model for in vivo studies. Simultaneously, translation and twist manipulations of thumb, forefinger, and wrist are acquired with electromagnetic (EM) and fiber-optic bend (FOB) sensors, respectively. These behavior data are then processed with wavelet packet decomposition (WPD) under 1-10 levels for feature extraction. The feature vectors are further fed into three candidate individual classifiers in the local learning layer. Furthermore, the local learning results from different manipulation behaviors are fused in the ensemble learning layer with three rule-based ensemble learning algorithms. In subject-dependent skill characteristics learning, the ensemble learning can achieve 100% accuracy, significantly outperforming the best local result (90%). Furthermore, ensemble learning can also maintain 73% accuracy in subject-independent schemes. These promising results demonstrate the great potential of the proposed method to facilitate skill learning in surgical robotics and skill assessment in clinical practice.

The clinical success of the percutaneous coronary intervention (PCI) is highly dependent on endovascular manipulation skills and dexterous manipulation strategies of interventionalists. However, the analysis of endovascular manipulations and related discussion for technical skill assessment are limited. In this study, a multilayer and multimodal-fusion architecture is proposed to recognize six typical endovascular manipulations. The synchronously acquired multimodal motion signals from ten subjects are used as the inputs of the architecture independently. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. The recognition metrics under the determined architecture are further used to assess technical skills. The experimental results indicate that the proposed architecture can achieve the overall accuracy of 96.41%, much higher than that of a single-layer recognition architecture (92.85%). In addition, the multimodal fusion brings significant performance improvement in comparison with single-modal schemes. Furthermore, the K -means-based skill assessment can obtain an accuracy of 95% to cluster the attempts made by different skill-level groups. These hopeful results indicate the great possibility of the architecture to facilitate clinical skill assessment and skill learning.

Surgical image segmentation is critical for surgical robot control and computer-assisted surgery. In the surgical scene, the local features of objects are highly similar, and the illumination interference is strong, which makes surgical image segmentation challenging. To address the above issues, a bilinear squeeze reasoning network is proposed for surgical image segmentation. In it, the space squeeze reasoning module is proposed, which adopts height pooling and width pooling to squeeze global contexts in the vertical and horizontal directions, respectively. The similarity between each horizontal position and each vertical position is calculated to encode long-range semantic dependencies and establish the affinity matrix. The feature maps are also squeezed from both the vertical and horizontal directions to model channel relations. Guided by channel relations, the affinity matrix is expanded to the same size as the input features. It captures long-range semantic dependencies from different directions, helping address the local similarity issue. Besides, a low-rank bilinear fusion module is proposed to enhance the model's ability to recognize similar features. This module is based on the low-rank bilinear model to capture the inter-layer feature relations. It integrates the location details from low-level features and semantic information from high-level features. Various semantics can be represented more accurately, which effectively improves feature representation. The proposed network achieves state-of-the-art performance on cataract image segmentation dataset CataSeg and robotic image segmentation dataset EndoVis 2018.

Intraoperative tracking of laparoscopic instruments is often a prerequisite for computer and robotic-assisted interventions. While numerous methods for detecting, segmenting and tracking of medical instruments based on endoscopic video images have been proposed in the literature, key limitations remain to be addressed: Firstly, robustness, that is, the reliable performance of state-of-the-art methods when run on challenging images (e.g. in the presence of blood, smoke or motion artifacts). Secondly, generalization; algorithms trained for a specific intervention in a specific hospital should generalize to other interventions or institutions. In an effort to promote solutions for these limitations, we organized the Robust Medical Instrument Segmentation (ROBUST-MIS) challenge as an international benchmarking competition with a specific focus on the robustness and generalization capabilities of algorithms. For the first time in the field of endoscopic image processing, our challenge included a task on binary segmentation and also addressed multi-instance detection and segmentation. The challenge was based on a surgical data set comprising 10,040 annotated images acquired from a total of 30 surgical procedures from three different types of surgery. The validation of the competing methods for the three tasks (binary segmentation, multi-instance detection and multi-instance segmentation) was performed in three different stages with an increasing domain gap between the training and the test data. The results confirm the initial hypothesis, namely that algorithm performance degrades with an increasing domain gap. While the average detection and segmentation quality of the best-performing algorithms is high, future research should concentrate on detection and segmentation of small, crossing, moving and transparent instrument(s) (parts).

The automatic defects detection for solar cell electroluminescence (EL) images is a challenging task, due to the similarity of defect features and complex background features. To address this problem, in this article a novel complementary attention network (CAN) is designed by connecting the novel channel-wise attention subnetwork with spatial attention subnetwork sequentially, which adaptively suppresses the background noise features and highlights the defect features simultaneously by employing the complementary advantage of the channel features and spatial position features. In CAN, the novel channel-wise attention subnetwork applies convolution operation to integrate the concatenated and discriminative output features extracted by global average pooling layer and global max pooling layer, which can make fully use of these informative features. Furthermore, a region proposal attention network (RPAN) is proposed by embedding CAN into region proposal network in faster R-CNN (convolution neutral network) to extract more refined defective region proposals, which is used to construct a novel end-to-end faster RPAN-CNN framework for detecting defects in raw EL image. Finally, some experimental results on a large-scale EL dataset including 3629 images, 2129 of which are defective, show that the proposed method performs much better than other methods in terms of defects classification and detection results in raw solar cell EL images.

The real-time localization of the guidewire endpoints is a stepping stone to computer-assisted percutaneous coronary intervention (PCI). However, methods for multi-guidewire endpoint localization in fluoroscopy images are still scarce. In this paper, we introduce a framework for real-time multi-guidewire endpoint localization in fluoroscopy images. The framework consists of two stages, first detecting all guidewire instances in the fluoroscopy image, and then locating the endpoints of each single guidewire instance. In the first stage, a YOLOv3 detector is used for guidewire detection, and a post-processing algorithm is proposed to refine the guidewire detection results. In the second stage, a Segmentation Attention-hourglass (SA-hourglass) network is proposed to predict the endpoint locations of each single guidewire instance. The SA-hourglass network can be generalized to the keypoint localization of other surgical instruments. In our experiments, the SA-hourglass network is applied not only on a guidewire dataset but also on a retinal microsurgery dataset, reaching the mean pixel error (MPE) of 2.20 pixels on the guidewire dataset and the MPE of 5.30 pixels on the retinal microsurgery dataset, both achieving the state-of-the-art localization results. Besides, the inference rate of our framework is at least 20FPS, which meets the real-time requirement of fluoroscopy images (6-12FPS).

Semantic segmentation of surgical instruments plays a critical role in computer-assisted surgery. However, specular reflection and scale variation of instruments are likely to occur in the surgical environment, undesirably altering visual features of instruments, such as color and shape. These issues make semantic segmentation of surgical instruments more challenging. In this paper, a novel network, Pyramid Attention Aggregation Network, is proposed to aggregate multi-scale attentive features for surgical instruments. It contains two critical modules: Double Attention Module and Pyramid Upsampling Module. Specifically, the Double Attention Module includes two attention blocks (i.e., position attention block and channel attention block), which model semantic dependencies between positions and channels by capturing joint semantic information and global contexts, respectively. The attentive features generated by the Double Attention Module can distinguish target regions, contributing to solving the specular reflection issue. Moreover, the Pyramid Upsampling Module extracts local details and global contexts by aggregating multi-scale attentive features. It learns the shape and size features of surgical instruments in different receptive fields and thus addresses the scale variation issue. The proposed network achieves state-of-the-art performance on various datasets. It achieves a new record of 97.10% mean IOU on Cata7. Besides, it comes first in the MICCAI EndoVis Challenge 2017 with 9.90% increase on mean IOU.

The real-time segmentation of surgical instruments plays a crucial role in robot-assisted surgery. However, it is still a challenging task to implement deep learning models to do real-time segmentation for surgical instruments due to their high computational costs and slow inference speed. In this paper, we propose an attention-guided lightweight network (LWANet), which can segment surgical instruments in real-time. LWANet adopts encoder-decoder architecture, where the encoder is the lightweight network MobileNetV2, and the decoder consists of depthwise separable convolution, attention fusion block, and transposed convolution. Depthwise separable convolution is used as the basic unit to construct the decoder, which can reduce the model size and computational costs. Attention fusion block captures global contexts and encodes semantic dependencies between channels to emphasize target regions, contributing to locating the surgical instrument. Transposed convolution is performed to upsample feature maps for acquiring refined edges. LWANet can segment surgical instruments in real-time while takes little computational costs. Based on 960x544 inputs, its inference speed can reach 39 fps with only 3.39 GFLOPs. Also, it has a small model size and the number of parameters is only 2.06 M. The proposed network is evaluated on two datasets. It achieves state-of-the- art performance 94.10% mean IOU on Cata7 and obtains a new record on EndoVis 2017 with a 4.10% increase on mean IOU.

The increasingly-used robotic systems can provide precise delivery and reduce X-ray radiation to medical staff in percutaneous coronary interventions (PCI), but natural manipulations of interventionalists are forgone in most robot-assisted procedures. Therefore, it is necessary to explore natural manipulations to design more advanced human-robot interfaces (HRI). In this study, a multilayer-multimodal fusion architecture is proposed to recognize six typical subpatterns of guidewire manipulations in conventional PCI. The synchronously acquired multimodal behaviors from ten subjects are used as the inputs of the fusion architecture. Six classification-based and two rule-based fusion algorithms are evaluated for performance comparisons. Experimental results indicate that the multimodal fusion brings significant accuracy improvement in comparison with single-modal schemes. Furthermore, the proposed architecture can achieve the overall accuracy of 96.90%, much higher than that of a singlelayer recognition architecture (92.56%). These results have indicated the potential of the proposed method for facilitating the development of HRI for robot-assisted PCI.

Surgical instrument segmentation is crucial for computer-assisted surgery. Different from common object segmentation, it is more challenging due to the large illumination variation and scale variation in the surgical scenes. In this paper, we propose a bilinear attention network with adaptive receptive fields to address these two issues. To deal with the illumination variation, the bilinear attention module models global contexts and semantic dependencies between pixels by capturing second-order statistics. With them, semantic features in challenging areas can be inferred from their neighbors, and the distinction of various semantics can be boosted. To adapt to the scale variation, our adaptive receptive field module aggregates multi-scale features and selects receptive fields adaptively. Specifically, it models the semantic relationships between channels to choose feature maps with appropriate scales, changing the receptive field of subsequent convolutions. The proposed network achieves the best performance 97.47% mean IoU on Cata7. It also takes the first place on EndoVis 2017, exceeding the second place by 10.10% mean IoU.

A challenging aspect of instrument segmentation in robotic surgery is to distinguish different parts of the same instrument. Parts with similar textures are common in a practical instrument and are difficult to distinguish. In this work, the authors introduce an end‐to‐end recurrent model that comprises a multiscale semantic segmentation network and a refinement model. Specifically, the semantic segmentation network uniformly transforms the input images in multiple scales into a semantic mask, and the refinement model is a single‐scale net recurrently optimising the above semantic mask. Through extensive experiments, the authors validate that the models with multiscale inputs perform better than those to fuse encoded feature maps and ones with spatial attention. Furthermore, the authors verify the effectiveness of the proposed model with state‐of‐the‐art performances on several robotic instrument datasets derived from MICCAI Endoscopic Vision Challenges.

Adversary models have been fundamental to the various cryptographic protocols and methods. However, their use in most of the branches of research in computer science is comparatively restricted, primarily in case of the research in cyberphysical security (e.g., vulnerability studies, position confidentiality). In this article, we propose an energy-aware green adversary model for its use in smart industrial environment through achieving confidentiality. Even though, mutually the hardware and the software parts of cyberphysical systems can be improved to decrease its energy consumption, this article focuses on aspects of conserving position and information confidentiality. On the basis of our findings (assumptions, adversary goals, and capabilities) from the literature, we give some testimonials to help practitioners and researchers working in cyberphysical security. The proposed model that runs on real-time anticipatory position-based query scheduling in order to minimize the communication and computation cost for each query, thus, facilitating energy consumption minimization. Moreover, we calculate the transferring/acceptance slots required for each query to avoid deteriorating slots. The experimental results confirm that the proposed approach can diminish energy consumption up to five times in comparison to existing approaches.

Guidewire tracking is a clinical challenge in percutaneous coronary intervention (PCI). The current practice of image-based and sensor-based tracking techniques is still limited by radiation exposure, contrast injection, device sterilization, and procedure safety. In this paper, an interventionalist-behavior-based data fusion framework is developed to provide a novel strategy for tracking guidewire motions in PCI. Four types of natural behavior were acquired from ten interventionalists while performing guidewire translation and rotation based on a simulation platform. Different numbers of behaviors are fused by a hierarchical framework with six local tracking models and three ensemble algorithms. After Gaussian mixture regression-based ensemble fusion, a three-behavior scheme can achieve average tracking errors of 1.07 ± 0.17 mm for guidewire translation, and 20.05 ± 3.36° for guidewire rotation. Relevant statistical analysis further reveals that this scheme outperforms the cases using fewer behaviors, and ensemble fusion brings significant error reduction compared with only local fusion. These meaningful results indicate the great potential of the proposed framework for promoting the improvement of guidewire tracking in PCI.

The choice of model orientation is a very important issue in Additive Manufacturing (AM). In this paper, the model orientation problem is formulated as a multi-objective optimization problem, aiming at minimizing the building time, the surface quality, and the supporting area. Then we convert the problem into a single-objective optimization in the linear-weighted way. After that, the Genetic Algorithm (GA) is used to solve the optimization problem and the process of GA is parallelized and implemented on GPU. Experimental results show that when dealing with complex models in AM, compared with CPU only implementation, the GPU based GA can speed up the process by about 50 times, which helps to significantly reduce the optimization time and ensure the quality of solutions. The GPU based parallel methods we proposed can help to reduce the execution time and improve the efficiency greatly, making the processes more efficient.

Interventional therapy for coronary bifurcation lesion has always been an intractable problem in percutaneous coronary intervention (PCI). Dual-guidewire detection can greatly assist physicians in interventional therapy of bifurcated lesions. Nevertheless, this task often comes with the challenges of X-ray images with low signal noise ratio (SNR) as well as the thinner structure of the guidewire compared to other interventional tools. In this paper, a fully automatic detection method based on an improved U-Net and the modified focal loss is proposed for dual-guidewire segmentation in 2D X-ray fluoroscopy, which accomplishes accurate and robust segmentation. The main contributions of this paper are twofold: (1) the proposed method not only addresses the extreme foreground-background class imbalance generated by the slender guidewire structure, but also solve the problem of misclassified examples caused by the guidewire-like structures and contrast agents; (2) the running speed is about 8 frames per second, which reaches near-real-time processing speed. Furthermore, data augmentation algorithm and transfer learning are used to further improve the performance. The proposed method was verified on clinical 2D X-ray image sequences of 30 patients, in which F1-score reached 0.932. The experiment results indicated that our approach is promising for assisting bifurcation lesion surgery.

Solving a thorny issue of real-time path planning for surgery robot in uncertain environments, a novel algorithm named bidirectional continuous tree search (BCTS) is proposed. Most partially observable markov decision process (POMDP) planners address challenges of unknown environments with discrete states, observations and actions, which are fail to automate the operative procedure. However, the BCTS method addresses the issue by handling POMDPs in continuous state, observation and action spaces. The proposed approach has a bidirectional search structure with the intent of greatly improving the calculation efficiency. Meanwhile, Bayesian optimization (BO) algorithm is considered to dynamically sample promising actions while we construct a belief tree. In view of the speed of BO process, the upper and lower bounds of the optimal action values given by fast informed bound (FIB) and point-based value iteration (PBVI) limit the search scope, so we can improve the speed of BO. In addition, we apply an optimal path planning generator, radial basis function neural network (RBFNN), to obtain a smoother trajectory. Finally, simulation of glaucoma surgery has been carried out to explore the best surgical approach. The results show that the introduced structure can effectively guide the surgery robot to perform surgical procedures and receive a real-time as well as smooth path.

Guidewire tip detection in the percutaneous coronary intervention is important. It assists physicians in navigating and is a prerequisite for clinic applications such as surgical skill assessment and robot assisted surgery. Nevertheless, accurate detection is not a trivial task due to the noisy background of the 2D X-ray image and the thin, deformable structure of the tip. In this paper, an automatic method based on cascaded convolution neural networks is proposed to segment the tip in the 2D X-ray image. The main contribution of the method is to use a cascade detection-segmentation structure to overcome the noisy background and the large deformation of the tip, achieve robust, high-precision segmentation. On the other hand, sufficient annotated training samples are necessary for convolution neural network models, while pixel-level annotating is tedious and time consuming. Accordingly, a novel data augmentation algorithm is introduced to improve the model generalization and performance, reduce the cost of data annotation. Evaluations were conducted on a dataset consisting of 22 different sequences of 2D X-ray images, 15 sequences for training and 7 sequences for evaluation. The proposed approach obtained tip precision of 0.532 pixels, F1score of 0.939, false tracking rate of 0.800%, and missing tracking rate of 9.900% on the test set. And the running speed is 4-5 frames per second.

Detection of surgical devices, in particular of guide-wire detection, is prerequisite during image-guided navigation in percutaneous coronary intervention (PCI). Guide-wire detection is a challenging task for following reasons: (i) X-ray images have a low signal-to-noise rate (SNR); (ii) there is a high similarity between guide-wires and some other adjacent anatomical skeletons' contours; (iii) guide-wires have various shapes and their motion is complex and nonlinear. Traditionally, guide-wires are detected using curve fitting method, and third-order B-spline curve model is always used to fit guide-wires, while B-spline fitting method has some obvious shortcomings such as it is a semi-automatic method which needs manual initialization, and it is not a real-time method because of high computational complexity. Recently, with the availability of large annotated datasets and the accessibility of hardware resources with GPUs, it is succeeded in detecting general objects with convolutional neural networks (ConvNet). In this paper, we present a novel image-based fully-automatic and real-time approach with ConvNet for guide-wires detection. ConvNet method is robust to guide-wires' various poses and other structures' effects. We evaluate our method on 22 different sequences of X-ray images. The detection accuracy evaluated by average precision (AP) reaches 89.2% and the detection speed achieves 40fps. Our experiment result shows a promising for accurate and real-time guide-wires detection in PCI navigation with ConvNet model.

For the treatment of cardiovascular diseases, clinical success of percutaneous coronary intervention is highly dependent on natural technical skills and dexterous manipulation strategies of surgeons. However, the increasing used robotic surgical systems have been designed without considering manipulation techniques, especially surgical behaviors and motion patterns. This has driven research towards exploitation of natural manipulation skills in recent years. In this paper, natural guidewire manipulations are analyzed and predicted using an sEMG-based nonlinear autoregressive neural network with exogenous inputs. The relationship between natural endovascular manipulation and guidewire rotation is built through the network. Two experiments at different rotational speed were performed to verify the effectiveness and robustness of the applied model. The experimental results show that the average predictive root mean error of five subjects is 15.61° at the low speed and 21.85° at the high speed. These favorable results could be of interest to improve existing robotic surgical systems.

The percutaneous coronary interventions (PCI) require complex operating skills of the interventional devices and make the surgeons being exposed to heavy X-ray radiation. Accurate delivery of the interventional devices and avoiding the radiation are especially important for the surgeons. This paper presents a novel dedicated dual-finger robotic hand (DRH) and a console to assist the surgeons to deliver the interventional devices in PCIs. The system is designed in the master-slave way which helps the surgeons to reduce the exposure to radiation. The mechanism of the DRH is bio-inspired and motions are decoupled in kinematics. In PCI procedures, the accuracy of the guidewire delivery and the catheter tip placement have significant effects on the surgical results. The performances of the DRH in delivering the guidewire and the balloon/stent catheter were evaluated by three surgical manipulations. The results show that the DRH has the ability to deliver the guidewire and the balloon/stent catheter precisely.