Person re-identification (ReID) aims to retrieve target pedestrian images given either visual queries (image-to-image, I2I) or textual descriptions (text-to-image, T2I). Although both tasks share a common retrieval objective, they pose distinct challenges: I2I emphasizes discriminative identity learning, while T2I requires accurate cross-modal semantic alignment. Existing methods often treat these tasks separately, which may lead to representation entanglement and suboptimal performance. To address this, we propose a unified framework named Hierarchical Prompt Learning (HPL), which leverages task-aware prompt modeling to jointly optimize both tasks. Specifically, we first introduce a Task-Routed Transformer, which incorporates dual classification tokens into a shared visual encoder to route features for I2I and T2I branches respectively. On top of this, we develop a hierarchical prompt generation scheme that integrates identity-level learnable tokens with instance-level pseudo-text tokens. These pseudo-tokens are derived from image or text features via modality-specific inversion networks, injecting fine-grained, instance-specific semantics into the prompts. Furthermore, we propose a Cross-Modal Prompt Regularization strategy to enforce semantic alignment in the prompt token space, ensuring that pseudo-prompts preserve source-modality characteristics while enhancing cross-modal transferability. Extensive experiments on multiple ReID benchmarks validate the effectiveness of our method, achieving state-of-the-art performance on both I2I and T2I tasks.

Introducing virtual reality technology into surgical training markedly enhances the development of surgical skills. It is critical to accurately assess surgeon proficiency post-training, as traditional evaluation methods often fail to distinguish effectively between novices and experts. Analysis of EEG signals during virtual surgical procedures, augmented by machine learning techniques, provides a robust method for improving skill assessment. The study finds that utilizing both traditional metrics and EEG rhythm indicators significantly increases classification accuracy. Specifically, when integrated with support vector machines and convolutional recurrent neural networks, these metrics enhance classification accuracy by 15.84% and 19.07%, respectively. This methodology offers a thorough tool for assessing surgical proficiency, underscoring the efficacy of EEG rhythms and machine learning in advancing surgical training evaluations.

Artificial intelligence-generated content (AIGC) aims to represent new content synthesized by learning rules from the existing data and has expanded its applications to the health domain, particularly generating medical robotic surgical scenes. Most pioneering methods rely primarily on 2D representations and thus will inevitably suffer from scene distortion when large surgical tool movements and intense soft tissue deformation are encountered. Recent works instead employ explicit 3D structural representations or implicit neural rendering to improve performance under large pose changes. Nevertheless, the fidelity of structure and texture is not so desirable, especially for novel-view synthesis. In this paper, we propose H-RSSG(high-fidelity robotic surgical scene generation with an implicit deformable neural radiance field), which achieves high-fidelity and free-view robotic-surgical-scenes synthesis. The H-RSSG explores the potential of artificial intelligence in generating high-fidelity medical robotic surgical scenes. In addition to generating robotic surgical scenes, the H-RSSG provides standardized evaluation metrics by establishing a benchmark dataset. Moreover, H-RSSG enhances the robustness of AIGC-generated scenes by combining adjacent view feature fusion and multi-view correspondence loss to improve spatial context consistency. A notable advantage of the H-RSSG is its integration of an extended transform-based depth perception module with a lightweight neural network used to estimate masking depths even in the presence of surgical instruments. This results in spatiotemporally consistent high-fidelity robotic surgical scenes. The results of the experiments conducted on the self-established and EndoNeRF datasets have a PRMSE of about 0.0341, an average improvement of 12.1% in PSNR, and an average improvement of 6.69% in SSIM, which demonstrates that the proposed H-RSSG can outperform the benchmark models’ performance, achieving state-of-the-art results. This study highlights the AIGC’s potential to create highly realistic surgical training environments for medical professionals and applications in the medical industry.Video is available at:H-RSSG.

Generalized Category Discovery (GCD) aims to classify unlabeled data from both known and unknown categories by leveraging knowledge from labeled known categories. While existing methods have made notable progress, they often overlook a hidden stumbling block in GCD: distracted attention. Specifically, when processing unlabeled data, models tend to focus not only on key objects in the image but also on task-irrelevant background regions, leading to suboptimal feature extraction. To remove this stumbling block, we propose Attention Focusing (AF), an adaptive mechanism designed to sharpen the model's focus by pruning non-informative tokens. AF consists of two simple yet effective components: Token Importance Measurement (TIME) and Token Adaptive Pruning (TAP), working in a cascade. TIME quantifies token importance across multiple scales, while TAP prunes non-informative tokens by utilizing the multi-scale importance scores provided by TIME. AF is a lightweight, plug-and-play module that integrates seamlessly into existing GCD methods with minimal computational overhead. When incorporated into one prominent GCD method, SimGCD, AF achieves up to 15.4% performance improvement over the baseline with minimal computational overhead. The implementation code is provided in https://github.com/Afleve/AFGCD.

Since the advent of virtual reality (VR), it has been implemented in medical education for surgical training and anatomy education so that the Internet of Medical Things (IoMT) could be further boosted. There have been rare studies on the research trends of the evaluation of endoscopic training through different XR modalities. Position-based dynamics (PBD) has been utilized to optimize the surgical thread simulation, This paper aims to quantitatively evaluate the training performance of each XR modality in general and in terms of the medical fields studied and outcomes measured. Sensors and devices are utilized to form the Internet of Medical Things for healthcare, where the data is uploaded to the cloud and then analyzed as follows before being fed back to the doctor so that he or she can understand his or her level of operation. Through subjective and objective evaluation, the potential promoting effects of vision and touch in module training were discussed.

The virtual surgery engine represents a crucial research domain within biomedical and information sciences. To address the real-time and realistic demands of virtual surgical robots for soft tissue deformation and cutting, the surface information and internal structure of organ models have been redefined. An enhanced Three-Parameter Mass-Ogden model, which incorporates nonlinearity and viscoelasticity in soft tissues, has been developed based on extended position dynamics. Cluster constraints for filling particles were introduced to improve the smoothness of surgical procedures. The relaxation and creep characteristics of real soft tissues were accounted for by evaluating the responses of various biological tissues to external stress and loads using the HY-0580 high-performance mechanical testing machine. Eight experiments were conducted for each tissue type, and five sets of valid data were averaged and fitted using the Three-Parameter Mass-Ogden mixed model. Surgical simulations were conducted using Abaqus, incorporating Young's modulus, stress-strain relationships, cutting depth, pressure distribution, real-time feedback, and comprehensive visualization. The model's effectiveness was further validated. The surgical platform was integrated into a virtual reality-based digital twin robot simulator for minimally invasive surgery, achieving a surgical operation refresh rate of 78.5 Hz, a visual refresh rate of 60 Hz, and a haptic feedback refresh rate of 1000 Hz. Comparative analysis with the Mass-Spring Model (MSM) and Finite Element Method (FEM) shows our model's superior balance of accuracy and efficiency. MSM is fast but imprecise, while FEM is accurate but computationally intensive.

Virtual reality technology has been widely used in surgical simulators, providing new opportunities for assessing and training surgical skills. Machine learning algorithms are commonly used to analyze and evaluate the performance of participants. However, their interpretability limits the personalization of the training for individual participants. Seventy-nine participants were recruited and divided into three groups based on their skill level in intracranial tumor resection. Data on the use of surgical tools were collected using a surgical simulator. Feature selection was performed using the Minimum Redundancy Maximum Relevance and SVM-RFE algorithms to obtain the final metrics for training the machine learning model. Five machine learning algorithms were trained to predict the skill level, and the support vector machine performed the best, with an accuracy of 92.41% and Area Under Curve value of0.98253. The machine learning model was interpreted using Shapley values to identify the important factors contributing to the skill level of each participant. This study demonstrates the effectiveness of machine learning in differentiating the evaluation and training of virtual reality neurosurgical per- formances. The use of Shapley values enables targeted training by identifying deficiencies in individual skills. This study provides insights into the use of machine learning for personalized training in virtual reality neurosurgery. The interpretability of the machine learning models enables the development of individualized training programs. In addition, this study highlighted the potential of explanatory models in training external skills.

The advancement of virtual reality technology has ushered in new developments in the medical field. The use of virtual surgery training simulators alleviates the paucity of training resources and high training expenses associated with traditional surgical capabilities. Regardless of the type of schooling, doctors must continue to educate themselves. The postoperative evaluation mechanism is incomplete. Traditional objective evaluation indicators are unable to meet surgeons' stringent expectations. The electroencephalograph (EEG) rhythm index is proposed in this article as a new tool for evaluating and distinguishing between novice and expert doctors. The experiment uses a cutting training module from neurosurgery training and compares it with established assessment metrics to determine the correct rate of classification of new evaluation metrics, classifying testers by both metrics and finding a 20% increase in correctness. Additionally, this article compares the energy topographic maps of different EEG rhythms of novices and experts. For classification, two-machine learning algorithms, SVM and random forest, are utilized at the same time. The findings reveal that the accuracy of distinguishing indicators based on EEG cycles is 10% higher than that of typical objective evaluation indicators, regardless of the categorization method. ROC curve analysis was also used to compare the two classification models. The AUC value for the EEG rhythm evaluation index model was 0.971, whereas the AUC value for the classic objective evaluation index model was 0.761, which explains the EEG rhythm evaluation index. The model demonstrates a categorization standard that is reliable.

Summary Realistic modeling of mechanical behavior of soft tissue has been recognized as an essential part for medical Internet of thing for minimal invasive surgery (MIS) training simulator. Therefore, the blockchain‐based constitutive model is crucial for mechanical response of soft tissue modeling. In this article, based on the Ogden second order model, a novel hyperplastic model was presented to describe the stress‐stretch relationship in the MIS training system. To validate this theoretical model, two experimental techniques (uniaxial compression and uniaxial tensile) were conducted to obtain data related to stress‐strain in the blockchain system, which plays an important role in investigating the mechanical behavior of soft tissue. Our results show that the new model has a satisfied coincidence of the experimental data than other existing models. Furthermore, the viscoelastic properties of soft tissue were investigated and a viscoelastic model based on three‐parameter was utilized to interpret the viscoelastic behavior of the soft tissue. The contributions of this article include several biomechanical tests that were performed to investigate the soft tissue hyperelastic and viscoelastic properties in the MIS system, and theoretical guidance for simulating soft tissue mechanical behavior in the blockchain‐based simulation system.

As a fundamental service in near future, medical digital twin (MDT) is the virtual replica of a person. MDT applies new technologies of IoT, AI and big data to predict the state of health and offer clinical suggestions. It is crucial to secure medical digital twins through deep understanding of the design of digital twins and applying the new vulnerability tolerant approach. In this paper, we present a new medical digital twin, which systematically combines Haptic-AR navigation and deep learning techniques to achieve virtual replica and cyber–human interaction. We report an innovative study of the cyber–human interaction performance in different scenarios. With the focus on cyber resilience, a new solution of vulnerability tolerant is the must in the real-world MDT scenarios. We propose a novel scheme for recognising and fixing MDT vulnerabilities, in which a new CodeBERT-based neural network is applied to better understand risky code and capture cybersecurity semantics. We develop a prototype of the new MDT and collect several real-world datasets. In the empirical study, a number of well-designed experiments are conducted to evaluate the performance of digital twin, cyber–human interaction and vulnerability detection. The results confirm that our new platform works well, can support clinical decision and has great potential in cyber resilience.

Minimally invasive surgery (MIS) is gradually replacing traditional open surgery. Novices need a lot of practice in the surgical simulator to master surgical skills. Therefore, this paper developed an extended reality-based minimally invasive surgery (MIS-XR) simulator, which includes VR, AR and IVR simulators, and introduces the traditional Box simulator to compare these four simulators. Twenty-two subjects were divided into the expert group (6) and novice group (16) and were invited to participate in the experiment. Face, content, and construct validation methods were used to evaluate the tactile sense, visual sense, scene authenticity, and performance of the four simulators. The cumulative summation was used to further analyze the learning curve of 30 times training for the novice group, to verify the effectiveness of the four simulators and determine which simulator can improve the operator’s surgical skills more quickly. The results of the face and content validation show that the Box simulator is the strongest among the four simulators in the tactile sense, IVR simulator is superior to the Box simulator and VR simulator in scene authenticity. The result of construction validation shows that the four simulators are not only useful in improving the surgical skills of novices but also retain the surgical skills after a period of rest. The MIS-XR simulator developed in this paper is effective and can be used as a training device of surgical skills for novices.

A digital-twin (DT)-enabled Internet of Medical Things (IoMT) system for telemedical simulation is developed, systematically integrated with mixed reality (MR), 5G cloud computing, and a generative adversarial network (GAN) to achieve remote lung cancer implementation. Patient-specific data from 90 lung cancer with pulmonary embolism (PE)-positive patients, with 1372 lung cancer control groups, were gathered from Qujing and Dehong, and then transmitted and preprocessed using 5G. A novel robust auxiliary classifier GAN (rAC-GAN)-based intelligent network is employed to facilitate lung cancer with the PE prediction model. To improve the accuracy and immersion during remote surgical implementation, a real-time operating room perspective from the perception layer with a surgical navigation image is projected to the surgeon’s helmet in the application layer using the DT-based MR guide clue with 5G. The accuracies of the area under the curve (AUC) of our new intelligent IoMT system were 0.92 and 0.93. Furthermore, the pathogenic features learned from our rAC-GAN model are highly consistent with the statistical epidemiological results. The proposed intelligent IoMT system generates significant performance improvement to process substantial clinical data at cloud centers and shows a novel framework for remote medical data transfer and deep learning analytics for DT-based surgical implementation.

Due to multitudinous vulnerabilities in sophisticated software programs, the detection performance of existing approaches requires further improvement. Multiple vulnerability detection approaches have been proposed to aid code inspection. Among them, there is a line of approaches that apply deep learning (DL) techniques and achieve promising results. This paper attempts to utilize CodeBERT which is a deep contextualized model as an embedding solution to facilitate the detection of vulnerabilities in C open-source projects. The application of CodeBERT for code analysis allows the rich and latent patterns within software code to be revealed, having the potential to facilitate various downstream tasks such as the detection of software vulnerability. CodeBERT inherits the architecture of BERT, providing a stacked encoder of transformer in a bidirectional structure. This facilitates the learning of vulnerable code patterns which requires long-range dependency analysis. Additionally, the multihead attention mechanism of transformer enables multiple key variables of a data flow to be focused, which is crucial for analyzing and tracing potentially vulnerable data flaws, eventually, resulting in optimized detection performance. To evaluate the effectiveness of the proposed CodeBERT-based embedding solution, four mainstream-embedding methods are compared for generating software code embeddings, including Word2Vec, GloVe, and FastText. Experimental results show that CodeBERT-based embedding outperforms other embedding models on the downstream vulnerability detection tasks. To further boost performance, we proposed to include synthetic vulnerable functions and perform synthetic and real-world data fine tuning to facilitate the model learning of C-related vulnerable code patterns. Meanwhile, we explored the suitable configuration of CodeBERT. The evaluation results show that the model with new parameters outperform some state-of-the-art detection methods in our dataset.

Synthetic aperture radar (SAR) target recognition is essential for SAR image interpretation. It has been widely used in national defense and national economy. At present, the SAR image detection and recognition methods based on convolutional neural network (CNN) have problems such as insufficient extraction of the feature information of SAR image targets, false targets caused by the interference of complex backgrounds, and low detection performance. The main reason is that the feature extraction of CNN is a local operation in space and time, which ignores the correlation between pixels and regions and the dependencies between channels in SAR images. In this paper, a non-local channel attention network (NLCANet) SAR image target recognition method is proposed based on the GoogLeNet structure combined with asymmetric pyramid non-local block (APNB) and squeeze-and-excitation block (SEB). APNB is added to the GoogLeNet framework to capture more context information and enhance the correlation between pixels and regions. SEB is added to the Inception structure to become Inception-SEB (ISEB), through which channel dependencies based on the fusion of different scale features can be obtained. The experimental results based on the moving and stationary target acquisition and recognition (MSTAR) dataset and the SAR ship detection dataset (SSDD) show that the proposed method improves the detection ability of targets in complex backgrounds and achieves better land-sea target recognition performance.

This article presents a novel extended reality (XR) and deep-learning-based Internet-of-Medical-Things (IoMT) solution for the COVID-19 telemedicine diagnostic, which systematically combines virtual reality/augmented reality (AR) remote surgical plan/rehearse hardware, customized 5G cloud computing and deep learning algorithms to provide real-time COVID-19 treatment scheme clues. Compared to existing perception therapy techniques, our new technique can significantly improve performance and security. The system collected 25 clinic data from the 347 positive and 2270 negative COVID-19 patients in the Red Zone by 5G transmission. After that, a novel auxiliary classifier generative adversarial network-based intelligent prediction algorithm is conducted to train the new COVID-19 prediction model. Furthermore, The Copycat network is employed for the model stealing and attack for the IoMT to improve the security performance. To simplify the user interface and achieve an excellent user experience, we combined the Red Zone’s guiding images with the Green Zone’s view through the AR navigate clue by using 5G. The XR surgical plan/rehearse framework is designed, including all COVID-19 surgical requisite details that were developed with a real-time response guaranteed. The accuracy, recall, F1-score, and area under the ROC curve (AUC) area of our new IoMT were 0.92, 0.98, 0.95, and 0.98, respectively, which outperforms the existing perception techniques with significantly higher accuracy performance. The model stealing also has excellent performance, with the AUC area of 0.90 in Copycat slightly lower than the original model. This study suggests a new framework in the COVID-19 diagnostic integration and opens the new research about the integration of XR and deep learning for IoMT implementation.

With the development of the Energy Internet of Things (EIoT), it is of great practical significance to study the security strategy and intelligent control system for solar thermal utilization system to optimize the operation efficiency and carry out intelligent dynamic adjustment. For buildings integrated with solar water heating systems, computational fluid dynamics simulation was used in analyzing the process of solar energy output. A method based on machine learning is proposed to predict energy conversion. Besides, the simulation and analysis are carried out in combination with the possible safety problems such as the vibration of the control system. This paper proposed a novel platform of EIoT for machine learning-based cybersecurity study and implemented the platform for the temperature monitoring system. After the evaluation of the machine learning-based cybersecurity study, the EIoT system demonstrated a high performance with the Extreme Gradient Boosting (XGBoost) training algorithm.

A novel intelligent navigation technique for accurate image-guided COVID-19 lung biopsy is addressed, which systematically combines augmented reality (AR), customized haptic-enabled surgical tools, and deep neural network to achieve customized surgical navigation. Clinic data from 341 COVID-19 positive patients, with 1598 negative control group, have collected for the model synergy and evaluation. Biomechanics force data from the experiment are applied a WPD-CNN-LSTM (WCL) to learn a new patient-specific COVID-19 surgical model, and the ResNet was employed for the intraoperative force classification. To boost the user immersion and promote the user experience, intro-operational guiding images have combined with the haptic-AR navigational view. Furthermore, a 3-D user interface (3DUI), including all requisite surgical details, was developed with a real-time response guaranteed. Twenty-four thoracic surgeons were invited to the objective and subjective experiments for performance evaluation. The root-mean-square error results of our proposed WCL model is 0.0128, and the classification accuracy is 97%, which demonstrated that the innovative AR with deep learning (DL) intelligent model outperforms the existing perception navigation techniques with significantly higher performance. This article shows a novel framework in the interventional surgical integration for COVID-19 and opens the new research about the integration of AR, haptic rendering, and deep learning for surgical navigation.

Chronic kidney disease has become one of the diseases with the highest morbidity and mortality in kidney diseases, and there are still some problems in surgery. During the operation, the surgeon can only operate on two-dimensional ultrasound images and cannot determine the spatial position relationship between the lesion and the medical puncture needle in real-time. The average number of punctures per patient will reach 3 to 4, Increasing the incidence of complications after a puncture. This article starts with ultrasound-guided renal biopsy navigation training, optimizes puncture path planning, and puncture training assistance. The augmented reality technology, combined with renal puncture surgery training was studied. This paper develops a prototype ultrasound-guided renal biopsy surgery training system, which improves the accuracy and reliability of the system training. The system is compared with the VR training system. The results show that the augmented reality training platform is more suitable as a surgical training platform. Because it takes a short time and has a good training effect.

Compared with traditional thoracotomy, video-assisted thoracoscopic surgery (VATS) has less minor trauma, faster recovery, higher patient compliance, but higher requirements for surgeons. Virtual surgery training simulation systems are important and have been widely used in Europe and America. Augmented reality (AR) in surgical training simulation systems significantly improve the training effect of virtual surgical training, although AR technology is still in its initial stage. Mixed reality has gained increased attention in technology-driven modern medicine but has yet to be used in everyday practice. This study proposed an immersive AR lobectomy within a thoracoscope surgery training system, using visual and haptic modeling to study the potential benefits of this critical technology. The content included immersive AR visual rendering, based on the cluster-based extended position-based dynamics algorithm of soft tissue physical modeling. Furthermore, we designed an AR haptic rendering systems, whose model architecture consisted of multi-touch interaction points, including kinesthetic and pressure-sensitive points. Finally, based on the above theoretical research, we developed an AR interactive VATS surgical training platform. Twenty-four volunteers were recruited from the First People's Hospital of Yunnan Province to evaluate the VATS training system. Face, content, and construct validation methods were used to assess the tactile sense, visual sense, scene authenticity, and simulator performance. The results of our construction validation demonstrate that the simulator is useful in improving novice and surgical skills that can be retained after a certain period of time. The video-assisted thoracoscopic system based on AR developed in this study is effective and can be used as a training device to assist in the development of thoracoscopic skills for novices.

To fuse infrared and visible images in wireless applications, the extraction and transmission of characteristic information security is an important task. The fused image quality depends on the effectiveness of feature extraction and the transmission of image pair characteristics. However, most fusion approaches based on deep learning do not make effective use of the features for image fusion, which results in missing semantic content in the fused image. In this paper, a novel trustworthy image fusion method is proposed to address these issues, which applies convolutional neural networks for feature extraction and blockchain technology to protect sensitive information. The new method can effectively reduce the loss of feature information by making the output of the feature extraction network in each convolutional layer to be fed to the next layer along with the production of the previous layer, and in order to ensure the similarity between the fused image and the original image, the original input image feature map is used as the input of the reconstruction network in the image reconstruction network. Compared to other methods, the experimental results show that our proposed method can achieve better quality and satisfy human perception.

Internet of Things (IoT) integrates a variety of software (e.g., autonomous vehicles and military systems) in order to enable the advanced and intelligent services. These software increase the potential of cyber-attacks because an adversary can launch an attack using system vulnerabilities. Existing software vulnerability analysis methods used to be relying on human experts crafted features, which usually miss many vulnerabilities. It is important to develop an automatic vulnerability analysis system to improve the countermeasures. However, source code is not always available (e.g., most IoT related industry software are closed source). Therefore, vulnerability detection on binary code is a demanding task. This article addresses the automatic binary-level software vulnerability detection problem by proposing a deep learning-based approach. The proposed approach consists of two phases: binary function extraction, and model building. First, we extract binary functions from the cleaned binary instructions obtained by using IDA Pro. Then, we employ the attention mechanism on top of a bidirectional long short-term memory for building the predictive model. To show the effectiveness of the proposed approach, we have collected datasets from several different sources. We have compared our proposed approach with a series of baselines including source code-based techniques and binary code-based techniques. We have also applied the proposed approach to real-world IoT related software such as VLC media player and LibTIFF project that used on Autonomous Vehicles. Experimental results show that our proposed approach betters the baselines and is able to detect more vulnerabilities.

To date, the Medical Internet of Things (MIoT) technology has been recognized and widely applied due to its convenience and practicality. The MIoT enables the application of machine learning to predict diseases of various kinds automatically and accurately, assisting and facilitating effective and efficient medical treatment. However, the MIoT are vulnerable to cyberattacks which have been constantly advancing. In this paper, we establish a MIoT platform and demonstrate a scenario where a trained Convolutional Neural Network (CNN) model for predicting lung cancer complicated with pulmonary embolism can be attacked. First, we use CNN to build a model to predict lung cancer complicated with pulmonary embolism and obtain high detection accuracy. Then, we build a copycat model using only a small amount of data labeled by the target network, aiming to steal the established prediction model. Experimental results prove that the stolen model can also achieve a relatively high prediction outcome, revealing that the copycat network could successfully copy the prediction performance from the target network to a large extent. This also shows that such a prediction model deployed on MIoT devices can be stolen by attackers, and effective prevention strategies are open questions for researchers.

Summary A novel security awareness surgical navigation system has been proposed for the accurate minimally invasive surgery with machine learning algorithms, haptic‐enabled devices, and customized surgical tools to guide the surgery with real‐time force and visual navigation. To provide a direct and simplified user interface during the operation, we combined traditional surgical guide images with AR‐based view and implemented a 3D reconstructed patient‐specific surgical environment includes with all surgical requisite details. In particular, we trained the surgical collected biomechanics haptic data by employed LSTM‐based RNN algorithm, and residual network for the intraoperative force manipulation prediction and classification, respectively. Experiments evaluation results on percutaneous therapy surgery demonstrated a higher performance and distinguished accuracy by the visual and haptic combined than the traditional navigation system. These preliminary study findings may suggested a new framework in the minimally invasive surgical navigation application and hint at the possibility integration of haptic, AR, and machine learning algorithms implementation in medical simulation. In addition, we take security into account when implementation this new framework.

Video-assisted thoracoscopic surgery (VATS), referred to as the commonest minimum invasive excision for located T1 or T2 lung carcinomas, requires a steep learning curve for the novice residents to acquire highly deliberate skills to achieve surgical competence. The aim of this study is to propose a virtual reality-based (VR) surgical educative simulator with realistic performance in both visual and haptic sensation for the VAST procedures. To provide an immersive and perceptual user interface, we combined the customized haptic-enabled thoracoscopic instruments with HTC VIVE helmet set in our simulation system. In particular, position based deformation (PBD) method on the GPU and a novel haptic rendering algorithm of surgical grasps and stapling operations are also been implemented for the surgical scene, respectively for the soft tissue deformation and intraoperative force manipulation simulation. Experiments by thoracic surgery professors and novices' evaluation results on our framework demonstrated a high performance and distinguished accurately. These study findings suggested a new cognitive model for the VATS surgical education integrate with haptic and VR implementation.

The purpose of this study is to investigate the possibility to use eye movements to control the laparoscope during a laparoscopic surgery. Laparoscopic surgery usually needs at least two doctors, a surgeon and a laparoscope assistant. The view of the operating surgeon is provided by the laparoscope assistant. As misunderstandings or conflicts of cooperation may happen, an ideal way is that the surgeon has a full control of all the instruments including the surgical tools and laparoscope. To achieve it, an eye based interaction method is introduced in this paper that allows surgeons to control the view by themselves. With recent developments in the eye tracker platforms and associated eye tracking technologies, many non-contact eye tracking systems are available. It can record where a person is looking at any time and a sequence of eye movements. This information can be used to know where is the attention and interest of the person on a display. As such, surgeon's attention can be captured and then be followed by moving the laparoscope to the region of interest. To have a safe and efficient evaluation on the usability, a virtual reality based laparoscopic surgery simulation is built. It is based on Unity with two haptic devices simulating the surgical tools, a 3D mouse providing 6 degrees-of-freedom control of the camera and an eye tracker capturing eyes' positions on a display. Experiments on moving a camera left, right, up, down, in, out and to specified locations using eyes are conducted, and moreover the performances of the proposed eye based self-control and the 3D mouse based other-control are compared. The results are promising where the proposed pointing method leads to 43.6% faster completion of the tasks against the traditional other-control method using the 3D mouse.

Percutaneous therapy is a common clinical operation in minimally invasive surgery. Yet, learning curve of this skillful manual operation is steep, which imposes negative impacts on its further advances. In this paper, we proposed a novel workflow to simulate percutaneous therapy through visuo-haptic rendering based on the clinical trials. Intraoperative puncture data, obtained by our 6DOF force recording system in the operating room, is fitted as the original force model for the haptic rendering. Patient-specific medical images were also segmented and reconstructed for the highly immersive virtual training scenario. Last but not least, medical professors and novices have also been invited to practice on our training scenario by employed the Global Rating Scale (GRS) questionnaire and parameter metrics recording to validate framework's performance. Posttest values in experts and novices' groups after training showed great progress with respect to pretest values in both GRS scores and objective evaluation.

Percutaneous surgery is a typical minimally invasive surgery. Featuring minimization in trauma and infection rate as well as rapid recovery time to patients, percutaneous therapy has replaced various traditional open surgery approaches and has become an essential approach for a series of clinic operations over the past decades. However, the practice and training for such a vocational manual skill is both difficult and expensive, which imposes negative impacts on its further advances. In this paper, we conducted an immersive needle insertion simulator for percutaneous surgery through visuo-haptic rendering. Multi-layered deformable tissue model with human anatomic textures are simulated and rendered. Mass-spring based force model and algorithm are also employed for realistic trocar needle insertion. Last but not least, a highly immersive virtual training scenario, integrated with a desktop haptic device is implemented to facilitate perceptive and hands-on experiences. Medical professional and trainees have also been invited to practice on the training scenario and provide subjective opinions in refining our implementation.