A 3D point cloud completion task is to generate completed 3D objects given partial observations. Auto-encoder-based models suffer from poor generalization ability to untrained 3D data. Current diffusion-based models add isotropic noise with the same variance in three x, y, z axes. More importantly, these models ignore real-world anisotropic evolution properties of 3D particles from a non-equilibrium state to thermodynamic equilibrium in the real physical world due to the velocity and energy thermodynamics of the particles, leading to unstable completions of 3D object topology. This paper presents a novel physically-based anisotropic 3D diffusion model (3DDM) to address these issues. We also present derivations of our proposed forward and reverse processes and a loss function in closed form, thus reproducibility. The 3DDM contains anisotropic energy-aware forward and reverse processes with a novel anisotropic quadratic loss function. The forward process adds anisotropic 3D Gaussian noises per-axis and mimics the thermal non-equilibrium evolution towards Maxwellian equilibrium based on velocity and kinetic energy evolutions of 3D particles in the real physical space. The reverse process learns to denoise along per-axis and per-timestep anisotropically. The anisotropic quadratic loss function penalizes errors along certain axes, yielding a highly flexible and anisotropic reverse diffusion process and a physically realistic generative model. The 3DDM denoises along x, y, z axes with different velocities from the non-equilibrium evolution, achieving fewer than 20 diffusion steps and strong generalization to unseen 3D objects and real-world scenes that were not trained.

A 3D point cloud completion task is to generate one or multiple completed 3D objects based on incomplete and partial input data. While single-output models suffer from the large missing input data, current multiple-output models require texts, images or depth maps to guide completion tasks, dramatically increasing network model sizes with limited output diversities. This paper proposes a novel probabilistic architecture, MCNet, to address these challenges. MCNet includes a latent shape prior module and a conditional probabilistic diffusion module to explore multiple completion results faithful to the partial input data. We hypothesise that the learnt latent space contains diverse latent shapes. Therefore, through a reverse process of latent shape prior distribution, we can recover a diverse range of latent shapes from the random samples of a Gaussian distribution. Based on the diverse latent shapes and the latent shape of the partial input, we design a conditional probabilistic diffusion model to convert a noise distribution into the distribution of multiple 3D shapes. Critically, MCNet is lightweight and generates flexible output point cloud resolutions without any further training while maintaining the model size constant with the increase of the input and output point cloud resolutions. Experimental results demonstrate that MCNet achieves state-of-the-art performance and also exhibits strong generalization to unseen 3D objects and real-world 3D scenes that are never trained. Code is available at https://github.com/LONG-XI/MCNet .

With the rapid development of medical imaging, sparse-view X-ray 3D reconstruction has become an essential technique for addressing low-dose X-ray imaging challenges. However, due to sparse angular sampling, traditional reconstruction methods often face challenges in handling complex bone and soft tissue structures, leading to information loss and insufficient detail capture. To address these issues, this paper proposes a sparse-view X-ray 3D reconstruction method based on Neural Radiance Fields (NeRF) with a dual-branch feature fusion framework. By synergistically extracting local and global features, this approach enhances the reconstruction of intricate bone and soft tissue structures. For specific applications in regions like the pelvis and aneurism, the method employs depthwise separable convolutions in the local branch to efficiently capture X-ray image details, enhancing the reconstruction of complex bone structures. In the global branch, a pooling Transformer with window mechanisms and hybrid positional encoding is introduced to capture the global features of soft tissue structures like aneurism. Experimental results demonstrate the superiority of this method on multiple medical imaging datasets, particularly in reconstructing complex bone regions and recovering details of soft tissue structures, compared to traditional methods and existing deep learning models.

Many existing Virtual Reality (VR) applications in the Digital Cultural Heritage (DCH) domain are for education purposes.As educational VR DCH experiences become more prevalent, it becomes increasingly important to understand the user and learner experience of such installations.This work reports on a user study (n=30) evaluating three educational VR DCH experiences using three existing User experience (UX) evaluation methodologies from related fields and three learning evaluation methodologies.A total of 31 participants were recruited for the experiment, resulting in a dataset of 30 valid records.Our research seeks to explore the relationship between UX and Learning experience (LX), and their impact on learning in VR DCH experiences.Our results suggest that UX and LX in educational VR DCH experiences can influence certain aspects of learning, such as retention, concentration, motivation, and flexibility.Additionally, specific aspects of the educational VR DCH experience captured evidence by three existing UX evaluation and three learning evaluation methodologies are identified.These include instrumental aspects (ease of use, learnability, efficiency, etc.), stimulation of new experiences, the role of interactions, immersion in VR DCH contexts and flexibility of learning pace and using learning materials.

AI-driven characters in extended reality (XR) healthcare simulations are increasingly used for clinical training, yet their effectiveness, implementation, and quality assurance remain poorly understood. We conducted a systematic review of 132 studies published between January 2015 and July 2025, including 11 randomised controlled trials (RCTs), sourced from biomedical, computing, and education databases and targeted proceedings. Most studies used virtual reality (62.1%) and focused on effectiveness (n = 71), with fewer examining implementation (n = 45) or quality assurance (n = 44). Meta-analysis of two RCTs found a large effect on knowledge and decision-making (Hedges’ g = 1.31, 95% CI 0.08–2.54, I 2 = 85%), while one RCT reported faster task performance with AI-driven characters (g = -0.68, 95% CI -1.32 to -0.04). Certainty of evidence was low due to small samples and high heterogeneity. Implementation success was often associated with phased roll-outs and faculty training, but quality assurance practices—particularly bias audits and transparency measures—were rarely documented. The review proposes the DASEX framework to address these gaps and guide future integration of AI-driven characters in XR training. • First systematic review of AI-driven characters in XR healthcare simulations. • Covers 132 studies (2015–2025) across effectiveness, implementation, and QA. • Meta-analysis of RCTs shows SMD 1.31 for knowledge and − 0.68 for task time. • Introduces DASEX framework for evaluating AI adaptivity, safety, and bias in AI driven healthcare simulations. • Discusses bias mitigation, LMIC access, and XR data privacy safeguards.

Abstract We propose Hi3DFace, a novel framework for simultaneous de‐occlusion and high‐fidelity 3D face reconstruction. To address real‐world occlusions, we construct a diverse facial dataset by simulating common obstructions and present TMANet, a transformer‐based multi‐scale attention network that effectively removes occlusions and restores clean face images. For the 3D face reconstruction stage, we propose a coarse‐medium‐fine self‐supervised scheme. In the coarse reconstruction pipeline, we adopt a face regression network to predict 3DMM coefficients for generating a smooth 3D face. In the medium‐scale reconstruction pipeline, we propose a novel depth displacement network, DDFTNet, to remove noise and restore rich details to the smooth 3D geometry. In the fine‐scale reconstruction pipeline, we design a GCN (graph convolutional network) refiner to enhance the fidelity of 3D textures. Additionally, a light‐aware network (LightNet) is proposed to distil lighting parameters, ensuring illumination consistency between reconstructed 3D faces and input images. Extensive experimental results demonstrate that the proposed Hi3DFace significantly outperforms state‐of‐the‐art reconstruction methods on four public datasets, and five constructed occlusion‐type datasets. Hi3DFace achieves robustness and effectiveness in removing occlusions and reconstructing 3D faces from real‐world occluded facial images.

ABSTRACT The use of generative AI to create responsive and adaptive game content has attracted considerable interest within the educational game design community, highlighting its potential as a tool for enhancing players' understanding of in‐game knowledge. However, designing effective player‐AI interaction to support knowledge representation remains unexplored. This paper presents AIKII, an AI‐enhanced Knowledge Interaction Interface designed to facilitate knowledge representation in educational games. AIKII employs various interaction channels to represent in‐game knowledge and support player engagement. To investigate its effectiveness and user learning experience, we implemented AIKII into The Journey of Poetry, an educational game centered on learning Chinese poetry, and conducted interviews with university students. The results demonstrated that our method fosters contextual and reflective connections between players and in‐game knowledge, enhancing player autonomy and immersion.

Abstract Existing 3D garment reconstruction methods are difficult to implement for online fashion design and e-commerce or special applications. This paper proposes a novel computer-vision framework for 3D garment digital reconstruction, which aims to reconstruct high-quality and realistic virtual 3D garments with fabric mechanic properties for 3D virtual try-on. The new segmentation scheme is proposed to separate the 3D garment point clouds from background points, which is suitable for 3D human shapes and is adaptive for different 3D garment models in different scenes. The new Statistical Outlier Removal algorithm and the learning-based method PointCleanNet are combined to remove noise and outliers in 3D garment point clouds, which provides high-fidelity and high-quality 3D garment point clouds. The 3D garment meshes are then reconstructed from their corresponding point clouds with a modified rolling ball algorithm. Finally, the meshes are improved and converted into physics-based virtual try-on 3D garments with fabric mechanic properties added, which enables the assessment of different body shapes with varied sizes for the same reconstructed 3D garment. Comparison experiments demonstrate that our framework achieves high-quality and realistic 3D garment reconstruction and accurate 3D virtual try-on from 2D garment images. We also demonstrate the proposed framework on a large range of various garments to show this approach has a great potential for garment future technology, such as online garment shopping, garment design and manufacturing.

Abstract In this paper, we propose a few-shot method for pose transfer of anime characters—given a source image of an anime character and a target pose, we transfer the pose of the target to the source character. Despite recent advances in pose transfer on real people images, these methods typically require large numbers of training images of different person under different poses to achieve reasonable results. However, anime character images are expensive to obtain they are created with a lot of artistic authoring. To address this, we propose a meta-learning framework for few-shot pose transfer, which can well generalize to an unseen character given just a few examples of the character. Further, we propose fusion residual blocks to align the features of the source and target so that the appearance of the source character can be well transferred to the target pose. Experiments show that our method outperforms leading pose transfer methods, especially when the source characters are not in the training set.

Abstract 3D object matching and registration on point clouds are widely used in computer vision. However, most existing point cloud registration methods have limitations in handling non‐rigid point sets or topology changes (e.g. connections and separations). As a result, critical characteristics such as large inter‐frame motions of the point clouds may not be accurately captured. This paper proposes a statistical algorithm for non‐rigid point sets registration, addressing the challenge of handling topology changes without the need to estimate correspondence. The algorithm uses a novel Break and Splice framework to treat the non‐rigid registration challenges as a reproduction process and a Dirichlet Process Gaussian Mixture Model (DPGMM) to cluster a pair of point sets. Labels are assigned to the source point set with an iterative classification procedure, and the source is registered to the target with the same labels using the Bayesian Coherent Point Drift (BCPD) method. The results demonstrate that the proposed approach achieves lower registration errors and efficiently registers point sets undergoing topology changes and large inter‐frame motions. The proposed approach is evaluated on several data sets using various qualitative and quantitative metrics. The results demonstrate that the Break and Splice framework outperforms state‐of‐the‐art methods, achieving an average error reduction of about 60% and a registration time reduction of about 57.8%.

Generating the missing data of 3D object point clouds from partial observations is a challenging task. Existing state-of-the-art learning-based 3D point cloud completion methods tend to use a limited number of categories/classes of training data and regenerate the entire point cloud based on the training datasets. As a result, output 3D point clouds generated by such methods may lose details (i.e. sharp edges and topology changes) due to the lack of multi-class training. These methods also lose the structural and spatial details of partial inputs due to the models do not separate the reconstructed partial input from missing points in the output. In this paper, we propose a novel deep learning network - TreeNet for 3D point cloud completion. TreeNet has two networks in hierarchical tree-based structures: TreeNet-multiclass focuses on multi-class training with a specific class of the completion task on each sub-tree to improve the quality of point cloud output; TreeNet-binary focuses on generating points in missing areas and fully preserving the original partial input. TreeNet-multiclass and TreeNet-binary are both network decoders and can be trained independently. TreeNet decoder is the combination of TreeNet-multiclass and TreeNet-binary and is trained with an encoder from existing methods (i.e. PointNet encoder). We compare the proposed TreeNet with five state-of-the-art learning-based methods on fifty classes of the public Shapenet dataset and unknown classes, which shows that TreeNet provides a significant improvement in the overall quality and exhibits strong generalization to unknown classes that are not trained.

Optimization plays a pivotal role in computer graphics and vision. Learning-based optimization algorithms have emerged as a powerful optimization technique for solving problems with robustness and accuracy because it learns gradients from data without calculating the Jacobian and Hessian matrices. The key aspect of the algorithms is the least-squares method, which formulates a general parametrized model of unconstrained optimizations and makes a residual vector approach to zeros to approximate a solution. The method may suffer from undesirable local optima for many applications, especially for point cloud registration, where each element of transformation vectors has a different impact on registration. In this paper, Reweighted Discriminative Optimization (RDO) method is proposed. By assigning different weights to components of the parameter vector, RDO explores the impact of each component and the asymmetrical contributions of the components on fitting results. The weights of parameter vectors are adjusted according to the characteristics of the mean square error of fitting results over the parameter vector space at per iteration. Theoretical analysis for the convergence of RDO is provided, and the benefits of RDO are demonstrated with tasks of 3D point cloud registrations and multi-views stitching. The experimental results show that RDO outperforms state-of-the-art registration methods in terms of accuracy and robustness to perturbations and achieves further improvement than non-weighting learning-based optimization.

Reducing accident rate is a primary goal of construction safety. In this paper, we present a large scale study of using virtual reality technology for safety training. Beyond the training, a technology framework is proposed to assess the fitness of construction workers (e.g. suitability of people with underlining health conditions to work under particular construction environments). The new virtual construction system consists of a Brain-Computer Interface (BCI) of electroencephalography (EEG) neural network to capture EEG signals of users during the virtual simulation training continuously to achieve user profiling. For real-time assessment of the accident susceptibility of a worker under various construction environments, a deep learning neural network is trained to process the EEG crops and a clipping training algorithm that classifies small segments of the EEG dataset is used to improve the computational performance of the system. Physiology data of the person during the training, i.e. blood pressure and heart rate, is also recorded. Based on the EEG data and the physiology data, a statistic model is used in the safety assessment framework to set up the risk standard. The study has tested 117 workers who were employed by the construction sites in Shanghai. People who were tested in the risk group were further underwent medical examinations for risk related medical conditions that deemed unsuitable for working in construction sites. Results show six of the nine workers identified by the VR system have been medically confirmed unsuitable, thus, over 80% accuracy of our virtual reality training and assessment system. Our proposed system can be used as a tool for understanding risk conditions of workers and safety training.

In this paper, we present an approach for generating dialogues for characters within the context of computational narratives \nusing personality-based features for deep neural networks. The approach integrates the requirements of both narrative genres and personality traits for the definition of character-based stylistic models. \nThe modelling of characters’ features from existing datasets of complete stories permits the generation of personality-rich character dialogues. We present early results from an evaluation based on a sample of characters’ personality traits across different narrative genres, \ndemonstrating variability in the resulting dialogues

Abstract Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.

Surgical smoke removal algorithms can improve the quality of intra-operative imaging and reduce hazards in image-guided surgery, a highly desirable post-process for many clinical applications. These algorithms also enable effective computer vision tasks for future robotic surgery. In this article, we present a new unsupervised learning framework for high-quality pixel-wise smoke detection and removal. One of the well recognized grand challenges in using convolutional neural networks (CNNs) for medical image processing is to obtain intra-operative medical imaging datasets for network training and validation, but availability and quality of these datasets are scarce. Our novel training framework does not require ground-truth image pairs. Instead, it learns purely from computer-generated simulation images. This approach opens up new avenues and bridges a substantial gap between conventional non-learning based methods and which requiring prior knowledge gained from extensive training datasets. Inspired by the Generative Adversarial Network (GAN), we have developed a novel generative-collaborative learning scheme that decomposes the de-smoke process into two separate tasks: smoke detection and smoke removal. The detection network is used as prior knowledge, and also as a loss function to maximize its support for training of the smoke removal network. Quantitative and qualitative studies show that the proposed training framework outperforms the state-of-the-art de-smoking approaches including the latest GAN framework (such as PIX2PIX). Although trained on synthetic images, experimental results on clinical images have proved the effectiveness of the proposed network for detecting and removing surgical smoke on both simulated and real-world laparoscopic images.

This paper proposes a stable and accurate object registration pipeline for markerless augmented reality applications. We present two novel algorithms for object recognition and matching to improve the registration accuracy from model to scene transformation via point cloud fusion. Whilst the first algorithm effectively deals with simple scenes with few object occlusions, the second algorithm handles cluttered scenes with partial occlusions for robust real-time object recognition and matching. The computational framework includes a locally supported Gaussian weight function to enable repeatable detection of 3D descriptors. We apply a bilateral filtering and outlier removal to preserve edges of point cloud and remove some interference points in order to increase matching accuracy. Extensive experiments have been carried to compare the proposed algorithms with four most used methods. Results show improved performance of the algorithms in terms of computational speed, camera tracking and object matching errors in semi-cluttered and partial-occluded scenes.

In this paper, we develop a new approach to blending of constant and varying parametric surfaces with curvature continuity. We propose a new mathematical model consisting of a vector-valued sixth-order partial differential equation (PDE) and time-dependent blending boundary constraints, and develop an approximate analytical solution of the mathematical model. The good accuracy and high computational efficiency are demonstrated by comparing the new approximate analytical solution with the corresponding accurate closed form solution. We also investigate the influence of the second partial derivatives on the continuity at trimlines, and apply the new approximate analytical solution in blending of constant and varying parametric surfaces with curvature continuity.

Markerless Augmented Reality (AR) registration using the standard Homography matrix is unstable, and for image-based registration it has very low accuracy. In this paper, we present a new method to improve the stability and the accuracy of marker-less registration in AR. Based on the Visual Simultaneous Localization and Mapping (V-SLAM) framework, our method adds a three-dimensional dense cloud processing step to the state-of-the-art ORB-SLAM in order to deal with mainly the point cloud fusion and the object recognition. Our algorithm for the object recognition process acts as a stabilizer to improve the registration accuracy during the model to the scene transformation process. This has been achieved by integrating the Hough voting algorithm with the Iterative Closest Points(ICP) method. Our proposed AR framework also further increases the registration accuracy with the use of integrated camera poses on the registration of virtual objects. Our experiments show that the proposed method not only accelerates the speed of camera tracking with a standard SLAM system, but also effectively identifies objects and improves the stability of markerless augmented reality applications.

Multi-user applications can be complex to develop due to their large or intricate nature. Many of the issues encountered are related to performance and security. These issues are exacerbated when the scale of the application increases. This paper introduces a novel distributed architecture called OpenGL|D (OpenGL Distributed). This technology enables an application to pass through the graphical calls between a Virtual Machine (VM) and the graphics processing unit (GPU) on the native host across a network. This ability allows applications to run inside a virtual machine (VM), whilst still benefiting from hardware accelerated performance from the GPU for the computationally intensive graphical processing. This allows for the development of 3D software requiring no dependencies on specific hardware or technology other than ANSI C and a network stack, demonstrating our approach to platform agnostic development and digital preservation.

Mixed Reality (MR) is of increasing interest within technology-driven modern medicine but is not yet used in everyday practice. This situation is changing rapidly, however, and this paper explores the emergence of MR technology and the importance of its utility within medical applications. A classification of medical MR has been obtained by applying an unbiased text mining method to a database of 1,403 relevant research papers published over the last two decades. The classification results reveal a taxonomy for the development of medical MR research during this period as well as suggesting future trends. We then use the classification to analyse the technology and applications developed in the last five years. Our objective is to aid researchers to focus on the areas where technology advancements in medical MR are most needed, as well as providing medical practitioners with a useful source of reference.

Abstract Despite great strides in past years are being made to generate motions of elastic materials such as cloth and biological skin in virtual world, unfortunately, the computational cost of realistic high‐resolution simulations currently precludes their use in interactive applications. Thin elastic materials such as cloth and biological skin often exhibit complex nonlinear elastic behaviors. However, modeling elastic nonlinearity can be computationally expensive and numerically unstable, imposing significant challenges for their use in interactive applications. This paper presents a novel simulation framework for simulating realistic material behaviors with interactive frame rate. Central to the framework is the use of a constraint‐based multi‐resolution solver for efficient and robust modeling of the material nonlinearity. We extend a strain‐limiting method to work on deformation gradients of triangulated surface models in three‐dimensional space with a novel data structure. The simulation framework utilizes an iterative nonlinear Gauss–Seidel procedure and a multilevel hierarchy structure to achieve computational speedups. As material nonlinearity are generated by enforcing strain‐limiting constraints at a multilevel hierarchy, our simulation system can rapidly accelerate the convergence of the large constraint system with simultaneous enforcement of boundary conditions. The simplicity and efficiency of the framework makes simulations of highly realistic thin elastic materials substantially fast and is applicable of simulations for interactive applications. Copyright © 2015 John Wiley & Sons, Ltd.

Simulation of deformable objects has become indispensable in many virtual reality applications. Linear finite element algorithms are frequently applied in interactive physics simulation in order to ensure computational efficiency. However, there exists a variety of situations in which higher order simulation accuracy is expected to improve physical behaviors of deformable objects to match their real-world counterparts. For example in the context of virtual surgery, interactive surgical manipulations mandate algorithmic requirements to maintain both interactive frame rates and simulation accuracy, presenting major challenges in simulation methods. In this paper, we present an interactive system for efficient finite element based simulation of hyperplastic solids with more accurate physics behaviors compared with that of standard corotational methods. Our approach begins with a physics model to mitigate drawbacks of the corotational linear elasticity in preserving energy and momenta. A new damping model is presented which takes into account the differential of rotation to compensate the loss of momenta due to rotations. Thus, more accurate simulations can be achieved with this new model, whereas standard corotational methods using rotated damping to handle energy dissipation does not preserve momenta. We then present a real time simulation framework for computing finite element based deformable solids with full capability allowing complex objects to collide and interact with each other. A constrained system is also provided for robust control and the ease of use the simulation system. We demonstrate the parallel implementation to enable realistic and stable physics behaviors of large deformations capable of handling unpredictable user inputs in interactive virtual environments. The implementation details and insights on practical considerations in implementation such as our experience in parallel computation of the physics for mesh-based finite element objects would be useful for people who wish to develop real-time applications in this area.

Many virtual reality-based applications involve simulations of micro-structures such as hair, fibers and textile yarns, as well as ropes, flexible wires and tubes. In virtual surgery, for example, flexible wires and tubes are common medical instruments and devices. Core to the simulations is the robust physics-based computation of elastic rods. In this paper, we present a volumetric finite element based approach to simulating rod-like objects with real-time performance suitable for interactive virtual environments. A sequence of Cosserat joints (tiny volumetric elastic joints) linked by rigid bar segments are used to compute the elastic rod objects. By construction, each of the joints is equipped with its own mass, degrees of freedom (DOFs) with a small volumetric deformation field to measure deformation energies due to stretching, shearing, bending, and twisting about the centerline curve of the long flexible object. Therefore, a generalized continuum formulation is derived to compute both bending and twisting deformations of elastic rods, resulting a simple and general simulation model to facilitate efficient physics computations, whereas conversional simulation methods for elastic rods require explicitly decoupling between bending and twisting deformations. In this paper, we show simulations of a wide range of object behaviors for interactive virtual reality applications.

Virtual reality based preoperative planning for Minimally Invasive Vascular Intervention is useful, not only for increasing the success rate of operation, but also used as a training tool for improving doctors' skills. In this paper we present an interactive 3D preoperative planning and training system with haptic device. In this system, we present an intelligent trajectory planning algorithm for searching an optimal path automatically along the centerline in two ways: from the suitable inserting point to the specific target location or to the most of objectives. Also, for the purpose of interactive training, we connect the haptic device to the end of guide wire and propose an algorithm that enables the simulator to model guide wire and catheter insertions realistically through essential operations i.e. pushing, pulling and twisting actions. We demonstrate experiment results to show that the 3D preoperative planning and training system is usable for simulating guide wire insertion procedures with complex blood vessel structures.

A virtual reality (VR) based training system for Minimally Invasive Vascular Surgery can be a very useful training tool for improving skills and reducing errors in operation. Computer vision techniques have the potential to be incorporated into a VR based training system for developing low cost, high accuracy and flexible systems in this area. In this paper, we present an interactive 3D training system that uses stereo vision to capture hand movements as the input operations for the system. The standard operations i.e. pushing, pulling and twisting are captured with stereo vision based on the improved Camshift tracking algorithm and parallel alignment model theory to acquire hand gestures information. We present a new approach to calculate virtual pushing/pulling force and turning angle as extra inputs for understanding these essential operations. In addition, an algorithm that enables the simulator to model guide wire and catheter insertions realistically is presented through these basic actions. The experiment results demonstrate that stereo vision based training system is useful and effective for simulating guide wire insertion procedures with low system cost and flexible operations.

Abstract Guidewire insertion is an imperative task of minimally invasive medical procedures. During the procedure, surgeons need to steer long flexible thin wires through patient's blood vessels to reach a clinical target. In this paper, we present a novel approach to model haptics of guidewire insertion process for training simulation. The algorithm also allows for the analysis of the insertion process through subtle physical behaviours of guidewires via force feedbacks. The method includes a 6‐DoF dynamic coupling between a rigid body, i.e. the virtual tool and the deformation of the wire simulated as an elastic rod. Instead of using the frictional contact force or the acceleration of the guidewire tip for haptic feedbacks, we compute constrained forces by directly connecting the virtual tool to the end of the guidewire. Therefore, the coupling scheme transmits haptic interactions through constrained dynamics between the virtual tool and the guidewire. Both positional and rotational control modes are implemented and evaluated with respect to the dynamics of the guidewire, user inputs and feedback forces. Experiments highlight the usability of our algorithm for an insertion procedure simulation with complex blood vessel structures. Copyright © 2011 John Wiley & Sons, Ltd.