Abstract The use of haptic simulation for emergency procedures in nursing training presents a viable, versatile and affordable alternative to traditional mannequin environments. In this paper, an evaluation is performed in a virtual environment with a head-mounted display and haptic devices, and also with a mannequin. We focus on a chest decompression, a life-saving invasive procedure used for trauma-associated cardiopulmonary resuscitation (and other causes) that every emergency physician and/or nurse needs to master. Participants’ heart rate and blood pressure were monitored to measure their stress level. In addition, the NASA Task Load Index questionnaire was used. The results show the approved usability of the VR environment and that it provides a higher level of immersion compared to the mannequin, with no statistically significant difference in terms of cognitive load, although the use of VR is perceived as a more difficult task. We can conclude that the use of haptic-enabled virtual reality simulators has the potential to provide an experience as stressful as the real one while training in a safe and controlled environment.

This paper describes the integration of haptics support into a virtual reality training simulation aimed at skills retention for paramedics. We focus on a chest decompression, a life-saving invasive procedure used for trauma-associated cardiopulmonary resuscitation (and other causes) that every emergency physician needs to master. It is not regularly performed by a paramedic, however, and therefore skills maintenance is a challenge. In our simulation, a virtual Russell PneumoFix-8 device is used to carry out the procedure and it is controlled with the 3D Systems Touch grounded force feedback device. We describe how this device has been integrated into an immersive virtual environment so that it or any other tool can be used at any location in the scene. Quantitative data has been obtained from an evaluation exercise carried out with 21 paramedics. The majority of these participants reported a good feeling of presence, according to the Spatial Presence Experience Scale. They indicated strongly that the use of haptic-enabled simulators that include the kind of interaction techniques implemented in our simulator would be beneficial for training and skills retention. The realism of using the simulator at a 1 to 1 scale was also highly scored. A System Usability Scale was also calculated and the results show that the simulator is close to an acceptable standard for usability but more work is needed. We will address this in future work.

Virtual reality (VR) and augmented reality (AR) have continued to increase in popularity over the past decade. However, there are still issues with how much space is required for room-scale VR and experiences are still lacking from haptic feedback. We present LevelEd SR, a substitutional reality level design workflow that combines AR and VR systems and is built for consumer devices. The system enables passive haptics through the inclusion of physical objects from within a space into a virtual world. A validation study (17 participants) has produced quantitative data that suggests players benefit from passive haptics in entertainment VR games with an improved game experience and increased levels of presence. Including objects, such as real-world furniture that is paired with a digital proxy in the virtual world, also opens up more spaces to be used for room-scale VR. We evaluated the workflow and found that participants were accepting of the system, rating it positively using the System Usability Scale questionnaire and would want to use it again to experience substitutional reality.

We present VRIA, a Web-based framework for creating Immersive Analytics (IA) experiences in Virtual Reality. VRIA is built upon WebVR, A-Frame, React and D3.js, and offers a visualization creation workflow which enables users, of different levels of expertise, to rapidly develop Immersive Analytics experiences for the Web. The use of these open-standards Web-based technologies allows us to implement VR experiences in a browser and offers strong synergies with popular visualization libraries, through the HTML Document Object Model (DOM). This makes VRIA ubiquitous and platform-independent. Moreover, by using WebVR's progressive enhancement, the experiences VRIA creates are accessible on a plethora of devices. We elaborate on our motivation for focusing on open-standards Web technologies, present the VRIA creation workflow and detail the underlying mechanics of our framework. We also report on techniques and optimizations necessary for implementing Immersive Analytics experiences on the Web, discuss scalability implications of our framework, and present a series of use case applications to demonstrate the various features of VRIA. Finally, we discuss current limitations of our framework, the lessons learned from its development, and outline further extensions.

Virtual reality entertainment and serious games popularity has continued to rise but the processes for level design for VR games has not been adequately researched. Our paper contributes LevelEd VR; a generic runtime virtual reality level editor that supports the level design workflow used by developers and can potentially support user generated content. We evaluated our LevelEd VR application and compared it to an existing workflow of Unity on a desktop. Our current research indicates that users are accepting of such a system, and it has the potential to be preferred over existing workflows for VR level design. We found that the primary benefit of our system is an improved sense of scale and perspective when creating the geometry and implementing gameplay. The paper also contributes some best practices and lessons learned from creating a complex virtual reality tool, such as LevelEd VR.

Abstract Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic‐based interactive MR framework that is beyond current geometry‐based approaches, offering a step change in generating high‐level context‐aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object‐specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real‐world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material‐aware prototype system for context‐aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real‐time semantic‐level interactions.

Surgical smoke removal algorithms can improve the quality of intra-operative imaging and reduce hazards in image-guided surgery, a highly desirable post-process for many clinical applications. These algorithms also enable effective computer vision tasks for future robotic surgery. In this article, we present a new unsupervised learning framework for high-quality pixel-wise smoke detection and removal. One of the well recognized grand challenges in using convolutional neural networks (CNNs) for medical image processing is to obtain intra-operative medical imaging datasets for network training and validation, but availability and quality of these datasets are scarce. Our novel training framework does not require ground-truth image pairs. Instead, it learns purely from computer-generated simulation images. This approach opens up new avenues and bridges a substantial gap between conventional non-learning based methods and which requiring prior knowledge gained from extensive training datasets. Inspired by the Generative Adversarial Network (GAN), we have developed a novel generative-collaborative learning scheme that decomposes the de-smoke process into two separate tasks: smoke detection and smoke removal. The detection network is used as prior knowledge, and also as a loss function to maximize its support for training of the smoke removal network. Quantitative and qualitative studies show that the proposed training framework outperforms the state-of-the-art de-smoking approaches including the latest GAN framework (such as PIX2PIX). Although trained on synthetic images, experimental results on clinical images have proved the effectiveness of the proposed network for detecting and removing surgical smoke on both simulated and real-world laparoscopic images.

This research project developed a Virtual Reality (VR) training simulator for paramedic procedures. Currently needle cricothyroidotomy and chest drain are modelled, which could form part of a larger system for training paramedics with VR in various other procedures. The simulator incorporates a number of advanced VR technologies including Oculus Rift and haptic feedback. We have gained input and feedback from NHS paramedics and several related organisation to design the system and provide feedback and evaluation of the preliminary working prototype.

This research project developed a Virtual Reality (VR) training simulator for the CPR procedure. This is designed for use training school children. It can also form part of a larger system for training paramedics with VR. The simulator incorporates a number of advanced VR technologies including Oculus Rift and Leap motion. We have gained input from NHS paramedics and several related organisation to design the system and provide feedback and evaluation of the preliminary working prototype.

Abstract The visual fidelity of realistic renderings in Computer Graphics depends fundamentally upon how we model the appearance of objects resulting from the interaction between light and matter reaching the eye. In this paper, we survey the research addressing appearance modelling of living human tissue. Among the many classes of natural materials already researched in Computer Graphics, living human tissues such as blood and skin have recently seen an increase in attention from graphics research. There is already an incipient but substantial body of literature on this topic, but we also lack a structured review as presented here. We introduce a classification for the approaches using the four types of human tissues as classifiers. We show a growing trend of solutions that use first principles from Physics and Biology as fundamental knowledge upon which the models are built. The organic quality of visual results provided by these Biophysical approaches is mainly determined by the optical properties of biophysical components interacting with light. Beyond just picture making, these models can be used in predictive simulations, with the potential for impact in many other areas.

Visualization and virtual environments (VEs) have been two interconnected parallel strands in visual computing for decades. Some VEs have been purposely developed for visualization applications, while many visualization applications are exemplary showcases in general-purpose VEs. Because of the development and operation costs of VEs, the majority of visualization applications in practice have yet to benefit from the capacity of VEs. In this paper, we examine this status quo from an information-theoretic perspective. Our objectives are to conduct cost-benefit analysis on typical VE systems (including augmented and mixed reality, theater-based systems, and large powerwalls), to explain why some visualization applications benefit more from VEs than others, and to sketch out pathways for the future development of visualization applications in VEs. We support our theoretical propositions and analysis using theories and discoveries in the literature of cognitive sciences and the practical evidence reported in the literatures of visualization and VEs.

Navigating a powered wheelchair and avoiding collisions is often a daunting task for new wheelchair users. It takes time and practice to gain the coordination needed to become a competent driver and this can be even more of a challenge for someone with a disability. We present a cost-effective virtual reality (VR) application that takes advantage of consumer level VR hardware. The system can be easily deployed in an assessment centre or for home use, and does not depend on a specialized high-end virtual environment such as a Powerwall or CAVE. This paper reviews previous work that has used virtual environments technology for training tasks, particularly wheelchair simulation. We then describe the implementation of our own system and the first validation study carried out using thirty three able bodied volunteers. The study results indicate that at a significance level of 5 percent then there is an improvement in driving skills from the use of our VR system. We thus have the potential to develop the competency of a wheelchair user whilst avoiding the risks inherent to training in the real world. However, the occurrence of cybersickness is a particular problem in this application that will need to be addressed.

We have developed an augmented reality tool for radiotherapy to view the real world scene, i.e. the patient on a treatment couch, combined with computer graphics content, such as planning image data and any defined outlines of organ structures. We have deployed our software to a number of consumer electronics devices (iPad, Android tablets, MS HoloLens). We suggest that, in contrast to other augmented reality tools explored for radiotherapy [1], due to the wide availability and low cost of the hardware platforms considered, associated with their increasing computational and graphic power, our system has strong potential as a tool for visualization of medical information for clinicians and other radiotherapy professionals, as a device for patient positioning for radiotherapy treatment, and as an educational tool for patients to visualize their treatment and demonstrate to patients e.g. the importance of compliance with instructions around bladder filling and rectal suppositories. Accuracy of virtual content placement and user evaluation of our system has been experimentally investigated.

This paper presents recent additions to our Wheelchair-VR application, in particular the use of different drive configurations. We have previously shown that Wheelchair-VR can be used to improve driving skills. Here we consider the utility of the application in allowing users who are in the process of purchasing or upgrading a wheelchair to experience different configurations and options in a cost-effective virtual environment. A preliminary study is presented, which suggests that this approach can be effective.

Planning treatment of acute cardiac events that limit the blood supply to major organs is particularly difficult for interventional cardiologists. The treatment of pathologies, such as vascular stenosis, can have numerous unforeseen consequences as the blood resumes its flow. Their decisions are largely based on 2D medical imagery and their own experience. This work in progress presents a virtual reality blood simulation tool that will augment and improve a clinicians decision-making arsenal, and an outline of the technologies required for this component of the cyber hospital of the future. The tool displays and provides interaction with vital information on how blood may continue to flow through the cardiovascular system after treatment. The blood flow will be simulated using a bespoke implementation of the increasingly effective smoothed particle hydrodynamics model.

Medical training systems often require the use of haptic interaction with deformable models and high visual fidelity. It is necessary to get a compromise between the geometric model and the deformation model in order to achieve the required frame rate for haptic interaction of 1000Hz. Frequently low level geometric models are used to describe the geometry of the object. This produces large data structures for which it is difficult to reach real-time force feedback.This paper proposes to use a novel model based on Bezier hyperpatches to represent the object, computing the deformation using a mass-spring system. This model can represent free form objects in a concise way. A small test application to simulate palpation with a haptic probe is presented.

Mixed Reality (MR) is of increasing interest within technology-driven modern medicine but is not yet used in everyday practice. This situation is changing rapidly, however, and this paper explores the emergence of MR technology and the importance of its utility within medical applications. A classification of medical MR has been obtained by applying an unbiased text mining method to a database of 1,403 relevant research papers published over the last two decades. The classification results reveal a taxonomy for the development of medical MR research during this period as well as suggesting future trends. We then use the classification to analyse the technology and applications developed in the last five years. Our objective is to aid researchers to focus on the areas where technology advancements in medical MR are most needed, as well as providing medical practitioners with a useful source of reference.

We have designed LiTu (La'Tu - Light Tube) as a customisable and low-cost (ca 30 Euros) human-computer interface. It is composed of an acrylic tube, a ball-bearing mirror, six LEDs and a webcam. Touching the tube causes frustrated internal reflection of light due to a change in the critical angle at the acrylic-skin boundary. Scattered light within the tube is reflected off the mirror into the camera at the opposite end for image processing. Illuminated contact regions in the video frames are segmented and processed to generate 2D information such as: pitch and volume, or x and y coordinates of a graphic. We demonstrate the functionality of LiTu both as a musical instrument and as an interactive computer graphics controller. For example, various musical notes can be generated by touching specific regions around the surface of the tube. Volume can be controlled by sliding a finger down the tube and pitch by sliding the finger radially. We demonstrate the adaptable nature of LiTu's touch interface and discuss our plans to explore future physical modifications of the device.

We are developing a novel device for measuring hand power grip using frustrated total internal reflection of light in acrylic. Our method uses a force sensitive resistor to calibrate the force of a power grip as a function of the area and light intensity. This research is work in progress but results so far augur well for its applicability in medical and other application areas. The grip measurement device allows the patient and doctor to see the change in grip over time and projects this information directly onto the back of the patient's hand.

Work in progress for the development of a novel virtual training environment for training a kidney biopsy procedure is presented. Our goal is to provide an affordable high fidelity simulation through the integration of some of the latest off-the-shelf technology components. The range of forces that are encountered during this procedure have been recorded using a custom designed force sensitive glove and then applied within the simulation.

We have designed and developed a virtual environment to train rugby ball passing skills. Seeking to validate the system's ability to correctly aid training, an initial experiment was performed to examine the effect of stereoscopic technology and the physical screen's setup on the user's ability to perceive virtual distances correctly. Four participants with at least 10 years' experience of playing local-level rugby took part in the experiment to compare the combinations of different user positions compared to the physical screen, the use of stereoscopic presentation and the use of a floor screen to extend the field of view of the virtual scene. Conversely to what was expected, the participants did not respond well to perceived distance and these variables had little effect on the distance thrown.

Abstract Guidewire insertion is an imperative task of minimally invasive medical procedures. During the procedure, surgeons need to steer long flexible thin wires through patient's blood vessels to reach a clinical target. In this paper, we present a novel approach to model haptics of guidewire insertion process for training simulation. The algorithm also allows for the analysis of the insertion process through subtle physical behaviours of guidewires via force feedbacks. The method includes a 6‐DoF dynamic coupling between a rigid body, i.e. the virtual tool and the deformation of the wire simulated as an elastic rod. Instead of using the frictional contact force or the acceleration of the guidewire tip for haptic feedbacks, we compute constrained forces by directly connecting the virtual tool to the end of the guidewire. Therefore, the coupling scheme transmits haptic interactions through constrained dynamics between the virtual tool and the guidewire. Both positional and rotational control modes are implemented and evaluated with respect to the dynamics of the guidewire, user inputs and feedback forces. Experiments highlight the usability of our algorithm for an insertion procedure simulation with complex blood vessel structures. Copyright © 2011 John Wiley & Sons, Ltd.

A needle insertion is a widely performed procedure used either to inject fluids, to retrieve samples or as an introducing conduit for more advanced procedures.Needle insertions, like most medical procedures, pose an inherent risk of complication to the patient. This risk has prompted the development of a variety of haptic training simulators to aid in the education of practitioners before they attempt needle procedures upon a human. A common trend in needle insertion simulation is to use and possibly modify pre-existing commercial haptic solutions,saving on development cost and time. This study reviews current needle inserting simulations, focusing on the haptic hardware solutions used. Five commercially available haptic devices (SensAble Technologies Omni,Desktop and Premium 1.5 6 DOF, Novint's Falcon and Mimics Mantis), are then tested to evaluate their effectiveness for use in needle insertion simulations. A conclusion is drawn, with advice for those producing needle insertion simulation solutions.

Interventional radiology is a rapidly expanding speciality using minimally invasive techniques to treat a multitude of clinical problems. Current work in progress aims to create an affordable virtual training tool to reduce training times and patient risk during a trainee practitioners learning cycle. The procedure of arterial catheterisation has been broken down into a number of subtasks, one of which requires an operator to locate the femoral artery pulse by palpation. This is performed in preparation for a needle insertion to allow the entry of a guide wire and catheter into the patient. This paper presents the current state of research into a unique solution for affordable haptic simulation of pulse palpation in a virtual environment.

Abstract We present an integrated system for training ultrasound (US) guided needle puncture. Our aim is to provide a validated training tool for interventional radiology (IR) that uses actual patient data. IR procedures are highly reliant on the sense of touch and so haptic hardware is an important part of our solution. A hybrid surface/volume haptic rendering of an US transducer is proposed to constrain the device to remain outside the bony structures when scanning the patient's skin. A volume haptic model is proposed that implements an effective model of needle puncture. Force measurements have been made on real tissue and the resulting data is incorporated into the model. The other input data required is a computed tomography (CT) scan of the patient that is used to create the patient specific models. It is also the data source for a novel simulation of a virtual US scanner, which is used to guide the needle to the correct location. Copyright © 2007 John Wiley & Sons, Ltd.

Abstract The medical domain provides excellent opportunities for the application of computer graphics, visualization and virtual environments, with the potential to help improve healthcare and bring benefits to patients. This survey paper provides a comprehensive overview of the state‐of‐the‐art in this exciting field. It has been written from the perspective of both computer scientists and practising clinicians and documents past and current successes together with the challenges that lie ahead. The article begins with a description of the software algorithms and techniques that allow visualization of and interaction with medical data. Example applications from research projects and commercially available products are listed, including educational tools; diagnostic aids; virtual endoscopy; planning aids; guidance aids; skills training; computer augmented reality and use of high performance computing. The final section of the paper summarizes the current issues and looks ahead to future developments.

The medical domain provides excellent opportunities for the application of computer graphics, visualization, and virtual environments, with the potential to help improve healthcare and bring benefits to patients. Possible applications include anatomical educational tools; patient education; diagnostic aids; virtual endoscopy; planning aids; guidance aids; skills training; and computer augmented reality. This talk provides a comprehensive overview of the state-of-the-art in this exciting field, including examples from research projects and commercially available products. The term cybermedicine was discussed and used to categorise those medical applications that can be delivered via the World Wide Web, preferably in the context of a collaborative virtual environment. The issues for effective cybermedicine was highlighted, and we will look ahead to future developments

Abstract If we were to have a Grid infrastructure for visualization, what technologies would be needed to build such an infrastructure, what kind of applications would benefit from it, and what challenges are we facing in order to accomplish this goal? In this survey paper, we make use of the term ‘visual supercomputing’ to encapsulate a subject domain concerning the infrastructural technology for visualization. We consider a broad range of scientific and technological advances in computer graphics and visualization, which are relevant to visual supercomputing. We identify the state‐of‐the‐art technologies that have prepared us for building such an infrastructure. We examine a collection of applications that would benefit enormously from such an infrastructure, and discuss their technical requirements. We propose a set of challenges that may guide our strategic efforts in the coming years.

The NOVICE project (Network-Oriented Visualization in a Clinical Environment) is developing a system for medical image data analysis and visualization through the Web with the intention of demonstrating the effectiveness of such a system in a clinical environment. One element being addressed by NOVICE is the provision of teleradiology tools. The paper describes how three-dimensional models can be included in a teleradiology application by utilising the Virtual Reality Modelling Language (VRML) and multi-user extensions, including annotation tools. The result is the world's first Web based system for 3D teleradiology. We present a detailed overview of the implementation and operation of this system, and indicate results from early use.

Applications of virtual reality (VR) and augmented reality (AR) in medicine require real-time visualization and modeling of large three-dimensional data sets. Consequently, these applications require powerful computation, extensive high-bandwidth memory, and fast communication links. In the past, the manufacturers of medical imaging equipment produced their own special-purpose proprietary hardware for image processing and solid graphics. Due to the developments in computer hardware in general and in graphics accelerators in particular, there is a trend toward replacing the proprietary hardware off-the-shelf (OTS) equipment. Computer graphics itself has advanced in its quest for realism. Generic algorithms such as shading, texture mapping, and volume rendering have been developed to meet the resultant ever increasing requirements. Advances in both the OTS CPU and graphics hardware have enabled real-time implementations of these algorithms, thereby facilitating many of the medical VR/AR applications used today. The development of graphics libraries such as OpenGL has also been an important factor. These libraries provide an underlying portable software platform that optimizes the utilization of the available graphics hardware. OpenGL has become a standard graphics application programming interface, particularly for graphics-intensive applications, and more and more OTS systems provide hardware implementations of OpenGL commands. The review paper follows the evolution of these technologies and examines their crucial role in enabling the appearance of the current VR/AR applications in medicine and provides a look at current trends and future possibilities.