Endotracheal Intubation is a lifesaving procedure in which a tube is passed through the mouth into the trachea (windpipe) to maintain an open airway and facilitate artificial respiration. It is a complex psychomotor skill, which requires significant training and experience to prevent complications. The current methods of training, including manikins and cadaver, have limitations in terms of their availability for early medical professionals to learn and practice. These training options also have limitations in terms of presenting high risk/ difficult intubation cases for experts to mentally plan their approach in high-risk scenarios prior to the procedure. In this paper, we present the design considerations and challenges of AirwayVR: virtual reality-based simulation trainer for intubation training for two different target audience (medical professionals) with two different objectives. The first one is to use AirwayVR as an introductory platform to learn and practice intubation in virtual reality for novice learners (Medical students and residents). The second objective is to utilize this technology as a Just-in-time training platform for experts to mentally prepare for a complex case prior to the procedure.

Endotracheal Intubation is a lifesaving procedure in which a tube is passed through the mouth into the trachea (windpipe) to maintain an open airway and facilitate artificial respiration. It is a complex psychomotor skill, which requires significant training and experience to prevent complications. The current methods of training, including manikins and cadaver, have limitations in terms of their availability for early medical professionals to learn and practice. These training options also have limitations in terms of presenting high risk/difficult intubation cases for experts to mentally plan their approach in high-risk scenarios prior to the procedure. In this demo, we present AirwayVR: virtual reality-based simulation trainer for intubation training. Our goal is to utilize virtual reality platform for intubation skills training for two different target audience (medical professionals) with two different objectives. The first one is to use AirwayVR as an introductory platform to learn and practice intubation in virtual reality for novice learners (Medical students and residents). The second objective is to utilize this technology as a Just-in-time training platform for experts to mentally prepare for a complex case prior to the procedure.

Objective: In recent years, the training of novice medical professionals with simulated environments such as virtual reality (VR) and augmented reality (AR) has increased dramatically. However, the usability of these technologies is limited due to the complexity involved in creating the clinical content. To be comparable to a clinical environment, the simulation platform should include real-world learning parameters such as patient physiology, emotions, and clinical team behaviors. Incorporating such nondeterministic parameters has historically required faculty to possess advanced programming skills. Lack of effective software for instructors to easily develop VR curriculum content is a hurdle in developing VR based curriculum. Method: We address this challenge through a software platform that simplifies the creation of Interactive Mixed Reality (IMR) scenarios. Three educational components we were able to embed into an IMR scenario includes 1) integrated 360-degree video recording of a clinical encounter to provide a first-person perspective, 2) rich annotated knowledge content, and 3) assessment questionnaire. We developed a sepsis prevention education scenario using the IMR software to demonstrate the potential of enhancing simulated medical training by accelerating clinical exposure for novice students. Result: An IRB approved study was conducted with a group of 28 novice students to evaluate the efficacy of the IMR technology. The participants provided feedback by answering demographics, NASA-TLX and system usability scale questionnaires. Significance: Our software is a step towards improving VR based education content development process. Conclusion: The studies conducted here provide preliminary evidence that the IMR software is a usable technology based on the NASA-TLX and system usability studies conducted. Future work will compare our new educational strategy for medical training with live simulation scenarios inside a hospital room and a simple video-based curriculum.

Endotracheal intubation is a procedure in which a tube is passed through the mouth into the trachea (windpipe) to maintain an airway and provide artificial respiration. This lifesaving procedure, utilized in many clinical situations, requires complex psychomotor skills. Healthcare providers need significant training and experience to acquire skills necessary for a quick and atraumatic endotracheal intubation to prevent complications. However, medical professionals have limited training platforms and opportunities to be trained on this procedure. In this poster, we present a virtual reality-based simulation trainer for intubation training. This VR based intubation trainer provides an environment for healthcare professionals to assimilate these complex psychomotor skills while also allowing a safe place to practice swift and atraumatic intubation. User survey results are presented demonstrating that VR is a promising platform to train medical professionals effectively for this procedure.

A tetrahedral mesh is used to perform finite element analysis (FEA) of surgical cuts in a medical simulator. Visualization of a tetrahedral mesh is an important challenge in the process of constructing a realistic simulator. As game engines become increasingly popular, many companies are beginning to develop their own engines, or using existing engines for software development. However, game engines such as Unity and Unreal do not accept volume mesh. What is more, during the cutting process, the topology of tetrahedral mesh will change, resulting in progressively difficult visualization. In this paper, we present a procedure to prepare a tetrahedral mesh for a surgery simulator. We will also show an example of cutting a sphere by deleting tetrahedral elements.

This paper demonstrates targeted cyber-physical attacks on teleoperated surgical robots. These attacks exploit vulnerabilities in the robot's control system to infer a critical time during surgery to drive injection of malicious control commands to the robot. We show that these attacks can evade the safety checks of the robot, lead to catastrophic consequences in the physical system (e.g., sudden jumps of robotic arms or system's transition to an unwanted halt state), and cause patient injury, robot damage, or system unavailability in the middle of a surgery. We present a model-based analysis framework that can estimate the consequences of control commands through real-time computation of robot's dynamics. Our experiments on the RAVEN II robot demonstrate that this framework can detect and mitigate the malicious commands before they manifest in the physical system with an average accuracy of 90%.

This paper presents a simulation-based safety training simulator for robot assisted surgery. While adverse events occur rarely during training, they could be fatal to the patients if they happen during real surgical procedures and are not handled properly by the surgical team. In this work we propose a hardware-in-the-loop robotic surgery simulator with high fidelity of the robot motion in a simulated environment, which is capable of reproducing adverse events during surgery. The proposed simulator is built upon the Raven-II open source surgical robot, integrated with a simulated surgeon console and a safety hazard injection engine, which automatically injects faults into modules of the robot control software. We simulate representative safety hazards seen in the adverse events, related to da Vinci™ robot, reported to the FDA MAUDE database. A novel haptic feedback strategy is provided to the operator when the underlying dynamics differ from the real robot states.

Summary form only given. In the video, experiments demonstrating the control of robot using a brain computer interface are shown. This is done using “actemes”. Actemes are fundamental units of action. The actemes can be combined to perform complex tasks. The user who performs the tasks imagines the complex actions as an ordered combination of actemes. The EEG of the user during the process is collected and classified in real-time using independent component analysis (ICA)[1, 2] and a neural network classifier[3, 4] to identify the actemes. This ordered set of actemes can be considered to be meaningful sentence in a language. The order in which the complex action is constructed is called the grammar. Two such tasks are demonstrated using these experiments. The first task involves insertion of round peg into a hole and rotating the peg inorder to simulate the insertion of a screw into threaded hole. The actemes involved are “insert” and “rotate”. The second task is a point-to-point movement of a peg in which the peg is to be moved from a circular zone at the bottom right to the circular zone centered at the cross hair at the top left of the screen. The representative image is shown in Fig 1.

In this work we propose a new Prediction from Expert Demonstration (PED) methodology to provide haptic assistance in robot assisted trocar surgery. Data was collected from expert (clinician) demonstrations for the procedure of trocar insertion. We encode a set of force, torque and penetration trajectories by using a Gaussian Mixture Model (GMM). A generalization of these profiles and associated parameters are retrieved by Gaussian Mixture Regression (GMR). A haptic assistance mode was devised to help novices perform the procedure based on the proposed PED model. We validated the methodology for surgical assistance on (n= 15) participants. The PED haptic model was tested for instrument deviation, penetration force and penetration depth. Preliminary study results showed that participants with PED haptic assistance performed the task with more consistency and exerted lesser penetration force than subjects without assistance.

In this paper, a shape decomposition methodology to decompose the cervical spine using an approximate convex methodology is proposed. The proposed methodology identifies the most concave L-ring neighborhood in a decomposition of a surface (manifold-2) and partitions it in order to reduce its concavity. A close approximation of the original surface by a set of convex surfaces hulls is then produced.

The generation of handwriting is a complex task. In order to accommodate for the large variations involved in handwritten words deformable templates need to be used. In this paper we propose a handwriting model, based on Active shape modeling (ASM). In a two-step generation process, a template-based ASM generates characters and a Gaussian mixture regression (GMR) model concatenates the generated characters. For real time generation of cursive handwriting an adaptation of Global local approximation (GLA) methodology is used to fit the generated models.

Computer based training has become an increasingly attractive alternative to traditional training methods for skill acquisition, and the topic of skill modeling has become one of great interest. One of the key problems in computer-based training is automatic skill evaluation, which requires precise and accurate skill modeling. In order to evaluate and model human skills it is necessary to identify a relevant set of attributes by which skill can be measured, interpreted and evaluated by computers. In this paper we present our work on attributed skill transfer and focus on the specific task of writing. Identifying the central attributes associated with writing and analyzing the consistencies of these attributes within and among subjects determines whether one unique expert model can be derived from a pool of experts. In our study key attributes of hand writing were identified and statistical analysis on subject data was conducted. In our analysis we found experts' behavior can be modeled using the parameters and the same could be used to distinguish between experts and novices, lending itself as an evaluation tool.

A necessary step to building effective writing skill training system requires developing good methods for modeling human skill adequately. A number of groups have represented character information in various languages for writing skill based on trajectory information. These methods are often data intensive, tedious to implement and do not encode force information. To overcome these restrictions a novel present a novel modeling methodology that has good accuracy, robustness, flexibility and encodes force information involved in writing characters. This modeling methodology, based on Global-Local Approximation technique, has the capability to provide temporal force or position information independent of time, decoupling velocity information of the sample data used to capture skilled tasks. This modeling approach can be extended many human skilled tasks such as surgery, art and sports.

In this paper we present a concept of a versatile 'haptics' based robotic path planning and operator training tool. Some of the earlier work in the area of robotic path planning has been visited. A suitable force model has been developed and implemented to test the method. As a first step, a system for the training of machine operators and path planners is developed. Experiments are carried out to prove the hypothesis. Preliminary results from these experiments have been presented.

Traditional steps in a discrete event manufacturing simulation are to prepare input variables, select simulation parameters, run the simulation and review the results after the execution is completed. The main drawback in such systems is that complex "what if scenario" analysis would require several simulation cycles before they divulge interesting or valuable information. In an approach proposed here, we have utilized computational steering concepts along with an interactive virtual environment to obtain results of a manufacturing simulation on the fly thus allowing quick changes and instant feedback from the system. A simple conceptual system was implemented to study this approach. Initial results have provided several interesting observations.

An interactive virtual factory testbed was developed as a part of a research to explore the applications of virtual environments in the area of manufacturing automation. This fully immersive factory built with special attention to details consists of a set of modular machines that a designer can drag and place in the factory to study issues such as plant layout, clusters and part flow analysis. The paper further discusses several cell formation and visualization techniques developed especially for the design of the virtual factory. A modified Boolean matrix approach algorithm was improved as an example of applying traditional factory design methods in developing virtual factories of the future. Finally a novel interactive part-machine matrix visualization interface was developed to provide a more intuitive view of the mathematical results.

An interactive virtual factory testbed was developed as a part of a research to explore the applications of virtual environments (VE) in the area of manufacturing automation. This fully immersive factory built with special attention to details consists of a set of modular machines that a designer can drag and place in the factory to study issues such as plant layout, clusters and part flow analysis. This paper further discusses several computational issues of developing such factories including the scene graph structures, frame based intersection detection and visual realism for creating a reliable virtual interface. Finally, a single cluster analysis algorithm is implemented as an example of applying traditional factory design methods in developing virtual factories of the future.

We propose a model of attribute laden virtual tools to aid manipulation, interaction and navigation in a virtual manufacturing environment for controlling robots and machines. Fundamental to the concept of using such virtual tools, and other object-oriented virtual entities of this type generally (e.g. virtual obstacles, virtual workpieces, virtual machines etc.), is that each virtual entity represents both the graphical representation, and other physical plus "intellectual" attributes of the real entity. The aim of the virtual tool concept is to create the next generation of interface which will solve problems associated with multiple hierarchies of menu, information and other tasks such as interaction, and manipulation associated with cyberspace.

This paper describes an interactive system for specifying robotic tasks using virtual tools which allows an operator to reach into a live video scene and direct robots to use corresponding real tools in complex scenarios that involve integrating a variety of otherwise autonomous technologies. This attribute rich virtual tools concept provides a human-machine interface that is robust to unanticipated developments and tunable to the specific requirements of a particular task. This interactive specification concept is applied to intermediate manufacturing processes.