Enabling collaborative telepresence in healthcare, especially surgical procedures, presents a critical challenge. The decompressive craniotomy procedure stands out as particularly complex and time-sensitive. The current teleconsultation approach relies on 2D color cameras, often offering only a fixed view and limited visual capabilities between experts and surgeons. However, teleconsultation can be addressed with Mixed Reality and immersive technology to potentially enable a better consultation of the procedure. We conducted an extensive user study focusing on decompressive craniotomy to investigate the advantages and challenges of our 3D teleconsultation system compared to a 2D video-based consultation system. Our 3D teleconsultation system leverages real-time 3D reconstruction of the patient and environment to empower experts to provide guidance and create virtual 3D annotations. The study utilized 3D-printed head models to perform a lifelike surgical intervention. It involved 14 medical residents and demonstrated an in-vitro 17% improvement in accurately describing the incision size on the patient’s head, contributing to potentially improved patient outcomes.

C-arm fluoroscopes are frequently used during surgeries for intraoperative guidance. Unfortunately, due to X-ray emission and scattering, increased radiation exposure occurs in the operating theatre. The objective of this work is to sensitize the surgeon to their radiation exposure, enable them to check on their exposure over time, and to help them choose their best position related to the C-arm gantry during surgery. First, we aim at simulating the amount of radiation that reaches the surgeon using the Geant4 software, a toolkit developed by CERN. Using a flexible setup in which two RGB-D cameras are mounted to the mobile C-arm, the scene is captured and modeled respectively. After the simulation of particles with specific energies, the dose at the surgeon's position, determined by the depth cameras, can be measured. The validation was performed by comparing the simulation results to both theoretical values from the C-arms user manual and real measurements made with a QUART didoSVM dosimeter. The average error was 16.46% and 16.39%, respectively. The proposed flexible setup and high simulation precision without a calibration with measured dosimeter values, has great potential to be directly used and integrated intraoperatively for dose measurement.

The interlocking of intramedullary nails is a technically demanding procedure which involves a considerable amount of X-ray acquisitions; one study lists as many as 48 to successfully complete the procedure and fix screws into 4-6 mm distal holes of the nail. We propose to design an augmented radiolucent drill to assist surgeons in completing the distal locking procedure without any additional X-ray acquisitions. Using an augmented reality fluoroscope that coregisters optical and X-ray images, we exploit solely the optical images to detect the augmented radiolucent drill and estimate its tip position in real-time. Consequently, the surgeons will be able to maintain the down the beam positioning required to drill the screws into the nail holes successfully. To evaluate the accuracy of the proposed augmented drill, we perform a preclinical study involving six surgeons and ask them to perform distal locking on dry bone phantoms. Surgeons completed distal locking 98.3% of the time using only a single X-ray image with an average navigation time of 1.4 ± 0.9 min per hole.

We present a unique simulator-based methodology for assessing both technical and nontechnical (cognitive) skills for surgical trainees while immersed in a complete medical simulation environment. Further, we have included two crisis scenarios which allow for the evaluation of the effect of cognitive strategy selection on the low-level surgical skills. Training these mixed-mode scenarios can thereby be evaluated on our platform, allowing for improved assessment and a stronger foundation for credentialing, with the potential to reduce the occurrence of adverse events in the operating room. Scientific evaluation and validation of our work is conducted together with 19 junior surgeons in order to achieve the following goals: 1) to provide a qualitative measure of usability, 2) to assess vertebroplasty technical performance of the surgeon, and 3) to explore the relationship between mental workload and surgical performance during crisis. Our results indicate that: 1) the surgeons scored the face validity of our modeled simulation environment very highly ( 4.68 ±0.48, using a 5-point Likert scale), 2) surgeon training enabled completion of tasks more quickly, and 3) the introduction of crisis scenarios negatively affected the surgeons' objective performance. Taken together, our results underscore the need to develop realistic simulation environments that prepare young residents to respond to emergent events in the operating room.

Education of anatomy is a challenging but crucial element in educating medical professionals, but also for general education of pupils. Our research group has previously developed a prototype of an Augmented Reality (AR) magic mirror which allows intuitive visualization of realistic anatomical information on the user. However, the current overlay is imprecise as the magic mirror depends on the skeleton output from Kinect. These imprecisions affect the quality of education and learning. Hence, together with clinicians we have defined bone landmarks which users can touch easily on their body while standing in front of the sensor. We demonstrate that these landmarks allow the proper deformation of medical data within the magic mirror and onto the human body, resulting in a more precise augmentation.

For trauma and orthopedic surgery, maneuvering a mobile C-arm fluoroscope into a desired position to acquire an X-ray is a routine surgical task. The precision and ease of use of the C-arm becomes even more important for advanced interventional imaging techniques such as parallax-free X-ray image stitching. Today's standard mobile C-arms have been modeled with only five degrees of freedom (DOF), which definitely restricts their motions in 3-D Cartesian space. In this paper, we present a method to model both the mobile C-arm and patient's table as an integrated kinematic chain having six DOF without constraining table position. The closed-form solutions for the inverse kinematics problem are derived in order to obtain the required values for all C-arm joint and table movements to position the fluoroscope at a desired pose. The modeling method and the closed-form solutions can be applied to general isocentric or nonisocentric mobile C-arms. By achieving this we develop an efficient and intuitive inverse kinematics-based method for parallax-free panoramic X-ray imaging. In addition, we implement a 6-DOF C-arm system from a low-cost mobile fluoroscope to optimally acquire X-ray images based solely on the computation of the required movement for each joint by solving the inverse kinematics on a continuous basis. Through simulation experimentation, we demonstrate that the 6-DOF C-arm model has a larger working space than the 5-DOF model. C-arm repositioning experiments show the practicality and accuracy of our 6-DOF C-arm system. We also evaluate the novel parallax-free X-ray stitching method on phantom and dry bones. Using five trials, results show that parallax-free panoramas generated by our method are of high visual quality and within clinical tolerances for accurate evaluation of long bone geometry (i.e., image and metric measurement errors are less than 1% compared to ground-truth).

Port position in minimally invasive surgeries is chosen to minimize the lesion of tissue and maximize the movability for endoscopic instruments. In this study, we present an evaluation of the potential of a 3D contextual in-situ visualization of the anatomic target region to help surgeons for three different surgical procedures decide where best to create ports and incisions to enable the insertion of a specific set of instruments.