We present an interactive and multimodal-based augmented reality system for computer-assisted surgery in the context of ear, nose and throat (ENT) treatment. The proposed processing pipeline uses fully digital stereoscopic imaging devices, which support multispectral and white light imaging to generate high resolution image data, and consists of five modules. Input/output data handling, a hybrid multimodal image analysis and a bi-directional interactive augmented reality (AR) and mixed reality (MR) interface for local and remote surgical assistance are of high relevance for the complete framework. The hybrid multimodal 3D scene analysis module uses different wavelengths to classify tissue structures and combines this spectral data with metric 3D information. Additionally, we propose a zoom-independent intraoperative tool for virtual ossicular prosthesis insertion (e.g. stapedectomy) guaranteeing very high metric accuracy in sub-millimeter range (1/10 mm). A bi-directional interactive AR/MR communication module guarantees low latency, while consisting surgical information and avoiding informational overload. Display agnostic AR/MR visualization can show our analyzed data synchronized inside the digital binocular, the 3D display or any connected head-mounted-display (HMD). In addition, the analyzed data can be enriched with annotations by involving external clinical experts using AR/MR and furthermore an accurate registration of preoperative data. The benefits of such a collaborative surgical system are manifold and will lead to a highly improved patient outcome through an easier tissue classification and reduced surgery risk.

Location tracking systems have become popular in recent years and have been in use in many areas such as logistics, transportation and medicine. In healthcare, various companies have been offering real time location system solutions to hospitals with great success. These systems are used for asset management, workflow improvement as well as patient and staff tracking. For test runs we decided on a low-cost location tracking system based on proximity detection and the technology Bluetooth Low Energy. The system was developed for the use in OR environments which comes with specific requirements. The goal of this work is to evaluate if people with Bluetooth Low Energy tags are reliably tracked and detected. These tests were conducted in the first stages of a development phase to see, how the tracking system behaved in a controlled office environment. Problems such as the inactivity of the tag and random changes into other nearby rooms because of multipath effects arise and are discussed.

The striving for cognitive operating rooms (OR) and their optimization towards patient safety and room occupancy raises the importance of detailed assessments of various data, which is available in the OR. In order to model surgical workflows it is necessary to acquire the actions of the surgical staff and the data from peripheral devices, surgical lights, operating tables as well as further information present in the OR. In this work, we present a centralized framework, which allows on the one hand the assessment and visualization of various device- and sensor information in real-time and on the other hand additional human workflow annotations in a collaborative manner during the operation. The network based system is fully integrated in a clinical OR and able to record data of the most principle devices during laparoscopic surgeries. Additional information assessed by human observers can be annotated easily via a web-based frontend in real-time on any handheld device like tablets or laptops. Due to its modular design the system is under constant development to expand its scope regarding further devices, sensors and manual annotation possibilities.

Recent Internet of Things (IoT) research has been aiming at interoperability of devices and the integration of sensor networks. The Future Internet - Private Public Partnership (FI-PPP) has created a whole array of different purpose-oriented modules with defined specifications, better known as Generic Enablers. This article gives an overview of legal, ethical and technical norms and standards to be considered when planning, developing and implementing modular medical architectures, integrating the Internet of Things (IoT) and Generic Enablers (GEs) in cutting edge, latest generation medical data networks.