Reliable recognition and localization of surgical instruments in endoscopic video recordings are foundational for a wide range of applications in computer- and robot-assisted minimally invasive surgery (RAMIS), including surgical training, skill assessment, and autonomous assistance. However, robust performance under real-world conditions remains a significant challenge. Incorporating surgical context - such as the current procedural phase - has emerged as a promising strategy to improve robustness and interpretability. To address these challenges, we organized the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) sub-challenge as part of the Endoscopic Vision (EndoVis) challenge at MICCAI 2024. We introduced a novel, multi-center dataset comprising thirteen full-length laparoscopic cholecystectomy videos collected from three distinct medical institutions, with unified annotations for three interrelated tasks: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation. Unlike existing datasets, ours enables joint investigation of instrument localization and procedural context within the same data while supporting the integration of temporal information across entire procedures. We report results and findings in accordance with the BIAS guidelines for biomedical image analysis challenges. The PhaKIR sub-challenge advances the field by providing a unique benchmark for developing temporally aware, context-driven methods in RAMIS and offers a high-quality resource to support future research in surgical scene understanding.

Medical applications, such as telemedicine or smart operation rooms, place stringent requirements on the underlying network architecture. 6G as the next-generation communication standard currently in research promises to satisfy the needs of such applications by utilizing advances in technology and networking concepts. One crucial concept for medical applications is the capability of using computing resources within the network. By placing the applications on such processing nodes in different locations within the Radio Access Network (RAN), the performance metrics of a medical application, such as latency, throughput, and availability can be optimized. However, problems arise when the available processing capabilities are not sufficient for all requested medical applications. In this paper, we formulate an Integer Linear Program (ILP) to address the problem of processing medical applications within the network when the processing capabilities are not sufficient. We consider the priority and different service levels of application functions and aim to place as many applications as possible with the best possible service quality. Additionally, we take into account that some applications must run in the network even if their priority is low. Furthermore, we propose a heuristic in order to obtain a good solution quickly. The evaluation of our solution and comparison to existing approaches shows an increase of accepted demands in the network by up to 35%.

The currently researched 6G communication standard promises to enable new applications with the native integration of new technologies and concepts. Especially the medical area will greatly benefit from the developments of new communication networks. As the communication infrastructure will be able to satisfy stringent and varying requirements, emerging medical applications such as telemedicine and telesurgery will be enabled. However, the concrete networking architecture remains an open question for research. In this paper, we propose a dedicated med-ical 6G Radio Access Network (RAN) architecture and discuss possible usage scenarios based on two medical use cases. First, a network architecture is presented which leverages in-network computing to execute medical applications. The main idea of this concept is the dynamic interaction between medical applications and the network. In particular, the placement of Modular Application Functions (MAFs) and their execution depends on the state of the network. Secondly, we map two medical use cases, namely a semiautonomous telerobotic examination suite and a context-sensitive medical environment, to the proposed network architecture and explain the interaction between our network architecture and these applications in detail. Our approach demonstrates the potential of the combined development of medical applications and their underlying communication architecture.

We propose a novel method to tackle the visual-inertial localization problem for constrained camera movements. We use residuals from the different modalities to jointly optimize a global cost function. The residuals emerge from IMU measurements, stereoscopic feature points, and constraints on possible solutions in SE(3). In settings where dynamic disturbances are frequent, the residuals reduce the complexity of the problem and make localization feasible. We verify the advantages of our method in a suitable medical use case and produce a dataset capturing a minimally invasive surgery in the abdomen. Our novel clinical dataset MITI is comparable to state-of-the-art evaluation datasets, contains calibration and synchronization and is available at [1].

When two or more users attempt to collaborate in the same space with Augmented Reality, they often encounter conflicting intentions regarding the occupation of the same working area and self-positioning around such without mutual interference. Augmented Reality is a powerful tool for communicating ideas and intentions during a co-assisting task that requires multi-disciplinary expertise. To relax the constraint of physical co-location, we propose the concept of Duplicated Reality, where a digital copy of a 3D region of interest of the users' environment is reconstructed in real-time and visualized in-situ through an Augmented Reality user interface. This enables users to remotely annotate the region of interest while being co-located with others in Augmented Reality. We perform a user study to gain an in-depth understanding of the proposed method compared to an in-situ augmentation, including collaboration, effort, awareness, usability, and the quality of the task. The result indicates almost identical objective and subjective results, except a decrease in the consulting user's awareness of co-located users when using our method. The added benefit from duplicating the working area into a designated consulting area opens up new interaction paradigms to be further investigated for future co-located Augmented Reality collaboration systems.

When users create hand-drawn annotations in Virtual Reality they often reach their physical limits in terms of precision, especially if the region to be annotated is small. One intuitive solution employs magnification beyond natural scale. However, scaling the whole environment results in wrong assumptions about the coherence between physical and virtual space. In this paper, we introduce Mag-noramas, a novel interaction method for selecting and extracting a region of interest that the user can subsequently scale and transform inside the virtual space. Our technique enhances the user's capabilities to perform supernaturally precise virtual annotations on virtual objects. We explored our technique in a user study within asimplified clinical scenario of a teleconsultation-supported craniectomy procedure that requires accurate annotations on a human head. Teleconsultation was performed asymmetrically between a remote expert in Virtual Reality that collaborated with a local user through Augmented Reality. The remote expert operates inside a reconstructed environment, captured from RGB-D sensors at the local site, and is embodied by an avatar to establish co-presence. The results show that Magnoramas significantly improve the precision of annotations while preserving usability and perceived presence measures compared to the baseline method. By hiding the 3D reconstruction while keeping the Magnorama, users can intentionally choose to lower their perceived social presence and focus on their tasks.

The striving for cognitive operating rooms (OR) and their optimization towards patient safety and room occupancy raises the importance of detailed assessments of various data, which is available in the OR. In order to model surgical workflows it is necessary to acquire the actions of the surgical staff and the data from peripheral devices, surgical lights, operating tables as well as further information present in the OR. In this work, we present a centralized framework, which allows on the one hand the assessment and visualization of various device- and sensor information in real-time and on the other hand additional human workflow annotations in a collaborative manner during the operation. The network based system is fully integrated in a clinical OR and able to record data of the most principle devices during laparoscopic surgeries. Additional information assessed by human observers can be annotated easily via a web-based frontend in real-time on any handheld device like tablets or laptops. Due to its modular design the system is under constant development to expand its scope regarding further devices, sensors and manual annotation possibilities.