Multi-class semantic segmentation remains a corner-stone challenge in computer vision. Yet, dataset creation remains excessively demanding in time and effort, especially for specialized domains. Active Learning (AL) mit-igates this challenge by selecting data points for annotation strategically. However, existing patch-based AL methods often overlook boundary pixels' critical information, essential for accurate segmentation. We present OREAL, a novel patch-based AL method designed for multi-class semantic segmentation. OREAL enhances boundary detection by employing maximum aggregation of pixel-wise uncertainty scores. Additionally, we introduce one-vs-rest entropy, a novel uncertainty score function that computes class-wise uncertainties while achieving implicit class balancing during dataset creation. Comprehensive experiments across di-verse datasets and model architectures validate our hypothesis.

Semantic segmentation is a complex task that relies heavily on large amounts of annotated image data. However, annotating such data can be time-consuming and resource-intensive, especially in the medical domain. Active Learning (AL) is a popular approach that can help to reduce this burden by iteratively selecting images for annotation to improve the model performance. In the case of video data, it is important to consider the model uncertainty and the temporal nature of the sequences when selecting images for annotation. This work proposes a novel AL strategy for surgery video segmentation, COWAL, COrrelationaWare Active Learning. Our approach involves projecting images into a latent space that has been fine-tuned using contrastive learning and then selecting a fixed number of representative images from local clusters of video frames. We demonstrate the effectiveness of this approach on two video datasets of surgical instruments and three real-world video datasets. The datasets and code will be made publicly available upon receiving necessary approvals.

Automatic recognition of surgical workflow is a growing area of interest with significant potential to become part of context-aware decision-support systems in future enhanced ORs and clinical suites. Applications range from post-operative analysis to intra-operative monitoring to providing automated assistance to the clinical staff. This work proposes, for the first time, automatic tool presence and episodes recognition in bronchoscopy images, using a newly annotated video dataset obtained from robotic bronchoscopy procedures. A novel multi-task architecture that utilizes Vision Transformers for the episode recognition task is used to achieve improved accuracy while reducing the computational burden during the training stage. The method is thoroughly validated on clinical procedure video data obtained across 135 procedures, displaying improved performance and training times compared to various state-of-the-art methods.

Increasing demands in endovascular intervention have motivated technical skill training and competency-based measures of performance. However, there are no well-established online metrics for technical skill assessment; few studies have explored operator behavioral patterns from catheter motion and operator hand motions. This paper proposes a platform for active online training and objective assessment of endovascular skills, through learning optimum catheter motions from multiple demonstrations. An ungrounded hand-held haptic device for providing intuitive haptic guidance to novice users based on this learnt information is also proposed. Statistical models are implemented to extract the underlying catheter motion patterns, and utilize them for performance evaluation and haptic guidance. The results show significant improvements in endovascular navigation for inexperienced operators. Finer catheter motions were achieved with the provided haptic guidance. The results suggest that the proposed platform can be integrated into current clinical training setups, and motivate the improvement of endovascular training platforms with better realism.

Whilst current minimally invasive surgical robots offer many advantages to the surgeon, most of them are still controlled using the traditional master-slave approach, without fully exploiting the complementary strengths of both the human user and the robot. This paper proposes a framework that provides a cooperative control approach to human-robot interaction. Typical teleoperation is enhanced by incorporating haptic guidance-based feedback for surgical tasks, which are demonstrated to and learned by the robot. Safety in the surgical scene is maintained during reproduction of the learned tasks by including the surgeon in the guided execution of the learned task at all times. Continuous Hidden Markov Models are used for task learning, real-time learned task recognition and generating setpoint trajectories for haptic guidance. Two different surgical training tasks were demonstrated and encoded by the system, and the framework was evaluated using the Raven II surgical robot research platform. The results indicate an improvement in user task performance with the haptic guidance in comparison to unguided teleoperation.

Despite the increased use of robotic catheter navigation systems, and the growing interest in surgical skill evaluation in the field of endovascular intervention, there is a lack of objective and quantitative metrics for performance evaluation. So far very little research has studied operator behavioral patterns using catheter kinematics, operator forces and motions, and catheter-tissue interactions. This paper proposes a framework for automated and objective assessment of performance by measuring catheter-tissue contact forces and operator motion patterns across different skill levels, and using language models to learn the underlying force and motion patterns that are characteristic of skill. Discrete HMMs are utilized to model operator behavior for varying skill levels performing different catheterization tasks, resulting in cross-validation classification accuracies of 94% (expert) and 98% (novice) using the force-based skill models, as well as 83% (expert) and 94% (novice) using the motion-based models. The results motivate the design of improved metrics for endovascular skill assessment with further applications towards performance evaluation of robot-assisted endovascular catheterization.

This paper presents a hand-held microsurgical forceps design with force-feedback capabilities designed for micromanipulation tasks. The device uses a customized force sensor that measures grasping forces over a range of 0-300mN and uses an actuator to exert amplified forces back on to the operator's fingertip in a mechanically-ungrounded setup. This allows perception of low force levels that are otherwise imperceptible to human touch. A customized force sensor design for the forceps grasping measurement is presented and a calibration experiment was conducted to validate its linearity and repeatability. A bench test of the device was conducted to demonstrate its intrinsic force-amplifying capabilities, with amplification factors of up to ×50 reported. A user study was conducted to confirm that the device could significantly improve human perception of grasping forces compared to conventional microsurgical forceps with the results demonstrating an order-of-magnitude improvement in force perception.

Robotically assisted catheterisation has attracted significant interest in recent years. However, few designs have made full use of the extensive experience and skills that interventional radiologists have acquired from conventional catheter navigation. Additionally, limited research has been conducted in quantifying the effectiveness of haptic feedback for catheterisation procedures. This paper presents a novel master-slave force feedback system for endovascular catheterisation that can be used in a natural setting with enhanced ergonomics. The system has been validated with detailed laboratory experiments and a comprehensive user study. The performance of the system was compared against manual catheterisation by measuring the forces exerted on a vascular phantom. The results showed a 76% reduction in the mean force applied and a 55% reduction in the maximum force over the course of the study, indicating that the force feedback system reduced the magnitude and duration of force exerted during a simulated endovascular procedure. This research provides important insights into the design of compact and ergonomic robotic catheter manipulators incorporating effective real-time force feedback for intraoperative navigation.