The authors regret that an affiliation was omitted for author Dehlela Shabir. The correct affiliation details are as follows: Dehlela Shabir a,b,1 a Department of Surgery, Hamad Medical Corporation, Doha, Qatar b Computer Science and Engineering Department, Qatar University, Doha, Qatar. The authors would like to apologise for any inconvenience caused.

Surgical skills proficiency lowers the incidence of adverse clinical outcomes during surgeries. Artificial intelligence (AI) has been applied for surgical skills assessment, especially in the field of minimally invasive surgeries (MIS). This paves the way for integrating AI for skills assessment in open surgeries as well. An overview of its applications can inform the scientific community and facilitate further developments. In this scoping review , we present the open surgeries and clinical settings where AI-based skill assessment has been applied, the kind of surgical data acquired for the AI-based algorithms, and the types of AI-based models used for automated skills assessment. A total of 40 articles were identified and included. Majority of the articles focused on macrosurgical suturing (45 %, n = 18). Most of the studies acquired data by capturing surgeon's hands (50 %, n = 20). About 35 % utilized deep learning algorithms , specifically convolutional neural networks (CNN) ( n = 14). The assessment input for the automation algorithms were predominantly hand movement. Around 37.5 % ( n = 15) of the studies assessed algorithm performance using classification accuracy . In the review, we compare conventional methods such as statistical modeling and custom algorithms with the emerging AI-based approaches. We also explore the utilization of object detection and temporal information for surgical skills assessment. We highlight the progress in automated skills assessment during open surgery with advancements in sensor technology, and AI algorithms with high prediction accuracies. Further developments in data acquisition and processing methods are essential to facilitate clinical implementation of such technologies.

Robotic scope assistant systems allow surgeons to adjust the operative field view during surgery by robotically maneuvering laparoscopes. A Human-Robot Interface (HRI) is used for issuing commands to these systems, with an interaction mode mapping these commands to laparoscope movements. Optimizing the HRI and interaction mode can streamline laparoscope positioning as well as reduce cognitive workload, helping the surgeon focus on the surgical procedure. Comparing and assessing various HRIs and interaction modes is essential for efficient laparoscope maneuvering. This study evaluates HRIs based on head-motion, eye-motion, hand-motion, and voice-input operating under three interaction modes (namely: discrete, continuous, and threshold). The participants performed a user study comparing different HRIs under two simulated surgical scenarios (one in a real environment and the other in a virtual environment). The results indicated that head and eye-based HRIs performed well in continuous interaction mode, while the voice-based interface suffered from a delay. Conversely, hand-based HRIs demonstrated superior performance in both scenarios across all evaluation parameters. The study provides a benchmark for the comparison of different HRIs and provides insights into the effectiveness, limitations, and potential advantages of different HRIs.

We present an analytic solution to the 3D Dubins path problem for paths composed of an initial circular arc, a straight component, and a final circular arc. These are commonly called CSC paths. By modeling the start and goal configurations of the path as the base frame and final frame of an RRPRR manipulator, we treat this as an inverse kinematics problem. The kinematic features of the 3D Dubins path are built into the constraints of our manipulator model. Furthermore, we show that the number of solutions is not constant, with up to seven valid CSC path solutions even in non-singular regions. An implementation of solution is available at https: //github.com/aabecker/dubins3D.

The work assesses the efficacy of computer based remote tele-mentoring system (i.e. when the mentor and mentee are physically separated) for teaching minimally invasive surgical skills. The visual cues used for tele-mentoring comprises real-time virtual surgical instruments' motion augmented onto the operative field and remotely controlled by the mentor. In the feasibility study, the surgical task of laparoscopic intracorporeal suturing was simulated among 18 mentor-mentee pairs. Three modes of mentoring were used. Mode-I included traditional learning using pre-recorded videos (in absence of a mentor). Mode-II used traditional in-person hands-on mentoring. In Mode-III, a tele-mentoring prototype was used that connected a mentee with a remote mentor. Error count and duration were recorded for a learning stage followed by a testing stage for the three modes. The results show the error count for Mode-III reduces significantly as compared to Mode-I in the learning stage. Similarly, the error count for Mode-III also reduces significantly as compared to Mode-I in the testing stage. The errors count for Mode-III were equivalent to that of Mode-II for both learning and teaching stages. Furthermore, in Mode-III the duration reduces from learning to testing stage exhibiting the learning effect. Thus, computer based remote tele-mentoring is effective and more convenient to demonstrate surgical sub-steps consisting of tool-tissue interaction facilitating surgical skill transfer.

Telemedicine applications have recently received substantial potential and interest, especially after the COVID-19 pandemic. Remote experience will help people get their complex surgery done or transfer knowledge to local surgeons, without the need to travel abroad. Even with breakthrough improvements in internet speeds, the delay in video streaming is still a hurdle in telemedicine applications. This imposes using image compression and region of interest (ROI) techniques to reduce the data size and transmission needs. This paper proposes a Deep Reinforcement Learning (DRL) model that intelligently adapts the ROI size and non-ROI quality depending on the estimated throughput. The delay and structural similarity index measure (SSIM) comparison are used to assess the DRL model. The comparison findings and the practical application reveal that DRL is capable of reducing the delay by 13% and keeping the overall quality in an acceptable range. Since the latency has been significantly reduced, these findings are a valuable enhancement to telemedicine applications.

5G-and-beyond networks are designed to fulfill the communication and computation requirements of various industries, which requires not only transporting the data, but also processing them to meet/address diverse key performance indicators (KPIs). Network Function Virtualization (NFV) has emerged to enable this vision by: (i) collecting the requirements of diverse services, using graphs of Virtual Network Functions (VNFs); and (ii) mapping these requirements into network management decisions. Because of the latter, we need to efficiently allocate computing and network resources to support the desired services, and because of the former such decisions must be jointly optimized considering all KPIs associated with supported services. Thus, this paper proposes an optimized, intelligent network slicing framework to maintain a high performance of network operation by supporting diverse and heterogeneous services, while meeting new KPIs, e.g., reliability, energy consumption, and data quality. Different from the existing works, which are mainly designed considering traditional metrics like throughput and latency, we present a novel methodology and resource allocation schemes that enable high-quality selection of radio points of access, VNF placement and data routing, as well as data compression ratios, from the end users to the cloud. Our results depict the efficiency of the proposed framework in enhancing the network performance when compared to baseline approaches that consider partial network view or fair resource allocation.

In a telementoring setup for open surgeries, the motion of virtual surgical instruments is superimposed onto the operative field. This assists a mentor to effectively convey to the mentee the information pertaining to the required tool-tissue interactions during the surgery. The aim of this work is to assess the effects of using a virtual reality environment at mentor's site (in contrast to conventional visualization on a two-dimensional screen) during surgical telementoring. A user study is conducted simulating motion of virtual surgical instrument in an open surgery. The results show that mentor is able to demonstrate with higher accuracy and in shorter duration, the required virtual surgical instrument motions. Thus, rendering information to the mentor in an immersive virtual reality environment assists in better understanding of the operative field and enhanced control of the virtual surgical instrument motion. This further aids in conveying accurate information pertaining to the tool-tissue interaction to the mentee.

The advent of intraoperative real-time image guidance has led to the emergence of new surgical interventional paradigms including image-guided robot assistance. Most often the use of an intraoperative imaging modality is limited to visual perception of the area of procedure. In this work, we propose a system for performing interventions with real-time Magnetic Resonance Imaging (rtMRI). The described computational core, processes on-the-fly rtMRI and generates dynamic guidance virtual fixture that in turn is used to update visualization and a force-feedback interface. The system was experimentally tested by applying it to a simulated Transapical Aortic Valve Implantation with a virtual robotic manipulator. The study results demonstrate significant improvement in the surgical task by decreasing the duration of the procedure and increasing safety in the presence of cardiac and breathing motion.

Telementoring in surgery facilitates the transfer of surgical knowledge from the mentor to the mentee. Augmented Reality (AR) further assists this transfer by overlaying visual cues (e.g., in the form of virtual surgical instrument motion) generated by the mentor onto the operative field of the mentee. In this work, we present a benchmark for comparing such AR based surgical telementoring systems. The results compare the network performances of these systems across different types of surgery (open or minimally invasive), based on the locations of the mentor and the mentee (inter- or intra- country), and finally the underlying networking protocols (RTMP versus WebRTC).

The work presents a practical Augmented Reality (AR) based surgical navigation system using optical see-through head-mounted display as a standalone solution, without the need of additional tracking hardware. Specifically, we propose a fiducial marker-based instrument tracking, which entirely relies on the built-in hardware of the Microsoft HoloLens 2. The tracking algorithm computes the pose of the tracked object from the real-time image obtained from the on-board front-facing RGB camera. The estimated transformation is then transmitted back to the HoloLens for visualization. Our experimental evaluation shows that the system can achieve 0.81 mm / 1.52 degree in tracking accuracy and sub-millimeter alignment accuracy.

This work presents an online trajectory generation algorithm using a sinusoidal jerk profile. The generator takes initial acceleration, velocity and position as input, and plans a multi-segment trajectory to a goal position under jerk, acceleration, and velocity limits. By analyzing the critical constraints and conditions, the corresponding closed-form solution for the time factors and trajectory profiles are derived. The proposed algorithm was first derived in Mathematica and then converted into a C++ implementation. Finally, the algorithm was utilized and demonstrated in ROS & Gazebo using a UR3 robot. Both the Mathematica and C++ implementations can be accessed at https://github.com/Haoran-Zhao/Jerk-continuous-online-trajectory-generator-with-constraints.git

Telemedicine is a rising technology that is gaining a lot of interest in the recent decades. Several applications of telemedicine are delay-sensitive and need to be operated in real-time. One of which is surgical tele-mentoring where a remote expert surgeon mentors local surgeons during an operation. While the advances done in telecommunications and robotics have made tele-mentoring possible in modern days, there are still many challenges that stop such telemedical applications from being completely legalized and approved as medical tools across the world. One of the main issues is the need for very high bandwidth to allow the surgery to be done accurately in real-time. Such bandwidth requirements are difficult to provide especially in rural areas with limited communications infrastructure. We propose an adaptive Region of Interest (ROI) detection and an optimization model that addresses the trade-off between the overall quality of a surgical video and the network delay. The model aims to maximize the size of the ROI, where highest video quality must be used, depending on the available network throughput, while avoiding excessive degradation of the quality of the background.

This work presents an interventional planning software to be used in conjunction with a robotic manipulator to perform transrectal MR guided prostate biopsies. The interventional software was designed taking in consideration a generic manipulator used under the two modes of operation: side-firing and end-firing of the biopsy needle. Studies were conducted with urologists using the software to plan virtual biopsies. The results show features of software relevant for operating efficiently under the two modes of operation.

This work presents a novel deep learning architecture called BNU-Net for the purpose of cardiac segmentation based on short-axis MRI images. Its name is derived from the Batch Normalized (BN) U-Net architecture for medical image segmentation. New generations of deep neural networks (NN) are called convolutional NN (CNN). CNNs like U-Net have been widely used for image classification tasks. CNNs are supervised training models which are trained to learn hierarchies of features automatically and robustly perform classification. Our architecture consists of an encoding path for feature extraction and a decoding path that enables precise localization. We compare this approach with a parallel approach named U-Net. Both BNU-Net and U-Net are cardiac segmentation approaches: while BNU-Net employs batch normalization to the results of each convolutional layer and applies an exponential linear unit (ELU) approach that operates as activation function, U-Net does not apply batch normalization and is based on Rectified Linear Units (ReLU). The presented work (i) facilitates various image preprocessing techniques, which includes affine transformations and elastic deformations, and (ii) segments the preprocessed images using the new deep learning architecture. We evaluate our approach on a dataset containing 805 MRI images from 45 patients. The experimental results reveal that our approach accomplishes comparable or better performance than other state-of-the-art approaches in terms of the Dice coefficient and the average perpendicular distance.

Prostate biopsy is considered as a definitive way for diagnosing prostate malignancies. Urologists are currently moving towards MR-guided prostate biopsies over conventional transrectal ultrasound-guided biopsies for prostate cancer detection. Recently, robotic systems have started to emerge as an assistance tool for urologists to perform MR-guided prostate biopsies. However, these robotic assistance systems are designed for a specific clinical environment and cannot be adapted to modifications or changes applied to the clinical setting and/or workflow. This work presents the preliminary design of a cable-driven manipulator developed to be used in both MR scanners and MR-ultrasound fusion systems. The proposed manipulator design and functionality are evaluated on a simulated virtual environment. The simulation is created on an in-house developed interventional planning software to evaluate the ergonomics and usability. The results show that urologists can benefit from the proposed design of the manipulator and planning software to accurately perform biopsies of targeted areas in the prostate.

This work presents a prototype holographic interface (HI) for the 3D visualization of MRI data for the purpose of planning neurosurgical procedures. The presented HI (i) immerses the operator to a mixed reality (MiR) scene, which includes MRI data and virtual renderings, (ii) facilitates interactive manipulation of the objects of the MiR scene for planning and (iii) is the front-end of a pipeline that links the operator to the MRI scanner for on-the-fly control of the scanner. Preliminary qualitative evaluation revealed that holographic visualization of high-resolution 3D MRI data offers an intuitive and interactive perspective of the complex brain vasculature and anatomical structures. These early work further suggests that immersive experience may be an unparalleled tool in better planning neurosurgical procedures. Further development is required to speed up the pipeline from the MRI scanner to the HI and incorporating means of manipulations other than gestures.

Needle-based image-guided interventions (NB-IGI) are well established practices that are rapidly expanding to a wider range of therapeutic and diagnostic interventions due to their impact on clinical outcomes. In parallel, a number of robotic manipulators are emerging to increase accuracy, decrease inter-operator variabilities, and reduce the duration of the procedure. Most current systems are application-specific, thereby are appropriate for a limited number of procedures and further exacerbating the already rising costs of healthcare. In order to expand clinical usage of robotic NB-IGI, and eventually reduce the burden on healthcare costs, this paper proposes a modular, customizable robotic systems concept. The concept includes simplified hardware and control techniques, and initial results demonstrate that accuracy is within clinically acceptable limits. Preliminary designs, implementations, and assessments of the proposed system demonstrate its potential and clinical value.

Reconstruction of the anterior cruciate ligament (ACL) through arthroscopy is one of the most common procedures in orthopaedics. It requires accurate alignment and drilling of the tibial and femoral tunnels through which the ligament graft is attached. Although commercial computer-assisted navigation systems exist to guide the placement of these tunnels, most of them are limited to a fixed pose without due consideration of dynamic factors involved in different knee flexion angles. This paper presents a new model for intraoperative guidance of arthroscopic ACL reconstruction with reduced error particularly in the ligament attachment area. The method uses 3D preoperative data at different flexion angles to build a subject-specific statistical model of knee pose. To circumvent the problem of limited training samples and ensure physically meaningful pose instantiation, homogeneous transformations between different poses and local-deformation finite element modelling are used to enlarge the training set. Subsequently, an anatomical geodesic flexion analysis is performed to extract the subject-specific flexion characteristics. The advantages of the method were also tested by detailed comparison to standard Principal Component Analysis (PCA), nonlinear PCA without training set enlargement, and other state-of-the-art articulated joint modelling methods. The method yielded sub-millimetre accuracy, demonstrating its potential clinical value.

Efficient and intuitive interfacing of the interventionalist to the information and tools available from image-guided robotic assisted surgeries is required to achieve the full benefit of these technologies. Ongoing research has been performed into the use of forbidden region guided fixtures (FRVF) for human-in-the-loop control of image-guided procedures via haptic force-feedback devices (FFD). Although commercially available FFD provide sufficient degrees-of-freedom (DoF), collaborating clinicians, as well as the results of our previous work indicate that these systems are not completely intuitive for controlling fixed-point access interventional tool which have a remote center of motion. Within this context, we introduce a new FFD which is designed with the same DoF constraints as a fixed-point access interventional tool. The device is tested in a clinical simulation of a robot assisted trans-apical valve implantation under guidance from real-time magnetic resonance imaging. Pre-acquired real-time images are used in the clinical simulation to dynamically update the FRVF and therefore provide guiding forces to allow the operator to see the safe boundaries of operation via a visualization interface and physically feel them through the FFD. Inertial and gravity compensation and per DoF dynamic response of the physical prototype are validated and the frequency response of the system demonstrates it is adequate for tactile sensing. During clinical simulation the operator was successfully able to maneuver the tool within the safe path to the region of interest with the guidance of visual and force-feedback.

Co-registering pre- and intra- operative MR data is an important yet challenging problem due to different acquisition parameters, resolutions, and plane orientations. Despite its importance, previous approaches are often computationally intensive and thus cannot be employed in real-time. In this paper, a novel three-step approach is proposed to dynamically register pre-operative 4D MR data with intra-operative 2D RT-MRI to guide intracardiac procedures. Specifically, a novel preparatory step, executed in the pre-operative phase, is introduced to generate bridging information that can be used to significantly speed up the on-the-fly registration in the intraoperative procedure. Our experimental results demonstrate an accuracy of 0.42 mm and a processing speed of 26 FPS of the proposed approach on an off-the-shelf PC. This approach, is in particularly developed for performing intra-cardiac procedures with real-time MR guidance.

Robot-assisted surgical procedures are perpetually evolving due to potential improvement in patient treatment and healthcare cost reduction. Integration of an imaging modality intraoperatively further strengthens these procedures by incorporating the information pertaining to the area of intervention. Such information needs to be effectively rendered to the operator as a human-in-the-loop requirement. In this work, we propose a guidance approach that uses real-time MRI to assist the operator in performing robot-assisted procedure in a beating heart. Specifically, this approach provides both real-time visualization and force-feedback based guidance for maneuvering an interventional tool safely inside the dynamic environment of a heart's left ventricle. Experimental evaluation of the functionality of this approach was tested on a simulated scenario of transapical aortic valve replacement and it demonstrated improvement in control and manipulation by providing effective and accurate assistance to the operator in real-time.

As a part of an ongoing project, in this paper we introduce the first version of a system which has a novel methodology for Cine (as in cinema) MRI based control of a cardiac robot for beating heart surgeries. The system uses the preoperative planning approach that we developed earlier, and integrates it to the intraoperative algorithms for controlling a robot and tracking some specific landmarks of a highly dynamical surgical field. In particular, our late studies presented herein aim to demonstrate the feasibility of integrating appropriate computational tools to achieve the volumetric image guidance for minimally invasive surgeries in the beating heart. We conceive of the system as practicable for in vitro experiments upon the completion of the first physical prototype, which may pave the way for expansion of the approach for other complex surgeries as well.