Employing a minimal array of electroencephalography (EEG) channels for neonatal sleep stage classification is essential for data acquisition in the Internet of Medical Things (IoMT), as single-channel and edge-based features can reduce data transfer and processing requirements, enhancing cost-effectiveness and practicality. In this paper, we evaluate the efficacy of a single channel and the viability of a binary classification scheme for discerning awake and sleep states and transitions to quiet sleep. For this, two datasets of EEG signals for neonate sleep analysis were recorded from Children's Hospital of Fudan University, Shanghai, comprising recordings from 64 and 19 neonates, respectively. From each epoch, a diverse ensemble of 490 features was extracted through a blend of discrete and continuous wavelet transforms (DWT, CWT), spectral statistics, and temporal features. In addition, we introduced an innovative hybrid univariate and ensemble feature selection approach with multidomain feature fusion, a stacking-based ensemble classifier that outperforms existing work. We achieved 90.37%, 91.13%, and 94.88% accuracy for sleep/awake, quiet sleep/non-quiet sleep, and quiet sleep/awake, respectively. This was corroborated by significant Kappa values of 77.5%, 80.29%, and 89.76%. Using SelectPercentile, we devised three distinct feature selection mechanisms: one using DWT, one with CWT, and another incorporating both spectral and temporal features. Subsequently, SelectKBest was used to determine the most effective features. For our stacked model, we incorporated a trifecta of the ExtraTree model with variable estimators, a Random Forest, and an Artificial Neural Network (ANN) as base classifiers, and for the final prediction phase, ANN was implemented again. The model's performance was evaluated using K-fold and leave-one-subject cross-validation.

Open Spina Bifida (OSB) is a congenital neural tube defect that affects approximately 1 in 1000 births worldwide. Robotic in-utero OSB repair provides a minimally invasive alternative to open-surgery, which places significant strain on both baby and mother. Recent advancements in da Vinci miniature continuum tools reduce port sizes through the uterus for access to the fetus with lower maternal risk. However, idiosyncrasies in continuum tool behaviour further complicate an already difficult procedure. Consequently, a high-fidelity da Vinci OSB repair simulator is presented featuring continuum tools for surgeon skills training. The simulator incorporates a plugin for suture physics handling, soft body physics for deformable tissues and implements haptic virtual fixtures for improved situational awareness during suturing. Quantitative validation demonstrated virtual tool accuracy, with a mean-squared continuum backbone error of 0.64 mm2and system-level end-effector trajectory errors averaging 3.25 mm for a helix tracing task. During suturing, high-fidelity performance was maintained. Four expert surgeons from relevant specialties provided positive qualitative feedback, reporting that the simulator accurately replicates real tool control and offers a realistic and valuable training experience. Ultimately, the simulator shows promise as a training platform for safer robotic in-utero OSB repair and facilitating the adoption of novel continuum wristed tools in clinical settings.

Automation in surgical robotics has the potential to improve patient safety and surgical efficiency, but it is difficult to achieve due to the need for robust perception algorithms. In particular, 6D pose estimation of surgical instruments is critical to enable the automatic execution of surgical maneuvers based on visual feedback. In recent years, supervised deep learning algorithms have shown increasingly better performance at 6D pose estimation tasks; yet, their success depends on the availability of large amounts of annotated data. In household and industrial settings, synthetic data, generated with 3D computer graphics software, has been shown as an alternative to minimize annotation costs of 6D pose datasets. However, this strategy does not translate well to surgical domains as commercial graphics software have limited tools to generate images depicting realistic instrument-tissue interactions. To address these limitations, we propose an improved simulation environment for surgical robotics that enables the automatic generation of large and diverse datasets for 6D pose estimation of surgical instruments. Among the improvements, we developed an automated data generation pipeline and an improved surgical scene. To show the applicability of our system, we generated a dataset of 7.5k images with pose annotations of a surgical needle that was used to evaluate a state-of-the-art pose estimation network. The trained model obtained a mean translational error of 2.59mm on a challenging dataset that presented varying levels of occlusion. These results highlight our pipeline's success in training and evaluating novel vision algorithms for surgical robotics applications.

Skull base surgery is a demanding field in which surgeons operate in and around the skull while avoiding critical anatomical structures including nerves and vasculature. While image-guided surgical navigation is the prevailing standard, limitation still exists requiring personalized planning and recognizing the irreplaceable role of a skilled surgeon. This paper presents a collaboratively controlled robotic system tailored for assisted drilling in skull base surgery. Our central hypothesis posits that this collaborative system, enriched with haptic assistive modes to enforce virtual fixtures, holds the potential to significantly enhance surgical safety, streamline efficiency, and alleviate the physical demands on the surgeon. The paper describes the intricate system development work required to enable these virtual fixtures through haptic assistive modes. To validate our system’s performance and effectiveness, we conducted initial feasibility experiments involving a medical student and two experienced surgeons. The experiment focused on drilling around critical structures following cortical mastoidectomy, utilizing dental stone phantom and cadaveric models. Our experimental results demonstrate that our proposed haptic feedback mechanism enhances the safety of drilling around critical structures compared to systems lacking haptic assistance. With the aid of our system, surgeons were able to safely skeletonize the critical structures without breaching any critical structure even under obstructed view of the surgical site.

Despite advancements in robotic-assisted surgery, automating complex tasks like suturing remains challenging due to the need for adaptability and precision. Learning-based approaches, particularly reinforcement learning (RL) and imitation learning (IL), require realistic simulation environments for efficient data collection. However, current platforms often include only relatively simple, non-dexterous manipulations and lack the flexibility required for effective learning and generalization. We introduce SurgicAI, a novel platform for development and benchmarking that addresses these challenges by providing the flexibility to accommodate both modular subtasks and more importantly task decomposition in RL-based surgical robotics. Compatible with the da Vinci Surgical System, SurgicAI offers a standardized pipeline for collecting and utilizing expert demonstrations. It supports the deployment of multiple RL and IL approaches, and the training of both singular and compositional subtasks in suturing scenarios, featuring high dexterity and modularization. Meanwhile, SurgicAI sets clear metrics and benchmarks for the assessment of learned policies. We implemented and evaluated multiple RL and IL algorithms on SurgicAI. Our detailed benchmark analysis underscores SurgicAI's potential to advance policy learning in surgical robotics. Details: https://github.com/surgical-robotics-ai/SurgicAI

The introduction of image-guided surgical navigation (IGSN) has greatly benefited technically demanding surgical procedures by providing real-time support and guidance to the surgeon during surgery. To develop effective IGSN, a careful selection of the surgical information and the medium to present this information to the surgeon is needed. However, this is not a trivial task due to the broad array of available options. To address this problem, we have developed an open-source library that facilitates the development of multimodal navigation systems in a wide range of surgical procedures relying on medical imaging data. To provide guidance, our system calculates the minimum distance between the surgical instrument and the anatomy and then presents this information to the user through different mechanisms. The real-time performance of our approach is achieved by calculating Signed Distance Fields at initialization from segmented anatomical volumes. Using this framework, we developed a multimodal surgical navigation system to help surgeons navigate anatomical variability in a skull base surgery simulation environment. Three different feedback modalities were explored: visual, auditory, and haptic. To evaluate the proposed system, a pilot user study was conducted in which four clinicians performed mastoidectomy procedures with and without guidance. Each condition was assessed using objective performance and subjective workload metrics. This pilot user study showed improvements in procedural safety without additional time or workload. These results demonstrate our pipeline's successful use case in the context of mastoidectomy.

Telesurgery has a clear potential for providing high-quality surgery to medically underserved areas like rural areas, battlefields, and spacecraft; nevertheless, effective methods to overcome unreliable communication systems are still lacking. Furthermore, it is not well understood how users react at the moment of communication loss and also during the loss. In this paper, we aim to analyze human response by proposing a telesurgery simulation framework that models an environment incorporating local and remote sites. Furthermore, this framework generates structural data for human behavior analysis and can provide different forms of assistance during the communication failure and at the communication recovery. We investigated three different types of assistance: User-centered, Robot-centered and Hybrid. A 12-person user-study was carried out using the proposed telesurgery simulation where participants completed a peg transfer task with random communication loss. The collected data was used to analyze the human response to a communication failure. The proposed Hybrid method reduced temporal demand with no increase in completion time compared to the baseline control method where users were unable to move the input device during the communication loss. The Hybrid method also significantly reduced both the task completion time and workload compared to the other two proposed methods (User-centered and Robot-centered).

Human-robot shared control, which integrates the advantages of both humans and robots, is an effective approach to facilitate efficient surgical operation. Learning from demonstration (LfD) techniques can be used to automate some of the surgical sub tasks for the construction of the shared control mechanism. However, a sufficient amount of data is required for the robot to learn the manoeuvres. Using a surgical simulator to collect data is a less resource-demanding approach. With sim-to-real adaptation, the manoeuvres learned from a simulator can be transferred to a physical robot. To this end, we propose a sim-to-real adaptation method to construct a human-robot shared control framework for robotic surgery. In this paper, a desired trajectory is generated from a simulator using LfD method, while dynamic motion primitives (DMP) is used to transfer the desired trajectory from the simulator to the physical robotic platform. Moreover, a role adaptation mechanism is developed such that the robot can adjust its role according to the surgical operation contexts predicted by a neural network model. The effectiveness of the proposed framework is validated on the da Vinci Research Kit (dVRK). Results of the user studies indicated that with the adaptive human-robot shared control framework, the path length of the remote controller, the total clutching number and the task completion time can be reduced significantly. The proposed method outperformed the traditional manual control via teleoperation.

We present an open-source framework that provides a low barrier to entry for real-time simulation, visualization, and interactive manipulation of user-specifiable soft-bodies, environments, and robots (using a human-readable front-end interface). The simulated soft-bodies can be interacted by a variety of input interface devices including commercially available haptic devices, game controllers, and the Master Tele-Manipulators (MTMs) of the da Vinci Research Kit (dVRK) with real-time haptic feedback. We propose this framework for carrying out multi-user training, user-studies, and improving the control strategies for manipulation problems. In this paper, we present the associated challenges to the development of such a framework and our proposed solutions. We also demonstrate the performance of this framework with examples of soft-body manipulation and interaction with various input devices.

Interactive simulators are used in several important applications which include the training simulators for teleoperated robotic laparoscopic surgery. While stateof-art simulators are capable of rendering realistic visuals and accurate dynamics, grasping is often implemented using kinematic simplification techniques that prevent truly multimanual manipulation, which is often an important requirement of the actual task. Realistic grasping and manipulation in simulation is a challenging problem due to the constraints imposed by the implementation of rigid-body dynamics and collision computation techniques in state-of-the-art physics libraries. We present a penalty based parametric approach to achieve multi-manual grasping and manipulation of complex objects at arbitrary postures in a real-time dynamic simulation. This approach is demonstrated by accomplishing multi-manual tasks modeled after realistic scenarios, which include the grasping and manipulation of a two-handed screwdriver task and the manipulation of a deformable thread.

The recent development of Robot-Assisted Minimally Invasive Surgery (RAMIS) has brought much benefit to ease the performance of complex Minimally Invasive Surgery (MIS) tasks and lead to more clinical outcomes. Compared to direct master-slave manipulation, semi-autonomous control for the surgical robot can enhance the efficiency of the operation, particularly for repetitive tasks. However, operating in a highly dynamic in-vivo environment is complex. Supervisory control functions should be included to ensure flexibility and safety during the autonomous control phase. This paper presents a haptic rendering interface to enable supervised semi-autonomous control for a surgical robot. Bayesian optimization is used to tune user-specific parameters during the surgical training process. User studies were conducted on a customized simulator for validation. Detailed comparisons are made between with and without the supervised semi-autonomous control mode in terms of the number of clutching events, task completion time, master robot end-effector trajectory and average control speed of the slave robot. The effectiveness of the Bayesian optimization is also evaluated, demonstrating that the optimized parameters can significantly improve users' performance. Results indicate that the proposed control method can reduce the operator's workload and enhance operation efficiency.

Over the past decade, Robot-Assisted Surgeries (RAS), have become more prevalent in facilitating successful operations. Of the various types of RAS, the domain of collaborative surgery has gained traction in medical research. Prominent examples include providing haptic feedback to sense tissue consistency, and automating sub-tasks during surgery such as cutting or needle hand-off - pulling and reorienting the needle after insertion during suturing. By fragmenting suturing into automated and manual tasks the surgeon could essentially control the process with one hand and also circumvent workspace restrictions imposed by the control interface present at the surgeon's side during the operation. This paper presents an exploration of a discrete reinforcement learning-based approach to automate the needle hand-off task. Users were asked to perform a simple running suture using the da Vinci Research Kit. The user trajectory was learnt by generating a sparse reward function and deriving an optimal policy using Q-learning. Trajectories obtained from three learnt policies were compared to the user defined trajectory. The results showed a root-mean-square error of [0.0044mm, 0.0027mm, 0.0020mm] in ℝ3. Additional trajectories from varying initial positions were produced from a single policy to simulate repeated passes of the hand-off task.

Robot Dynamic Simulators offer convenient implementation and testing of physical robots, thus accelerating research and development. While existing simulators support most real-world robots with serially linked kinematic and dynamic chains, they offer limited or conditional support for complex closed-loop robots. On the other hand, many of the underlying physics computation libraries that these simulators employ support closed-loop kinematic chains and redundant mechanisms. Such mechanisms are often utilized in surgical robots to achieve constrained motions (e.g., the remote center of motion (RCM)). To deal with such robots, we propose a new simulation framework based on a front-end description format and a robust real-time dynamic simulator. Although this study focuses on surgical robots, the proposed format and simulator are applicable to any type of robot. In this manuscript, we describe the philosophy and implementation of the front-end description format and demonstrate its performance and the simulator’s capabilities using simulated models of real-world surgical robots.

Surgical robots for laparoscopy consist of several patient side slave manipulators that are controlled via surgeon operated master telemanipulators. Commercial surgical robots do not perform any sub-tasks - even of repetitive or noninvasive nature - autonomously or provide intelligent assistance. While this is primarily due to safety and regulatory reasons, the state of such automation intelligence also lacks the reliability and robustness for use in high-risk applications. Recent developments in continuous control using Artificial Intelligence and Reinforcement Learning have prompted growing research interest in automating mundane sub-tasks. To build on this, we present an inspired Asynchronous Framework which incorporates realtime dynamic simulation - manipulable with the masters of a surgical robot and various other input devices - and interfaces with learning agents to train and potentially allow for the execution of shared sub-tasks. The scope of this framework is generic to cater to various surgical (as well as non-surgical) training and control applications. This scope is demonstrated by examples of multi-user and multi-manual applications which allow for realistic interactions by incorporating distributed control, shared task allocation and a well-defined communication pipe-line for learning agents. These examples are discussed in conjunction with the design philosophy, specifications, system-architecture and metrics of the Asynchronous Framework and the accompanying Simulator. We show the stability of Simulator while achieving real-time dynamic simulation and interfacing with several haptic input devices and a training agent at the same time.

The use of robotics for laparoscopic surgery has been an established field for over a decade. However, with the influx of advanced tools and algorithms for general purpose robotics, there is a need to incorporate these advancements into medical robotics technology. The daVinci Research Kit and its software framework provides a step towards these advancements. This paper presents the development of new tools and utilization of previously developed tools used for general purpose robotics, and their tailored use in medical robotics. Additionally, a method for computing haptic forces for tele-operated surgical robots is presented. The technique utilizes elastic, Spherical Proxy Regions (SPR) to readily compute directional interaction forces and manipulate them to create a dynamic behavior at the surgeon/user's manipulator.