Robotic assistance for experimental manipulation in the life sciences is expected to enable favorable outcomes, regardless of the skill of the scientist. Experimental specimens in the life sciences are subject to individual variability and hence require intricate algorithms for successful autonomous robotic control. As a use case, we are studying the cranial window creation in mice. This operation requires the removal of an 8-mm circular patch of the skull, which is approximately 300 μm thick, but the shape and thickness of the mouse skull significantly varies depending on the strain of the mouse, sex, and age. In this work, we develop an autonomous robotic drilling system with no offline planning, consisting of a trajectory planner with execution-time feedback with drilling completion level recognition based on image and force information. In the experiments, we first evaluate the image-and-force-based drilling completion level recognition by comparing it with other state-of-the-art deep learning image processing methods and conduct an ablation study in eggshell drilling to evaluate the impact of each module on system performance. Finally, the system performance is further evaluated in postmortem mice, achieving a success rate of 70% (14/20 trials) with an average drilling time of 9.3 min.

Robotic systems for manipulation in millimeter scale often use a camera with high magnification for visual feedback of the target region. However, the limited field-of-view (FoV) of the microscopic camera necessitates camera motion to capture a broader workspace environment. In this work, we propose an autonomous robotic control method to constrain a robot-held camera within a designated FoV. Furthermore, we model the camera extrinsics as part of the kinematic model and use camera measurements coupled with a U-Net based tool tracking to adapt the complete robotic model during task execution. As a proof-of-concept demonstration, the proposed framework was evaluated in a bi-manual setup, where the microscopic camera was controlled to view a tool moving in a pre-defined trajectory. The proposed method allowed the camera to stay 94.1% of the time within the real FoV, compared to 54.4% without the proposed adaptive control.

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multicenter study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and post-processing (66%). The “typical” lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

Vitreoretinal surgery pertains to the treatment of delicate tissues on the fundus of the eye using thin instruments. Surgeons frequently rotate the eye during surgery, which is called orbital manipulation, to observe regions around the fundus without moving the patient. In this paper, we propose the autonomous orbital manipulation of the eye in robot-assisted vitreoretinal surgery with our tele-operated surgical system. In a simulation study, we preliminarily investigated the increase in the manipulability of our system using orbital manipulation. Furthermore, we demonstrated the feasibility of our method in experiments with a physical robot and a realistic eye model, showing an increase in the view-able area of the fundus when compared to a conventional technique. Source code and minimal example available at https://github.com/mmmarinho/icra2023_orbitalmanipulation.

Robotic assistance for experimental manipulation in the life sciences is expected to enable precise manipulation of valuable samples, regardless of the skill of the scientist. Experimental specimens in the life sciences are subject to individual variability and deformation, and therefore require autonomous robotic control. As an example, we are studying the installation of a cranial window in a mouse. This operation requires the removal of the skull, which is approximately 300 um thick, to cut it into a circular shape 8 mm in diameter, but the shape of the mouse skull varies depending on the strain of mouse, sex and week of age. The thickness of the skull is not uniform, with some areas being thin and others thicker. It is also difficult to ensure that the skulls of the mice are kept in the same position for each operation. It is not realistically possible to measure all these features and pre-program a robotic trajectory for individual mice. The paper therefore proposes an autonomous robotic drilling method. The proposed method consists of drilling trajectory planning and image-based task completion level recognition. The trajectory planning adjusts the z-position of the drill according to the task completion level at each discrete point, and forms the 3D drilling path via constrained cubic spline interpolation while avoiding overshoot. The task completion level recognition uses a DSSD-inspired deep learning model to estimate the task completion level of each discrete point. Since an egg has similar characteristics to a mouse skull in terms of shape, thickness and mechanical properties, removing the egg shell without damaging the membrane underneath was chosen as the simulation task. The proposed method was evaluated using a 6-DOF robotic arm holding a drill and achieved a success rate of 80% out of 20 trials.

Surgical robots are usually controlled using a priori models based on the robots’ geometric parameters, which are calibrated before the surgical procedure. One of the challenges in using robots in real surgical settings is that those parameters can change over time, consequently deteriorating control accuracy. In this context, our group has been investigating online calibration strategies without added sensors. In one step toward that goal, we have developed an algorithm to estimate the pose of the instruments’ shafts in endoscopic images. In this study, we build upon that earlier work and propose a new framework to more precisely estimate the pose of a rigid surgical instrument. Our strategy is based on a novel pose estimation model called MBAPose and the use of synthetic training data. Our experiments demonstrated an improvement of 21 % for translation error and 26 % for orientation error on synthetic test data with respect to our previous work. Results with real test data provide a baseline for further research.

Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. In this work, we propose a new method to estimate the pose of the surgical instruments' shafts using a monocular endoscope. Our method is based on the use of an automatically annotated training dataset and an improved pose-estimation deep-learning architecture. In preliminary experiments, we show that our method can surpass state of the art vision-based marker-less pose estimation techniques (providing an error decrease of 55% in position estimation, 64% in pitch, and 69% in yaw) by using artificial images.

Minimally Invasive Surgery (MIS) is one of the most successful applications of surgical robots. Although the introduction of robotic technology has brought a number of benefits, further advancements in MIS are limited by the size and bending radius of instruments. In this paper, we present a compliant four degree-of-freedom manipulator that consists of elastic elements with partly thinner structures. The proposed mechanism allows the elastic element to deform locally, thus minimizing its bending radius while the low number of mechanical parts greatly contributes to its compactness. This paper describes the design strategy, optimization method using FEA, prototype implementation, and evaluations. The evaluations reveal high accuracy and repeat accuracy, which are key elements for robotic instruments in MIS. Further, the prototype is able to exert sufficient force and it is possible to perform a simulated needle insertion task using the manipulator, demonstrating the feasibility of the proposed mechanism.

In adult laparoscopy, robot-aided surgery is a reality in thousands of operating rooms worldwide, owing to the increased dexterity provided by the robotic tools. Many robots and robot control techniques have been developed to aid in more challenging scenarios, such as pediatric surgery and microsurgery. However, the prevalence of case-specific solutions, particularly those focused on non-redundant robots, reduces the reproducibility of the initial results in more challenging scenarios. In this paper, we propose a general framework for the control of surgical robotics in constrained workspaces under teleoperation, regardless of the robot geometry. Our technique is divided into a slave-side constrained optimization algorithm, which provides virtual fixtures, and with Cartesian impedance on the master side to provide force feedback. Experiments with two robotic systems, one redundant and one non-redundant, show that smooth teleoperation can be achieved in adult laparoscopy and infant surgery.

Robotic assistance allows surgeons to perform dexterous and tremor-free procedures, but robotic aid is still under-represented in procedures with constrained workspaces, such as deep brain neurosurgery and endonasal surgery. In these procedures, surgeons have restricted vision to areas near the surgical tooltips, which increases the risk of unexpected collisions between the shafts of the instruments and their surroundings. In this paper, our vector-field-inequalities method is extended to provide dynamic active-constraints to any number of robots and moving objects sharing the same workspace. The method is evaluated with experiments and simulations in which robot tools have to avoid collisions autonomously and in real-time, in a constrained endonasal surgical environment. Simulations show that with our method the combined trajectory error of two robotic systems is optimal. Experiments using a real robotic system show that the method can autonomously prevent collisions between the moving robots themselves and between the robots and the environment. Moreover, the framework is also successfully verified under teleoperation with tool-tissue interactions.

Robotic assistance allows surgeons to perform dexterous and tremor-free procedures, but is still underrepresented in deep brain neurosurgery and endonasal surgery where the workspace is constrained. In these conditions, the vision of surgeons is restricted to areas near the surgical tool tips, which increases the risk of unexpected collisions between the shafts of the instruments and their surroundings, in particular in areas outside the surgical field-of-view. Active constraints can be used to prevent the tools from entering restricted zones and thus avoid collisions. In this paper, a vector field inequality is proposed that guarantees that tools do not enter restricted zones. Moreover, in contrast with early techniques, the proposed method limits the tool approach velocity in the direction of the forbidden zone boundary, guaranteeing a smooth behavior and that tangential velocities will not be disturbed. The proposed method is evaluated in simulations featuring two eight degrees-of-freedom manipulators that were custom-designed for deep neurosurgery. The results show that both manipulator-manipulator and manipulator-boundary collisions can be avoided using the vector field inequalities.

The automation of surgical tasks has great potential for improving the performance of robotic surgery. The learning-from-demonstration approach has thus far been employed by many researchers when planning the trajectories of robotic instruments for automated surgical tasks. However, previous methods are applicable only when the demonstrations and trajectory generation are performed under the same initial conditions. In this paper, we propose an algorithm that learns through demonstrations and generates a trajectory regardless of the initial conditions. The variance of the demonstrated trajectories over the initial conditions is modeled and learned using a statistical method, and the learned trajectories are generalized and used to generate a trajectory. As an example, the bi-manual looping task of a surgical thread was demonstrated by changing the initial positions of the robot arms, and a trajectory was then planned given these initial arbitrary positions. The proposed algorithm was verified through simulations and experiments using an actual robotic surgical system.

Advanced robotic assistance in microsurgery, such as automation, requires an accurate estimation of the state of the robotic forceps. In this paper, we propose a robust and accurate forceps tracking method to estimate the full state of the forceps (i.e., the position, posture, and grip parameters) using visual information obtained from stereo microscopic images and kinematic information obtained from the robotic sensory information, forward kinematics, and hand-eye coordination. An online method for updating the hand-eye coordination was also developed using an extended Kalman filter to cancel the hand-eye coordination errors caused by the repositioning of the microscope. The experimental results showed that the proposed method could accurately and robustly estimate the state of the robotic forceps even after the repositioning of the microscope.

Loss of haptic sensation in a master-slave system is one of the open problems in robotic surgery, and recognition of surgical situations through haptic sensation is a challenge. In this paper we propose an autonomous risk-detection system for a master-slave surgical robotic system in order to estimate a property of an object (i.e., contact impedance) using a force sensor mounted on a surgical robotic instrument. The system autonomously detects the risk based on the estimated contact impedance and accordingly activates the motion at the slave unit as well as the force feedback at the master unit. We implemented the proposed method in a teleoperated master-slave system to detect the perforation risk of a membranous object. The performance of the system was evaluated through experiments. The classification accuracy for perforation risk was about 98.5 % in fourfold cross-validation. The experiments verified that the risk detection system accurately detected the perforation risk and improved the safety of the master-slave system.

Forceps tracking is an important element of high-level surgical assistance such as visual servoing and surgical motion analysis. In many computer vision algorithms, artificial markers are used to enable robust tracking; however, markerless tracking methods are more appropriate in surgical applications due to their sterilizability. This paper describes a robust, efficient tracking algorithm capable of estimating the full state parameters of a robotic surgical instrument on the basis of projective contour modeling using a 3-D CAD model of the forceps. Thus, the proposed method does not require any artificial markers. The likelihood of the contour model was measured using edge distance transformation to evaluate the similarity of the projected CAD model to the microscopic image, followed by particle filtering to estimate the full state of the forceps. Experimental results in simulated surgical environments indicate that the proposed method is robust and time-efficient, and fulfills real-time processing requirements.

In this paper, a reconfigurable modular robotic system is proposed to augment the dexterity of endoluminal interventions in the gastrointestinal tract. In the proposed system, miniaturized robotic modules are ingested and assembled in the stomach cavity. The assembled robot can change its configuration according to the target location, thus enabling complicated surgical tasks. The robotic assembly, the robotic configuration and the surgical tasks are controlled via wireless bidirectional communication. Based on this concept, early prototypes of the robotic modules were designed and fabricated. The developed module has 2DOF (±90° of bending and 360° of rotation), measures 15.4 mm in diameter and 36.5 mm in length. It weighs 5.6 g and contains a Li-Po battery, two brushless DC motors, and a custom-made control board capable of wireless communication. The performance of the bending and rotational motion was evaluated and the future work has been discussed.

An automated design generation algorithm for a serial kinematic chain is presented for the reconfigurable robot used in a novel endoluminal surgical procedure (European Union project ARES). The algorithm produces the possible topologies, given the design constraints, desired performance, and available modules, such that all constraints are satisfied for every point in the desired workspace. This is achieved through the use of interval analysis methods and branch-and-bound loop that searches through the end-effector pose and the design parameter spaces. The resulting algorithm is demonstrated through an example of a serial chain manipulator made of the reconfigurable modules of the surgical robot for the application. The results are presented and discussed.

A bending laser manipulator of 2.4 mm in diameter has been developed for intrauterine fetal surgery. This manipulator deflects a laser fiber in any direction thorough 90 degrees. The results of a positioning test and in vitro/in vivo tests are reported. Meanwhile, creep tests for fetal rat tissue of 16 to 20 days in gestation were performed to evaluate fetal tissue fragility. Unique features of fetal rat tissue compared to other soft organs are discussed and a viscoelastic model of the fetal rat tissue was proposed. The result of the modeling will be used not only for fabricating fetal tissue phantom but also for the force control of robotic application for fetal surgery.

We propose a new surgical robotic system for intrauterine fetal surgery in an Open MRI. The target disease of the fetal surgery is spina bifida or myelomeningocele that is incomplete closure in the spinal column and one of the common fetal diseases. In the proposed surgical process, the abdominal wall and uterine wall would not widely be opened but rather surgical instruments inserted through the small holes in both walls to perform minimally invasive surgery. In this paper, a prototype of the micro manipulator of diameter is 2.4mm and bending radius 2.45 mm is presented. The diameter and bending radius of this manipulator is one of the smallest ever developed among surgical robots to the best of the knowledge of the investigating authors. The mechanism of the manipulator includes two ball joints and is driven using four wires able to bend through 90 degrees in any direction. The features of the mechanism include a small diameter, small bending radius, ease of fabrication, high rigidity and applicability for other surgical applications. Although the manipulator is not yet MRI compatible, the feature of the prototype demonstrated the feasibility of robotic intrauterine fetal surgery.