TransSeptal Puncture (TSP) is a key step in many minimally invasive cardiac procedures, enabling access to the left atrium by crossing the interatrial septum from the right atrium.The task demands extreme precision, as excessive force may cause cardiac tamponade.Robotic platforms can improve precision and repeatability, but most of the existing systems are designed for training or teleoperation rather than autonomous execution, and do not incorporate adaptive impedance modulation.This work presents a learning-from-demonstration framework based on probabilistic impedance modeling to investigate the feasibility of learning and embedding expert force-regulation strategies for autonomous execution of the TSP under cliniciandefined targets.Specifically, Gaussian Mixture Models, trained on teleoperated demonstrations, capture the relationship between contact force and operator stiffness, enabling real-time modulation of impedance through Gaussian Mixture Regression.The adaptive controller was deployed on a 7-DoF robotic platform and validated on fossa ovalis phantoms of varying compliance and on ex vivo porcine tissue.Performance was evaluated using clinically relevant metrics: Needle Puncture Force (NPF), Tenting Distance (TD), and Needle Stopping Space (NSS).Compared to fixed-stiffness baselines, the proposed controller reduced NPF by up to 35%, maintained TD within safe limits, and limited NSS below 1 mm in nominal anatomies, with consistent performance also observed under extreme phantom anatomies and ex vivo tissue.These findings demonstrate that embedding humanlike impedance modulation can enable safe, anatomy-aware autonomous control, advancing robotic TSP toward clinical feasibility.Note to Practitioners-This study proposes a learning-based adaptive impedance control scheme for TSP, a delicate step in minimally invasive cardiac procedures where excessive puncture forces can lead to severe complications.While robotic systems can enhance precision and repeatability, existing platforms cannot typically autonomously adapt to patient-specific anatomical variability, relying instead on operator input.The proposed approach consists of two main steps.First, impedance profiles are extracted from teleoperated demonstrations, capturing the relationship between applied force and operator stiffness through a probabilistic model.Then, during autonomous execution, the robot leverages this model to modulate its impedance in real time, adapting its behavior to the compliance of the cardiac tissue to perform safe and precise punctures.By embedding human-like impedance modulation into the control loop, this framework combines surgical expertise with robotic precision, improving procedural safety and consistency.Beyond TSP, the same strategy can be applied to other surgical tasks that require adaptation to different patient anatomies, such as vascular access,

In robot-assisted minimally invasive surgery, optimal camera positioning is crucial for effective visualization and manipulation of tissues, which impacts the success of procedures. Traditional camera control can increase cognitive workload, lead to suboptimal camera viewpoints, and complicate surgical tasks. We propose an autonomous camera system that uses situational awareness and real-time prediction of user intent from kinematic data, and adjusts the camera position dynamically during a simulated suturing task. Such a system reduces the need to manually adjust the camera, allowing users to stay focused on the procedure. We demonstrated the framework in a user study with eight non-expert participants. They used the da Vinci Research Kit to control a simulated camera and instruments in a suturing task. We compared the performances in the suturing task with the autonomous and the teleoperated camera control. The autonomous system reduced execution time by 43% , shortened path length by 30% , and decreased completion cost by 35%. These results serve as a proof of concept for a situationally aware camera system and suggest that autonomous camera control can improve efficiency and simplify surgical workflow.

This paper proposes a quality-driven adaptive control framework for robotic vascular anatomies scanning to facilitate the acquisition of high-quality ultrasound (US) images. Specifically, a novel probability-based US image quality evaluation metric for vascular anatomies is introduced, leveraging an image segmentation network to establish a mapping between the controlled variables of the robot (e.g., pose and force) and US image quality. Furthermore, an adaptive US probe control strategy driven by US image quality is developed to optimize real-time image acquisition, with its stability rigorously proven. To assess the effectiveness of the proposed framework, two experiments were conducted on a human tissue-mimicking phantom, encompassing both static and dynamic scenarios. The experimental results demonstrate that the proposed framework ensures stable contact force and significantly enhances US image quality for robot-assisted vascular anatomy imaging, even in the presence of external disturbances.

Lower-limb exosuits are particularly relevant for individuals with some degree of mobility impairment, such as post-stroke patients or older adults with reduced movement capabilities. This study aims to investigate the mental workload (MWL) assessment of XoSoft, a lower-limb soft exoskeleton, using and comparing subjective and objective physiological metrics. The NASA-TLX questionnaire, the average percentage change in pupil size (APCPS), and the Baevsky stress index (SI) are compared. The experiments were conducted on 18 healthy subjects while walking and involved mathematical tasks to create a double-task condition. The results show a complex interaction between task difficulty, exoskeleton activation, and pupillary dynamics, suggesting that the subject might reach a saturated condition under a high mental load. Besides, the data indicate that pupil diameter may be an objective mental workload indicator that correlates with subjective NASA-TLX questionnaires. The discordant indications from the stress index suggest how different metrics of the ocular and cardiac levels respond differently to various stimuli and dynamics. Research has also revealed ocular asymmetry, with the right eye more sensitive to cognitive load.

This work presents a compliant and passive shared control framework for teleoperated robot-assisted tasks. Inspired by the human operator's capability of continuously regulating the arm impedance to perform contact-rich tasks, a novel control schema, exploiting the variable impedance control framework for force tracking is proposed. Moreover, bilateral teleoperation and shared control strategies are implemented to alleviate the human operator's workload. Furthermore, a global energy tank-based approach is integrated to enforce the system's passivity. The proposed framework is first evaluated to assess the force-tracking capability when the robot autonomously performs contact-rich tasks, e.g., in an ultrasound scanning scenario. Then, a validation experiment is conducted utilizing the proposed shared control framework. Finally, the system's usability is investigated with 12 users. The experiment results in system assessment revealed a maximum median error of 0.25 N across all the force-tracking experiment setups, i.e., constant and time-varying ones. Then, the validation experiment demonstrated significant improvements regarding the force tracking tasks compared to conventional control methods, and the system passivity was preserved during the task execution. Finally, the usability experiment shows that the human operator workload is significantly reduced by$54.6 \%$compared to the other two control modalities. The proposed framework holds significant potential for the execution of remote robot-assisted medical procedures, such as palpation and ultrasound scanning, particularly in addressing deformation challenges while ensuring safety, compliance, and system passivity.

Despite impressive advancements of industrial collaborative robots, their potential remains largely untapped due to the difficulty in balancing human safety and comfort with fast production constraints. To help address this challenge, we present PRO-MIND, a novel human-in-the-loop framework that exploits valuable data about the human coworker to optimize robot trajectories. By estimating human attention and mental effort, our method dynamically adjusts safety zones and enables on-the-fly alterations of the robot path to enhance human comfort and optimal stopping conditions. Moreover, we formulate a multiobjective optimization to adapt the robot's trajectory execution time and smoothness based on the current human psychophysical stress, estimated from heart rate variability and frantic movements. These adaptations exploit the properties of B-spline curves to preserve continuity and smoothness, which are crucial factors in improving motion predictability and comfort. Evaluation in two realistic case studies showcases the framework's ability to restrain the operators' workload and stress and to ensure their safety while enhancing human–robot productivity. Further strengths of PRO-MIND include its adaptability to each individual's specific needs and sensitivity to variations in attention, mental effort, and stress during task execution.

Endovascular intervention is a minimally invasive method for treating cardiovascular diseases. Although fluoroscopy, known for real-time catheter visualization, is commonly used, it exposes patients and physicians to ionizing radiation and lacks depth perception due to its 2D nature. To address these limitations, a study was conducted using teleoperation and 3D visualization techniques. This in-vitro study involved the use of a robotic catheter system and aimed to evaluate user performance through both subjective and objective measures. The focus was on determining the most effective modes of interaction. Three interactive modes for guiding robotic catheters were compared in the study: 1) Mode GM, using a gamepad for control and a standard 2D monitor for visual feedback; 2) Mode GH, with a gamepad for control and HoloLens providing 3D visualization; and 3) Mode HH, where HoloLens serves as both control input and visualization device. Mode GH outperformed other modalities in subjective metrics, except for mental demand. It exhibited a median tracking error of 4.72 mm, a median targeting error of 1.01 mm, a median duration of 82.34 s, and a median natural logarithm of dimensionless squared jerk of 40.38 in the in-vitro study. Mode GH showed 8.5%, 4.7%, 6.5%, and 3.9% improvements over Mode GM and 1.5%, 33.6%, 34.9%, and 8.1% over Mode HH for tracking error, targeting error, duration, and dimensionless squared jerk, respectively. To sum up, the user study emphasizes the potential benefits of employing HoloLens for enhanced 3D visualization in catheterization. The user study also illustrates the advantages of using a gamepad for catheter teleoperation, including user-friendliness and passive haptic feedback, compared to HoloLens. To further gauge the potential of using a more traditional joystick as a control input device, an additional study utilizing the Haption Virtuose robot was conducted. It reveals the potential for achieving smoother trajectories, with a 38.9% reduction in total path length compared to a gamepad, potentially due to its larger range of motion and single-handed control.

Lymphadenectomy generally accompanies various oncology surgeries to remove infected cancer cells. However, there are two limitations in robot-assisted lymphadenectomy: 1) lymph nodes are not visible during operation since they are hidden by the superficial fat layer; 2) intra-operative bleeding may occur during lymph node removal caused by collisions between surgical instruments and delicate blood vessels (arteries or veins) near the lymph nodes. Therefore, we propose a framework integrating augmented reality and virtual fixtures to address these limitations. Augmented reality intra-operatively visualizes the hidden lymph nodes by projecting the corresponding 3D pre-operative model, and virtual fixtures are used to provide force feedback to surgeons to avoid possible collisions when they operate the surgical instruments to resect the lymph nodes surrounding the blood vessel. Ten human subjects were invited to perform an emulated lymphadenectomy based on the da Vinci robot in a dry lab. Experimental results demonstrated that the proposed framework can keep localizing the hidden lymph nodes, and reduce the number of collisions (21% and 48% reduction rates using two different force models compared to the standard setup, respectively) between the instruments and the delicate blood vessel during lymph node resection. It shows the potential to enhance the safety of robot-assisted lymphadenectomy.

Preserving and encouraging mobility in the elderly and adults with chronic conditions is of paramount importance. However, existing walking aids are either inadequate to provide sufficient support to users’ stability or too bulky and poorly maneuverable to be used outside hospital environments. In addition, they all lack adaptability to individual requirements. To address these challenges, this paper introduces WANDER, a novel Walking Assistive omNi-Directional Exo-Robot. It consists of an omnidirectional platform and a robust aluminum structure mounted on top of it, which provides partial body weight support. A comfortable and minimally restrictive coupling interface embedded with a force/torque sensor allows to detect users’ intentions, which are translated into command velocities by means of a variable admittance controller. An optimization technique based on users’ preferences, i.e., Preference-Based Optimization (PBO) guides the choice of the admittance parameters (i.e., virtual mass and damping) to better fit subject-specific needs and characteristics. Experiments with twelve healthy subjects exhibited a significant decrease in energy consumption and jerk when using WANDER with PBO parameters as well as improved user performance and comfort. The great interpersonal variability in the optimized parameters highlights the importance of personalized control settings when walking with an assistive device, aiming to enhance users’ comfort and mobility while ensuring reliable physical support.

This paper presents a method to compute smooth ankle trajectories for lower limb exoskeletons with powered ankle joints. The proposed approach defines ankle trajectories using four polynomial functions, each representing one of the four primary phases of gait. These polynomials are computed according to different safety constraints. During the single support phase, ground contact constraints are enforced. In the swing phase, an optimization problem is solved to achieve minimum jerk planning while respecting a set of equality and inequality constraints designed to minimize the risk of stumbling. The used approach focuses on making the ankle joint able to smoothly adapt in real-time to different walking styles defined by user-selected gait parameters such as step length and clearance. The primary aim is to improve the user experience by producing a secure and comfortable walking pattern. To validate the effectiveness of the proposed method, the new ankle trajectories were tested on a group of healthy volunteers using the TWIN lower limb exoskeleton.

The integration of high-level assistance algorithms in surgical robotics training curricula may be beneficial in establishing a more comprehensive and robust skillset for aspiring surgeons, improving their clinical performance as a consequence. This work presents the development and validation of a haptic-enhanced Virtual Reality simulator for surgical robotics training, featuring 8 surgical tasks that the trainee can interact with thanks to the embedded physics engine. This virtual simulated environment is augmented by the introduction of high-level haptic interfaces for robotic assistance that aim at re-directing the motion of the trainee’s hands and wrists toward targets or away from obstacles, and providing a quantitative performance score after the execution of each training exercise.An experimental study shows that the introduction of enhanced robotic assistance into a surgical robotics training curriculum improves performance during the training process and, crucially, promotes the transfer of the acquired skills to an unassisted surgical scenario, like the clinical one.

Fetoscopy laser photocoagulation is a widely adopted procedure for treating Twin-to-Twin Transfusion Syndrome (TTTS). The procedure involves photocoagulation pathological anastomoses to restore a physiological blood exchange among twins. The procedure is particularly challenging, from the surgeon's side, due to the limited field of view, poor manoeuvrability of the fetoscope, poor visibility due to amniotic fluid turbidity, and variability in illumination. These challenges may lead to increased surgery time and incomplete ablation of pathological anastomoses, resulting in persistent TTTS. Computer-assisted intervention (CAI) can provide TTTS surgeons with decision support and context awareness by identifying key structures in the scene and expanding the fetoscopic field of view through video mosaicking. Research in this domain has been hampered by the lack of high-quality data to design, develop and test CAI algorithms. Through the Fetoscopic Placental Vessel Segmentation and Registration (FetReg2021) challenge, which was organized as part of the MICCAI2021 Endoscopic Vision (EndoVis) challenge, we released the first large-scale multi-center TTTS dataset for the development of generalized and robust semantic segmentation and video mosaicking algorithms with a focus on creating drift-free mosaics from long duration fetoscopy videos. For this challenge, we released a dataset of 2060 images, pixel-annotated for vessels, tool, fetus and background classes, from 18 in-vivo TTTS fetoscopy procedures and 18 short video clips of an average length of 411 frames for developing placental scene segmentation and frame registration for mosaicking techniques. Seven teams participated in this challenge and their model performance was assessed on an unseen test dataset of 658 pixel-annotated images from 6 fetoscopic procedures and 6 short clips. For the segmentation task, overall baseline performed was the top performing (aggregated mIoU of 0.6763) and was the best on the vessel class (mIoU of 0.5817) while team RREB was the best on the tool (mIoU of 0.6335) and fetus (mIoU of 0.5178) classes. For the registration task, overall the baseline performed better than team SANO with an overall mean 5-frame SSIM of 0.9348. Qualitatively, it was observed that team SANO performed better in planar scenarios, while baseline was better in non-planner scenarios. The detailed analysis showed that no single team outperformed on all 6 test fetoscopic videos. The challenge provided an opportunity to create generalized solutions for fetoscopic scene understanding and mosaicking. In this paper, we present the findings of the FetReg2021 challenge, alongside reporting a detailed literature review for CAI in TTTS fetoscopy. Through this challenge, its analysis and the release of multi-center fetoscopic data, we provide a benchmark for future research in this field.

Robot-assisted surgery is rapidly developing in the medical field, and the integration of augmented reality shows the potential to improve the operation performance of surgeons by providing more visual information. In this paper, we proposed a markerless augmented reality framework to enhance safety by avoiding intra-operative bleeding, which is a high risk caused by collision between surgical instruments and delicate blood vessels (arteries or veins). Advanced stereo reconstruction and segmentation networks are compared to find the best combination to reconstruct the intra-operative blood vessel in 3D space for registration with the pre-operative model, and the minimum distance detection between the instruments and the blood vessel is implemented. A robot-assisted lymphadenectomy is emulated on the da Vinci Research Kit in a dry lab, and ten human subjects perform this operation to explore the usability of the proposed framework. The result shows that the augmented reality framework can help the users to avoid the dangerous collision between the instruments and the delicate blood vessel while not introducing an extra load. It provides a flexible framework that integrates augmented reality into the medical robotic platform to enhance safety during surgery.

This paper presents an Augmented Reality (AR)assisted robot learning framework for Minimally Invasive Surgery (MIS) tasks. The proposed framework exploits an external optical tracking system to collect human demonstration. Gaussian Mixture Model (GMM) and Gaussian Mixture Regression (GMR) are utilized to encode and generate a robust desired trajectory for transferring to the real robot for the MIS task. The HoloLens 2 Head-Mounted-Display (HMD) is integrated for intuitive visualization of the robot configuration under the constraint of a small incision on the patient's abdominal cavity during the demonstration phase. Experiments are conducted to verify the feasibility and performance of the proposed framework and compared it with the kinesthetic teaching-based modality in a tumor resection MIS task. The results illustrate that the proposed AR-assisted robot learning framework requires lower workload demand, achieves higher performance and efficiency, and ensures the feasibility of the learned results for reproduction on a real robot for MIS tasks.

In this paper, a position-based visual-servoing control approach is introduced for a robotic camera holder to improve ergonomics and reduce mental stress during brain surgery. The visual tracking system controls and moves the robotic camera holder by following a selected surgical instrument without the need for artificial markers. The system was validated using a 7 Degree-of-Freedoms (DoFs) redundant robotic manipulator with an eye-in-hand stereo camera configuration and compared with conventional control methods using NASA TLX survey and four objective metrics, including execution time, time out of field of view (FoV), target score, and path length. Experimental results demonstrate that the proposed system can reduce the surgeon's workload during brain surgery-related task execution, improve ergonomics and achieve higher performance than traditional control methods.

This paper proposes a combined optimization and learning method for impact-friendly, non-prehensile catching of objects at non-zero velocity. Through a constrained Quadratic Programming problem, the method generates optimal trajectories up to the contact point between the robot and the object to minimize their relative velocity and reduce the impact forces. Next, the generated trajectories are updated by Kernelized Movement Primitives, which are based on human catching demonstrations to ensure a smooth transition around the catching point. In addition, the learned human variable stiffness (HVS) is sent to the robot's Cartesian impedance controller to absorb the post-impact forces and stabilize the catching position. Three experiments are conducted to compare our method with and without HVS against a fixed-position impedance controller (FP-IC). The results showed that the proposed methods outperform the FP-IC while adding HVS yields better results for absorbing the post-impact forces.

Increased demand for less invasive procedures has accelerated the adoption of Intraluminal Procedures (IP) and Endovascular Interventions (EI) performed through body lumens and vessels. As navigation through lumens and vessels is quite complex, interest grows to establish autonomous navigation techniques for IP and EI for reaching the target area. Current research efforts are directed toward increasing the Level of Autonomy (LoA) during the navigation phase. One key ingredient for autonomous navigation is Motion Planning (MP) techniques. This paper provides an overview of MP techniques categorizing them based on LoA. Our analysis investigates advances for the different clinical scenarios. Through a systematic literature analysis using the PRISMA method, the study summarizes relevant works and investigates the clinical aim, LoA, adopted MP techniques, and validation types. We identify the limitations of the corresponding MP methods and provide directions to improve the robustness of the algorithms in dynamic intraluminal environments. MP for IP and EI can be classified into four subgroups: node, sampling, optimization, and learning-based techniques, with a notable rise in learning-based approaches in recent years. One of the review's contributions is the identification of the limiting factors in IP and EI robotic systems hindering higher levels of autonomous navigation. In the future, navigation is bound to become more autonomous, placing the clinician in a supervisory position to improve control precision and reduce workload.

Over the past years, action recognition techniques have gained significant attention in computer vision and robotics research. Nevertheless, their performances in realistic applications, despite dedicated efforts to collect and annotate medium/large datasets, remain far from satisfactory, especially when it comes to applications in the field of human-robot collaboration. In response to this shortfall, we create a dataset not dispersive in its classes but sectoral, i.e., dedicated exclusively to the industrial environment and human-robot collaboration. Specifically, we describe our ongoing collection of the ’HRI30’ database for industrial action recognition from videos, containing 30 categories of industrial-like actions and 2940 manually annotated clips. We test our dataset on multiple action detection approaches and compare it with the HMDB51 and UCF101 public datasets using the best-performing approach. We define a baseline of 86.55% Top-1 accuracy and 99.76% Top-5 accuracy, hoping that this dataset will encourage research towards understanding actions in collaborative industrial scenarios. The dataset can be downloaded at the following link: 10.5281/zenodo.5833411

In hybrid industrial environments, workers' comfort and positive perception of safety are essential requirements for successful acceptance and usage of collaborative robots. This paper proposes a novel human-robot interaction framework in which the robot behaviour is adapted online according to the operator's cognitive workload and stress. The method exploits the generation of B-spline trajectories in the joint space and formulation of a multi-objective optimisation problem to online adjust the total execution time and smoothness of the robot trajectories. The former ensures human efficiency and productivity of the workplace, while the latter contributes to safeguarding the user's comfort and cognitive ergonomics. The performance of the proposed framework was evaluated in a typical industrial task. Results demonstrated its capability to enhance the productivity of the human-robot dyad while mitigating the cognitive workload induced in the worker.

Navigation inside luminal organs is an arduous task that requires non-intuitive coordination between the movement of the operator's hand and the information obtained from the endoscopic video. The development of tools to automate certain tasks could alleviate the physical and mental load of doctors during interventions allowing them to focus on diagnosis and decision-making tasks. In this paper we present a synergic solution for intraluminal navigation consisting of a 3D printed endoscopic soft robot that can move safely inside luminal structures. Visual servoing based on Convolutional Neural Networks (CNNs) is used to achieve the autonomous navigation task. The CNN is trained with phantoms and in-vivo data to segment the lumen and a model-less approach is presented to control the movement in constrained environments. The proposed robot is validated in anatomical phantoms in different path configurations. We analyze the movement of the robot using different metrics such as task completion time smoothness error in the steady-state mean and maximum error. We show that our method is suitable to navigate safely in hollow environments and conditions which are different than the ones the network was originally trained on.

The recognition of actions performed by humans and the anticipation of their intentions are important enablers to yield sociable and successful collaboration in human-robot teams. Meanwhile, robots should have the capacity to deal with multiple objectives and constraints, arising from the collaborative task or the human. In this regard, we propose vision techniques to perform human action recognition and image classification, which are integrated into an Augmented Hierarchical Quadratic Programming (AHQP) scheme to hierarchically optimize the robot's reactive behavior and human ergonomics. The proposed framework allows one to intuitively command the robot in space while a task is being executed. The experiments confirm increased human ergonomics and usability, which are fundamental parameters for reducing musculoskeletal diseases and increasing trust in automation.

Structural intervention cardiology (SIC) interventions are crucial procedures for correcting heart valves, walls, and muscle form defects. However, the possibility of embolization or perforation, as well as the lack of transparent vision and autonomous surgical equipment, make it difficult for the clinician. This paper proposes a robot-assisted tendon-driven catheter and machine learning-based path planner to overcome these challenges. Firstly, an analytical inverse kinematic model is constructed to convert the tip location in the Cartesian space to the tendons' displacement. Then inverse reinforcement learning algorithm is employed to calculate the optimal path to avoid possible collisions between the catheter tip and the atrial wall. Moreover, a closed-loop feedback controller is adopted to improve positioning accuracy in a direct distal position measurement manner. Simulation and experiments are designed and conducted to demonstrate the feasibility and performance of the proposed system.

Simulation tools are essential for robotics research, especially for those domains in which safety is crucial, such as Human-Robot Collaboration (HRC). However, it is challenging to simulate human behaviors, and existing robotics simulators do not integrate functional human models. This work presents Open-VICO, an open-source toolkit to integrate virtual human models in Gazebo focusing on vision-based human tracking. In particular, Open-VICO allows to combine in the same simulation environment realistic human kinematic models, multi-camera vision setups, and human-tracking techniques along with numerous robot and sensor models thanks to Gazebo. The possibility to incorporate pre-recorded human skeleton motion with Motion Capture systems broadens the landscape of human performance behavioral analysis within Human-Robot Interaction (HRI) settings. To describe the functionalities and stress the potential of the toolkit four specific examples, chosen among relevant literature challenges in the field, are developed using our simulation utils: i) 3D multi-RGB-D camera calibration in simulation, ii) creation of a synthetic human skeleton tracking dataset based on OpenPose, iii) multi-camera scenario for human skeleton tracking in simulation, and iv) a human-robot interaction example. The key of this work is to create a straightforward pipeline which we hope will motivate research on new vision-based algorithms and methodologies for lightweight human-tracking and flexible human-robot applications.

Human factors and ergonomics are the essential constituents of teleoperation interfaces, which can significantly affect the human operator’s performance. Thus, a quantitative evaluation of these elements and the ability to establish reliable comparison bases for different teleoperation interfaces are the keys to select the most suitable one for a particular application. However, most of the works on teleoperation have so far focused on the stability analysis and the transparency improvement of these systems and do not cover the important usability aspects. In this article, we propose a foundation to build a general framework for the analysis of human factors and ergonomics in employing diverse teleoperation interfaces. The proposed framework will go beyond the traditional subjective analyses of usability by complementing it with online measurements of human body configurations. As a result, multiple quantitative metrics, such as joints’ usage, range of motion comfort, center of mass divergence, and posture comfort, are introduced. To demonstrate the potential of the proposed framework, two different teleoperation interfaces are considered, and real-world experiments with 11 participants performing a simulated industrial remote pick-and-place task are conducted. The quantitative results of this analysis are provided, and compared with subjective questionnaires, illustrating the effectiveness of the proposed framework.

Recently, the human-like behavior on the anthropomorphic robot manipulator is increasingly accomplished by the kinematic model establishing the relationship of an anthropomorphic manipulator and human arm motions. Notably, the growth and broad availability of advanced data science techniques facilitate the imitation learning process in anthropomorphic robotics. However, the enormous dataset causes the labeling and prediction burden. In this article, the swivel motion reconstruction approach was applied to imitate human-like behavior using the kinematic mapping in robot redundancy. For the sake of efficient computing, a novel incremental learning framework that combines an incremental learning approach with a deep convolutional neural network is proposed for fast and efficient learning. The algorithm exploits a novel approach to detect changes from human motion data streaming and then evolve its hierarchical representation of features. The incremental learning process can fine-tune the deep network only when model drifts detection mechanisms are triggered. Finally, we experimentally demonstrated this neural network's learning procedure and translated the trained human-like model to manage the redundancy optimization control of an anthropomorphic robot manipulator (LWR4+, KUKA, Germany). This approach can hold the anthropomorphic kinematic structure-based redundant robots. The experimental results showed that our architecture could not only enhance the regression accuracy but also significantly reduce the processing time of learning human motion data.

Keyhole neurosurgery is challenging, due to the complex anatomy of the brain and the inherent risk of damaging vital structures while reaching the surgical target. This paper presents a path planner for safe and effective neurosurgical interventions. The strengths of the proposed framework lay in the integration of multiple risk structures combined into a deductive method for fast and intuitive user interaction, and a modular architecture. The tool is intended to support neurosurgeons at quickly determining the most appropriate surgical trajectory through the brain matter with minimized risk; the user interface guides the user through the decision making process and helps save planning time of neurosurgical interventions. Risk structures and trajectories can be visualized in an intuitive way, thanks to a 3D brain surgery simulator developed with Unity. A qualitative evaluation with clinical experts shows the practical relevance, while a quantitative performance and functionality analysis proves the robustness and effectiveness of the system with respect to literature.

Today’s robots are expected to fulfill different requirements originated from executing complex tasks in uncertain environments, often in collaboration with humans. To deal with this type of multi-objective control problem, hierarchical least-square optimization techniques are often employed, defining multiple tasks as objective functions, listed in hierarchical manner. The solution to the Inverse Kinematics problem requires to plan and constantly update the Cartesian trajectories. However, we propose an extension to the classical Hierarchical Quadratic Programming formulation, that allows to optimally generate these trajectories at control level. This is achieved by augmenting the optimization variable, to include the Cartesian reference and allow for the formulation of an adaptive compliance controller, which retains an impedancelike behaviour under external disturbances, while switching to an admittance-like behavior when collaborating with a human. The effectiveness of this approach is tested using a 7-DoF Franka Emika Panda manipulator in three different collaborative scenarios.

Prolonged remote tele-locomanipulation of multi degrees-of-freedom mobile manipulators requires a compromise between the system’s performance and the operator’s ergonomics. Neglecting this demand can significantly affect either the task completion or the level of comfort to achieve it. However, the simultaneous consideration of these key factors has received less attention in the literature. To respond to this demand, in this work, we introduce a new teleoperation setup, which integrates the features of an ergonomic and a highly maneuverable interface into a unified solution. The ergonomic part of the interface implements a 3D mouse-like functionality, enabling the execution of long navigation tasks for the floating base. The highly manoeuvrable interface instead, enables the operator to perform dynamic or more precise manipulation by moving his/her arm in space. The locomotion and manipulation modes of the follower robot are controlled separately, which can be easily and seamlessly switched by the operator by pressing a button at any moment. Furthermore, due to the follower manipulator’s redundancy, this robot is controlled by a hierarchical quadratic programming technique which enables the definition of a set of secondary tasks to be executed in the robot’s nullspace. Finally, to demonstrate the advantages and disadvantages of the proposed user interfaces, five participants are asked to perform two different experiments: (i) target selection task on a moving surface and (ii) remote path tracking on a fixed surface. The quantitative and qualitative analyses show the effectiveness of the proposed interface during the teleoperation tasks, especially when it comes to the precise and dynamic task execution.

The use of assistive technologies in industrial environments to improve human ergonomics and comfort in repetitive and high effort tasks have increased considerably in the last decade. Predominantly, the goal is to provide additional physical support through lightweight and wearable devices, without posing major constraints to the human body movements. Towards achieving this objective, in this work we present a novel actuation mechanism for a soft assistive device, by taking into account the human elbow torque-angle profile. The proposed design integrates a single motor coupled with an elastic bungee and a cam-spool mechanism to enable energy exchange during the elbow flexion movement, while allowing for free-motions during the extension of the joint. A cable-driven transmission with passive elastic attachments is employed to implement compliant couplings with the wearer and to achieve easy donning/doffing. Experiments are conducted on two 3D printed functional prototypes. Results suggest that the assistive elbow torque is effectively transmitted with an average 90% success for balancing a 5N payload, and the free-motion range of 108° is measured for both flexion and extension.

This paper proposes a novel integrated dynamic method based on Behavior Trees for planning and allocating tasks in mixed human robot teams, suitable for manufacturing environments. The Behavior Tree formulation allows encoding a single job as a compound of different tasks with temporal and logic constraints. In this way, instead of the well-studied offline centralized optimization problem, the role allocation problem is solved with multiple simplified online optimization sub-problem, without complex and cross-schedule task dependencies. These sub-problems are defined as Mixed-Integer Linear Programs, that, according to the worker-actions related costs and the workers' availability, allocate the yet-to-execute tasks among the available workers. To characterize the behavior of the developed method, we opted to perform different simulation experiments in which the results of the action-worker allocation and computational complexity are evaluated. The obtained results, due to the nature of the algorithm and to the possibility of simulating the agents' behavior, should describe well also how the algorithm performs in real experiments.

BACKGROUND AND OBJECTIVES: During Twin-to-Twin Transfusion Syndrome (TTTS), abnormal vascular anastomoses in the monochorionic placenta can produce uneven blood flow between the fetuses. In the current practice, this syndrome is surgically treated by closing the abnormal connections using laser ablation. Surgeons commonly use the inter-fetal membrane as a reference. Limited field of view, low fetoscopic image quality and high inter-subject variability make the membrane identification a challenging task. However, currently available tools are not optimal for automatic membrane segmentation in fetoscopic videos, due to membrane texture homogeneity and high illumination variability. METHODS: To tackle these challenges, we present a new deep-learning framework for inter-fetal membrane segmentation on in-vivo fetoscopic videos. The framework enhances existing architectures by (i) encoding a novel (instance-normalized) dense block, invariant to illumination changes, that extracts spatio-temporal features to enforce pixel connectivity in time, and (ii) relying on an adversarial training, which constrains macro appearance. RESULTS: We performed a comprehensive validation using 20 different videos (2000 frames) from 20 different surgeries, achieving a mean Dice Similarity Coefficient of 0.8780±0.1383. CONCLUSIONS: The proposed framework has great potential to positively impact the actual surgical practice for TTTS treatment, allowing the implementation of surgical guidance systems that can enhance context awareness and potentially lower the duration of the surgeries.

Surgical planning of percutaneous interventions has a crucial role to guarantee the success of minimally invasive surgeries. In the last decades, many methods have been proposed to reduce clinician work load related to the planning phase and to augment the information used in the definition of the optimal trajectory. In this survey, we include 113 articles related to computer assisted planning (CAP) methods and validations obtained from a systematic search on three databases. First, a general formulation of the problem is presented, independently from the surgical field involved, and the key steps involved in the development of a CAP solution are detailed. Secondly, we categorized the articles based on the main surgical applications, which have been object of study and we categorize them based on the type of assistance provided to the end-user.

Learning manipulation skills from open surgery provides more flexible access to the organ targets in the abdomen cavity and this could make the surgical robot working in a highly intelligent and friendly manner. Teaching by demonstration (TbD) is capable of transferring the manipulation skills from human to humanoid robots by employing active learning of multiple demonstrated tasks. This work aims to transfer motion skills from multiple human demonstrations in open surgery to robot manipulators in robot-assisted minimally invasive surgery (RA-MIS) by using TbD. However, the kinematic constraint should be respected during the performing of the learned skills by using a robot for minimally invasive surgery. In this article, we propose a novel methodology by integrating the cognitive learning techniques and the developed control techniques, allowing the robot to be highly intelligent to learn senior surgeons' skills and to perform the learned surgical operations in semiautonomous surgery in the future. Finally, experiments are performed to verify the efficiency of the proposed strategy, and the results demonstrate the ability of the system to transfer human manipulation skills to a robot in RA-MIS and also shows that the remote center of motion (RCM) constraint can be guaranteed simultaneously. Note to Practitioners-This article is inspired by limited access to the manipulation of laparoscopic surgery under a kinematic constraint at the point of incision. Current commercial surgical robots are mostly operated by teleoperation, which is representing less autonomy on surgery. Assisting and enhancing the surgeon's performance by increasing the autonomy of surgical robots has fundamental importance. The technique of teaching by demonstration (TbD) is capable of transferring the manipulation skills from human to humanoid robots by employing active learning of multiple demonstrated tasks. With the improved ability to interact with humans, such as flexibility and compliance, the new generation of serial robots becomes more and more popular in nonclinical research. Thus, advanced control strategies are required by integrating cognitive functions and learning techniques into the processes of surgical operation between robots, surgeon, and minimally invasive surgery (MIS). In this article, we propose a novel methodology to model the manipulation skill from multiple demonstrations and execute the learned operations in robot-assisted minimally invasive surgery (RA-MIS) by using a decoupled controller to respect the remote center of motion (RCM) constraint exploiting the redundancy of the robot. The developed control scheme has the following functionalities: 1) it enables the 3-D manipulation skill modeling after multiple demonstrations of the surgical tasks in open surgery by integrating dynamic time warping (DTW) and Gaussian mixture model (GMM)-based dynamic movement primitive (DMP) and 2) it maintains the RCM constraint in a smaller safe area while performing the learned operation in RA-MIS. The developed control strategy can also be potentially used in other industrial applications with a similar scenario.

Path planning algorithms for steerable needles in medical applications must guarantee the anatomical obstacle avoidance, reduce the insertion length, and ensure the compliance with the needle kinematics. The majority of the solutions from the literature focus either on fast computation or on path optimality, the former at the expense of suboptimal paths, the latter by making unbearable the computation in case of a high-dimensional workspace. In this article, we implement a three-dimensional path planner for neurosurgical applications, which keeps the computational cost consistent with standard preoperative planning algorithms and fine-tunes the estimated pathways in accordance to multiple optimization objectives. From a user-defined entry point, our method confines a sample-based path search within a subsection of the original workspace considering the degree of curvature admitted by the needle. An evolutionary optimization procedure is used to maximize the obstacle avoidance and reduce the insertion length. The pool of optimized solutions is examined through a cost function to determine the best path. Simulations on one dataset showed the ability of the planner to save time and overcome the state of the art in terms of obstacle avoidance, insertion length, and probability of failure, proving this algorithm as a valid planning method for complex environments.

Objective: In the last years, Robot-assisted partial nephrectomy (RAPN) is establishing as elected treatment for renal cell carcinoma (RCC). Reduced field of view, field occlusions by surgical tools, and reduced maneuverability may potentially cause accidents, such as unwanted vessel resection with consequent bleeding. Surgical Data Science (SDS) can provide effective context-aware tools for supporting surgeons. However, currently no tools have been exploited for automatic vessels segmentation from nephrectomy laparoscopic videos. Herein, we propose a new approach based on adversarial Fully Convolutional Neural Networks (FCNNs) to kidney vessel segmentation from nephrectomy laparoscopic vision. Methods: The proposed approach enhances existing segmentation framework by (i) encoding 3D kernels for spatio-temporal features extraction to enforce pixel connectivity in time, and (ii) perform training in adversarial fashion, which constrains vessels shape. Results: We performed a preliminary study using 8 different RAPN videos (1871 frames), the first in the field, achieving a median Dice Similarity Coefficient of 71.76%. Conclusions: Results showed that the proposed approach could be a valuable solution with a view to assist surgeon during RAPN.

U reteroscopy is becoming the first surgical treatment option for the majority of urinary affections. This procedure is performed using an endoscope which provides the surgeon with the visual information necessary to navigate inside the urinary tract. Having in mind the development of surgical assistance systems, that could enhance the performance of surgeon, the task of lumen segmentation is a fundamental part since this is the visual reference which marks the path that the endoscope should follow. This is something that has not been analyzed in ureteroscopy data before. However, this task presents several challenges given the image quality and the conditions itself of ureteroscopy procedures. In this paper, we study the implementation of a Deep Neural Network which exploits the advantage of residual units in an architecture based on U-Net. For the training of these networks, we analyze the use of two different color spaces: gray-scale and RGB data images. We found that training on gray-scale images gives the best results obtaining mean values of Dice Score, Precision, and Recall of 0.73, 0.58, and 0.92 respectively. The results obtained shows that the use of residual U-Net could be a suitable model for further development for a computer-aided system for navigation and guidance through the urinary system.

The complexity of surgical operation can be released significantly if surgical robots can learn the manipulation skills by imitation from complex tasks demonstrations such as puncture, suturing, and knotting, etc.. This paper proposes a reinforcement learning algorithm based manipulation skill transferring technique for robot-assisted Minimally Invasive Surgery by Teaching by Demonstration. It employed Gaussian mixture model and Gaussian mixture Regression based dynamic movement primitive to model the high-dimensional human-like manipulation skill after multiple demonstrations. Furthermore, this approach fascinates the learning and trial phase performed offline, which reduces the risks and cost for the practical surgical operation. Finally, it is demonstrated by transferring manipulation skills for reaching and puncture using a KUKA LWR4+ robot in a lab setup environment. The results show the effectiveness of the proposed approach for modelling and learning of human manipulation skill.

In this paper, an Internet of Things-based human-robot collaborative control scheme is developed in Robot-assisted Minimally Invasive Surgery scenario. A hierarchical operational space formulation is designed to exploit the redundancies of the 7-DoFs redundant manipulator to handle multiple operational tasks based on their priority levels, such as guaranteeing a remote center of motion constraint and avoiding collision with a swivel motion without influencing the undergoing surgical operation. Furthermore, the concept of the Internet of Robotic Things is exploited to facilitate the best action of the robot in human-robot interaction. Instead of utilizing compliant swivel motion, HTC VIVE PRO controllers, used as the Internet of Things technology, is adopted to detect the collision. A virtual force is applied to the robot elbow, enabling a smooth swivel motion for human-robot interaction. The effectiveness of the proposed strategy is validated using experiments performed on a patient phantom in a lab setup environment, with a KUKA LWR4+ slave robot and a SIGMA 7 master manipulator. By comparison with previous works, the results show improved performances in terms of the accuracy of the RCM constraint and surgical tip.

In this paper, a bilateral teleoperation control of a serial robot manipulator, which guarantees a Remote Center of Motion (RCM) constraint in its kinematic level, is developed. A two-layered approach based on the energy tank model is proposed to achieve haptic feedback on the end effector with a pedal switch. The redundancy of the manipulator is exploited to maintain the RCM constraint using the decoupled Cartesian Admittance Control. Transparency and stability of the proposed bilateral teleoperation are demonstrated using a KUKA LWR4+ serial robot and a Sigma 7 haptic manipulator with an RCM constraint in augmented reality. The results prove that the control can achieve not only the bilateral teleoperation but also maintain the RCM constraint.

Path planning algorithms for steerable catheters, must guarantee anatomical obstacles avoidance, reduce the insertion length and ensure the compliance with needle kinematics. The majority of the solutions in literature focuses on graph based or sampling based methods, both limited by the impossibility to directly obtain smooth trajectories. In this work we formulate the path planning problem as a reinforcement learning problem and show that the trajectory planning model, generated from the training, can provide the user with optimal trajectories in terms of obstacle clearance and kinematic constraints. We obtain 2D and 3D environments from MRI images processing and we implement a GA3C algorithm to create a path planning model, able to generalize on different patients anatomies. The curvilinear trajectories obtained from the model in 2D and 3D environments are compared to the ones obtained by A* and RRT* algorithms. Our method achieves state-of-the-art performances in terms of obstacle avoidance, trajectory smoothness and computational time proving this algorithm as valid planning method for complex environments.

In this paper, an improved motion planning scheme is proposed for surgical robot control with multiple active constraints, including joint constraints, joint velocity constraints and remote center of motion constraints. It introduces an improved recurrent neural network (RNN) to optimize the online motion planning respect to multiple constraints. The demonstrated surgical operation trajectory is derived using teaching by demonstration. An improved motion planning scheme using the novel recurrent neural network is then designed to achieve the accurate task tracking under the multiple constraints. The general quadratic performance index is adopted to represent the constraints. Finally, the effectiveness of the proposed algorithm is demonstrated using KUKA LWR4+ robot in a lab setup environment.

Robot-Assisted systems for Minimally Invasive Surgery enhance the surgeon capability, however, direct control over both the surgical tools and the endoscope results in an increased workload that leads to longer operation times. This work investigates the introduction of SCAN (System for Camera Autonomous Navigation) to overcome this limitation. An experimental study involving 12 participants was carried out with the da Vinci Research Kit. Each user tested two novel camera control modalities, autonomous and semi-autonomous, as well as the current manual control of the camera, while carrying out a dry-lab task. Among the camera control modalities, the autonomous navigation achieved better objective performances and the highest user confidence. Moreover, the autonomous control (along with the semi-autonomous one) was able to optimize some metrics related to the robotic surgery workflow.

The objective of this paper is to present a personalisable human ergonomics framework that integrates a method for real-time identification of a human model and an ergonomics monitoring function. The human model is based on a floating base structure and on a Statically Equivalent Serial Chain (SESC) model used for the estimation of the whole-body centre of Mass (CoM). A recursive linear regression algorithm (i.e., Kalman filter) is developed to achieve the online identification of the SESC parameters. A visual feedback provides a minimum set of suggested human poses to speed up the identification process, while enhancing the model accuracy based on a convergence value. The online ergonomics monitoring function computes and displays the overloading effects on body joints in heavy lifting tasks. The overloading joint torques are calculated based on the displacement of the Center of Pressure (CoP) between the measured one and the estimated one. Unlike our previous work, the entire process, from the model identification (personalisation) to ergonomics monitoring, is performed in real-time. We evaluated the efficacy of the proposed method through human experiments during model identification and load lifting tasks. Results demonstrate the high exploitation potential of the framework in industrial settings, due to its fast personalisation and ergonomics monitoring capacity.

Full teleoperation of mobile robots during the execution of complex tasks not only demands high cognitive and physical effort but also generates less optimal trajectories compared to autonomous controllers. However, the use of the latter in cluttered and dynamically varying environments is still an open and challenging topic. This is due to several factors such as sensory measurement failures and rapid changes in task requirements. Shared-control approaches have been introduced to overcome these issues. However, these either present a strong decoupling that makes them still sensitive to unexpected events, or highly complex interfaces only accessible to expert users. In this work, we focus on the development of a novel and intuitive shared-control framework for target detection and control of mobile robots. The proposed framework merges the information coming from a teleoperation device with a stochastic evaluation of the desired goal to generate autonomous trajectories while keeping a human-in-control approach. This allows the operator to react in case of goal changes, sensor failures, or unexpected disturbances. The proposed approach is validated through several experiments both in simulation and in a real environment where the users try to reach a chosen goal in the presence of obstacles and unexpected disturbances. Operators receive both visual feedback of the environment and voice feedback of the goal estimation status while teleoperating a mobile robot through a control-pad. Results of the proposed method are compared to pure teleoperation proving a better time-efficiency and easiness-of-use of the presented approach.

For the time varying optimization problem, the tracking error cannot converge to zero at the finite time because of the optimal solution changing over time. This paper proposes a novel varying parameter recurrent neural network (VPRNN) based hierarchical optimization of a 7-DoF surgical manipulator for Robot-Assisted Minimally Invasive Surgery (RAMIS), which guarantees task tracking, Remote Center of Motion (RCM) and manipulability index optimization. A theoretically grounded hierarchical optimization framework based is introduced to control multiple tasks based on their priority. Finally, the effectiveness of the proposed control strategy is demonstrated with both simulation and experimental results. The results show that the proposed VPRNN-based method can optimal three tasks at the same time and have better performance than previous work.

The objective of this paper is to present a mathematical tool for real-time tracking of whole-body compressive forces induced by external physical solicitations. This tool extends and enriches our recently introduced ergonomics monitoring system to asses the level of risk associated with human physical activities in human-robot collaboration contexts. The methods developed so far only considered the effect of the external loads on joint torque variations. However, even for negligible values of the joint torque overloadings (e.g., in singular configurations), the effect of compressive forces, defined by the internal/pushing forces among body links, can be significant. Accordingly, we propose the joint compressive forces as an additional real-time index for the assessment of human ergonomics. First, a simulation study is performed to validate the method. Then, follows a laboratory study on five subjects to compare the trend of the joint compressive forces with muscle activities. Results demonstrate the significance of the proposed index in the development of a comprehensive human ergonomics monitoring framework. Based on such a framework, robotic strategies as well as feedback interfaces can be employed to guide and optimise the human movement toward more convenient body configurations thus avoiding pain and consequent injuries.

Autonomous goal detection and navigation control of mobile robots in remote environments can help to unload human operators from simple, monotonous tasks allowing them to focus on more cognitively stimulating actions. This can result in better task performances, while creating user-interfaces that are understandable by non-experts. However, full autonomy in unpredictable and dynamically changing environments is still far from becoming a reality. Thus, teleoperated systems integrating the supervisory role and instantaneous decision-making capacity of humans are still required for fast and reliable robotic operations. This work presents a novel shared-autonomy framework for goal detection and navigation control of mobile manipulators. The controller exploits human-gaze information to estimate the desired goal. This is used together with control-pad data to predict user intention, and to activate the autonomous control for executing a target task. Using the control-pad device, a user can react to unexpected disturbances and halt the autonomous mode at any time. By releasing the control-pad device (e.g., after avoiding an instantaneous obstacle) the controller smoothly switches back to the autonomous mode and navigates the robot towards the target. Experiments for reaching a target goal in the presence of unknown obstacles are carried out to evaluate the performance of the proposed shared-autonomy framework over seven subjects. The results prove the accuracy, time-efficiency, and ease-of-use of the presented shared-autonomy control framework.

Kinematic trajectories recorded from surgical robots contain information about surgical gestures and potentially encode cues about surgeon's skill levels. Automatic segmentation of these trajectories into meaningful action units could help to develop new metrics for surgical skill assessment as well as to simplify surgical automation. State-of-the-art methods for action recognition relied on manual labelling of large datasets, which is time consuming and error prone. Unsupervised methods have been developed to overcome these limitations. However, they often rely on tedious parameter tuning and perform less well than supervised approaches, especially on data with high variability such as surgical trajectories. Hence, the potential of weak supervision could be to improve unsupervised learning while avoiding manual annotation of large datasets. In this paper, we used at a minimum one expert demonstration and its ground truth annotations to generate an appropriate initialization for a GMM-based algorithm for gesture recognition. We showed on real surgical demonstrations that the latter significantly outperforms standard task-agnostic initialization methods. We also demonstrated how to improve the recognition accuracy further by redefining the actions and optimising the inputs.

Among the numerous risk factors associated to work-related musculoskeletal disorders (WMSD), repetitive and monotonous movements with light-weight tools are one of the most frequently cited. Such tasks may indeed result in the excessive accumulation of local muscle fatigue, causing severe injuries in human joints. Accordingly, this paper proposes a new whole-body fatigue model to evaluate the cumulative effect of the overloading torque induced on the joints over time by light payloads. The proposed model is then integrated into a human-robot collaboration (HRC) framework to set the timing of a body posture optimisation procedure guided by the robot assistance, by the time fatigue overcomes a threshold in any joint. Our overloading fatigue model is based on an estimation method we developed in a previous work, to monitor joint torque variations due to external forces in real-time. To account for individuals' different perception of fatigue, the fatigue ratio parameter in the model is computed experimentally for each subject. The proposed model is first studied on ten subjects by means of an electromyography analysis. Next, its performance is assessed in a painting task and finally evaluated within the HRC framework, which is proved to be able to reduce the risk of injuries caused by excessive fatigue accumulation.

Transcatheter aortic valve implantation (TAVI) is a minimally invasive surgical technique to treat aortic heart valve diseases. According to current clinical guidelines, the implanted prosthetic valve replacing the native one is selected based on pre-operative size assessment of the aortic annulus through different imaging techniques. That very often leads to suboptimal device selection resulting in major complications, such as prosthetic valve leakage or interruption of the cardiac electrical signal. In this paper, we propose a new, intra-operative approach to determine the diameter of theaortic annulus exploiting intra-balloon pressure and volume data, acquired from a robotised valvuloplasty balloon catheter. An inflation device, capable of collecting real-time intra-balloon pressure and volume data, was designed and interfaced with a commercially available valvuloplasty balloon catheter. A sizing algorithm allowing to precisely estimate the annular diameter was integrated. The algorithm relies on a characterised analytical model of the balloon free inflation and an iterative method based on linear regression. In vitro tests were performed on idealised aortic phantoms. Experimental results show that pressure-volume data can be used to determine annular diameters bigger than the unstretched diameter of the balloon catheter. For these cases, the proposed approach exhibited good precision (maximum average error 0.93%) and good repeatability (maximum standard deviation ±0.11 mm).

This paper proposes a manipulability optimization control of a 7-DoF robot manipulator for Robot-Assisted Minimally Invasive Surgery (RAMIS), which at the same time guarantees a Remote Center of Motion (RCM). The first degree of redundancy of the manipulator is used to achieve an RCM constraint, the second one is adopted for manipulability optimization. A hierarchical operational space formulation is introduced to integrate all the control components, including a Cartesian compliance control involving the main surgical task, a first null-space controller for the RCM constraint, and a second null-space controller for manipulability optimization. Experiments with virtual surgical tasks, in an augmented reality environment, were performed to validate the proposed control strategy using the KUKA LWR 4 +. The results demonstrate that end-effector accuracy and RCM constraint can be guaranteed, along with improving the manipulability of the surgical tip.

Robot-assisted laparoscopic prostatectomy (RALP) is a treatment for prostate cancer that involves complete or nerve sparing removal prostate tissue that contains cancer. After removal the bladder neck is successively sutured directly with the urethra. The procedure is called urethrovesical anastomosis and is one of the most dexterity demanding tasks during RALP. Two suturing instruments and a pair of needles are used in combination to perform a running stitch during urethrovesical anastomosis. While robotic instruments provide enhanced dexterity to perform the anastomosis, it is still highly challenging and difficult to learn. In this paper, we presents a vision-guided needle grasping method for automatically grasping the needle that has been inserted into the patient prior to anastomosis. We aim to automatically grasp the suturing needle in a position that avoids hand-offs and immediately enables the start of suturing. The full grasping process can be broken down into: a needle detection algorithm; an approach phase where the surgical tool moves closer to the needle based on visual feedback; and a grasping phase through path planning based on observed surgical practice. Our experimental results show examples of successful autonomous grasping that has the potential to simplify and decrease the operational time in RALP by assisting a small component of urethrovesical anastomosis.

Hands-on training is an indispensable part of surgical practice. As the tools used in the operating room become more intricate, the demand for efficient training methods increases. This work proposes a robotic assistance-as-needed method for training with surgical teleoperated robots. The method adapts the intensity of the assistance according to the trainee's current and past performance while gradually increasing the level of control of the trainee as the training progresses. The work includes an experiment comprising 160 acquisition sessions from 16 novice subjects performing a bimanual teleoperated exercise with a da Vinci Research Kit surgical console. Results capture the subtleties in the task's learning curve with and without robotic assistance and hint at the potential of robotic assistance for complex visuomotor training. Although robotic assistance for motor learning has received mixed results that range from beneficial to detrimental effects, this study shows such assistance may increase the rate of learning of certain skills in complex motor tasks.

In this paper, an automatic planner for minimally invasive neurosurgery is presented. The solution provides the neurosurgeon with the best path to connect a user-defined entry point with a target in accordance with a specific cost function. The approach guarantees the avoidance of obstacles which can be found along the insertion pathway. The method is tailored to the EDEN2020* programmable bevel-tip needle, a multisegment steerable probe intended to be used to perform drug delivery for the treatment of glioblastomas. A sample-based heuristic search inspired by the BIT* algorithm is used to define the asymptotically-optimal solution in terms of path length, followed by a smoothing phase to meet the required kinematic constraints of the needle. To account for inaccuracies in catheter modeling, which could determine unexpected control errors over the insertion procedure, an uncertainty margin is defined in order to increase the algorithm's safety. The feasibility of the proposed solution was demonstrated by testing the method in simulated neurosurgical scenarios with different degrees of obstacle occupancy and against other sample-based algorithms present in literature: RRT, RRT* and an enhanced version of the RRT-Connect.

In this paper, a teleoperation control of a 7-DoF robot manipulator for Minimally Invasive Surgery (MIS), which guarantees a safety-enhanced compliant behavior in the null space, is described. The redundancy of the manipulator is exploited to provide a flexible workspace for nurses or other staff (assisting physicians, patient support). The issue with safety and accurate surgical task execution may arise in the presence of human-robot interaction. Based on the implemented impedance control of tele-operated MIS tasks, a safety enhanced constraint is applied on the compliant null space motion. At the same time, the control approach integrates an adaptive fuzzy compensator to guarantee the accuracy of the surgical tasks during the uncertain human-robot interaction. The performance of the proposed algorithm is verified with virtual surgical tasks. The results showed that the compliant null space motion is constrained in a safe area, and also that the accuracy of tool tip is improved, providing a flexible and safe collaborative behavior in the null space for human-robot interaction during surgical tasks.

Robust representation of actions and its sequences for complex robotic tasks would transform robot's understanding to execute robotic tasks efficiently. The challenge is to understand action sequences for highly unstructured environments and to represent and construct action and action sequences. In this manuscript, we present a review of literature dealing with representation of action and action sequences for robot task planning and execution. The methodological review was conducted using Google Scholar and IEEE Xplore, searching the specific keywords. This manuscript gives an overview of current approaches for representing action sequences in robotics. We propose a classification of different methodologies used for action sequences representation and describe the most important aspects of the reviewed publications. This review allows the reader to understand several options that do exist in the research community, to represent and deploy such action representations in real robots.

Surgical workflow modeling is becoming increasingly useful to train surgical residents for complex surgical procedures. Rule-based surgical workflows have shown to be useful to create context-aware systems. However, manually constructing production rules is a time-intensive and laborious task. With the expansion of new technologies, large video archive can be created and annotated exploiting and storing the experts knowledge. This paper presents a prototypical study of automatic generation of production rules, in the Horn-clause, using the First Order Inductive Learner (FOIL) algorithm applied to annotated surgical videos of Thoracentesis procedure and of its feasibility to use in context-aware system framework. The algorithm was able to learn 18 rules for surgical workflow model with 0.88 precision, and 0.94 F1 score on the standard video annotation data, representing entities of the surgical workflow, which was used to retrieve contextual information on Thoracentesis workflow for its application to surgical training.

Prostate cancer is the most common non-cutaneous cancer in America. Tumor detection involves non-invasive screening tests, but positive results must be confirmed by a prostate biopsy. About twelve random samples are obtained during the biopsy, which is a systematic procedure traditionally performed with trans-rectal ultrasound (TRUS) guidance to determine prostate location. Recently, methods of fusion between TRUS and preoperative MRI have been introduced in order to perform targeted biopsies aimed to reduce the number of samples to few suspicious areas. Since the TRUS displaces the prostate during the procedure, the preoperative MRI does not match patient anatomy. Therefore, complex MRI deformation algorithms are needed. However, despite the substantial increase in complexity and cost, there is no strong evidence that the TRUS-MRI fusion actually improves accuracy and surgical outcomes.

Haptically enabled hands-on or tele-operated surgical robotic systems provide a unique opportunity to integrate pre- and intra-operative information into physical actions through active constraints (also known as virtual fixtures). In many surgical procedures, including cardiac interventions, where physiological motion complicates tissue manipulation, dynamic active constraints can improve the performance of the intervention in terms of safety and accuracy. The non-energy-storing class of dynamic guidance constraints attempt to assist the clinician in following a reference path, while guaranteeing that the control system will not generate undesired motion due to stored potential energy. An important aspect that has not received much attention from the researchers is that while these methods help increase the performance, they should by no means distract the user systematically. In this paper, a viscosity-based dynamic guidance constraint is introduced that continuously redirects the tool's motion towards the reference path. The proportionality and continuity of generated forces make the method less distracting and subjectively appealing. The performance is validated and compared with two existing non-energy-storing methods through extensive experimentation.

Surgeons can benefit from the cooperation with a robotic assistant during the repetitive execution of precise targeting tasks on soft tissues, such as during brain cortex stimulation procedures in open-skull neurosurgery. Position-based force-to-motion control schemes may not be suitable solution to provide the manipulator with the high compliance desirable during guidance along wide trajectories. A new torque controller with non-linear force feedback (FFE) is presented to provide augmented haptic perception to the operator, during the instrument's placement on the tissue. The FFE controller was experimentally validated with a pool of non-expert users using brain-mimicking gelatin phantoms (8%-16% concentration). Besides providing hand tremor rejection for a stable holding of the tool, the FFE controller was proved to allow for a safer tissue contact with respect to both robotic assistance without force feedback and freehand executions (50% and 75% reduction of the indentation depth, respectively). Future work will address the evaluation of the safety features of the FFE controller with expert surgeons on a realistic brain phantom, also accounting for unpredictable tissue motions as during seizures due to cortex stimulation.

In present days, the number of application in which robots and users share the same workspace is increasing, as long as the need of cooperation between them. To achieve a smooth cooperation, in particular in surgical applications, the robot needs to timely change its behavior to adapt to the needs of the user. In this work, a simplified scenario for neurosurgery was defined in which the user interacts with the robot through a Graphical User Interface (GUI) and by touching the robot links and, based to those events and on the current status, different control modes are enabled in the high level controller we developed, such as autonomous, cooperative and teleoperation. Experiments were performed to measure the performances and safety of the developed high level controller in handling the transitions between two states by checking the continuity of data from the robot and from an external measurement system. Results proved that the trajectories of the end effector and links during the switching phase are continuous and thus the modular high level controller developed switches control safely without undesired deviation from desired course.

Robotic systems have been introduced in surgery to increase the intervention accuracy. In this framework, the ROBOCAST system is an optically controlled multi-robot chain aimed at enhancing the accuracy of surgical probe insertion during keyhole neurosurgery procedures. The system is composed by three robots, connected as a multiple kinematic chain (serial, parallel and linear) totaling 13 degrees of freedom (DoFs) and is it is used to automatically align the probe onto the desired trajectory. This paper presents an iterative approach for aligning the surgical probe with the planned target pose, reducing both the translation and the rotation errors. An experimental protocol was designed, in order to assess the system performances in terms of residual targeting errors and convergence ratio. The proposed targeting procedures allows obtaining (0.06 ± 0.02) mm and (0.8 ± 0.2) × 10-3rad as residual median errors, thus satisfying the operational requirements (1 mm). The performances proved to be independent upon the robots calibration accuracy.