Videos are a powerful media to learn activities through guided physical training such as surgery, especially when they are produced following human learning models and not as "how-to" videos. However, their success greatly depends on how they are integrated into the extensive curricula of domains where learning occurs through guided practice. In this work, we investigate the impact of integrating video as a learning tool into the learning curricula of surgery. We created a pedagogical video on surgical hysterectomy through a model based on the Conceptual Fields theory (Vergnaud) and performed two rounds of interviews with seven medical residents, who watched the video freely during their residency in gynecology-obstetrics as they trained with experts. We find that videos can complement guided physical training, as they can provide the rationale behind expert action, something that is difficult to explicit during guided training. Still, their linear and static nature limits their integration as true adjuncts. We discuss our vision of moving towards interactive videos created with an ontological approach, developed in a workshop with four expert surgeons, which involves the ability to navigate through levels of information and layers of representations, so that experts can represent information to learners according to pedagogical models that complement their complex and extensive learning curricula.

One of the major functions brought by robots in Minimally Invasive Surgery is endoscope holding. This consists, for the user, in placing the camera at a desired location which the robot will maintain still once he/she releases it. This behavior is usually achieved with rigid position servoing, leading to possibly high forces generated and safety issues. Model-based weight compensation is an alternative solution. However, endoscopic cameras' weight is difficult to model as their gravity parameters can change during the same surgery. In this paper, an algorithm is presented as an option to cope with this variability in the gravity model without using rigid position servoing. The surgeon first positions the camera in a comanipulation mode (gravity compensation). When he/she releases the camera, if the gravity model is not accurate, the endoscope presents a drift. In this case, a controller brings the endoscope back to its release position by combining low gain position control and model adaptation. Once stabilized, the system is switched back to a zero-stiffness mode. Two in-vitro experiments were performed in which a user manipulates an endoscope whose configuration of mass is changed. In one case, the mass in the gravity model was set to half of the actual one. In the second case, a variable weight was attached to the endoscope. The algorithm successfully updated the model for each experiment reducing position errors by 95% and 57%, respectively.

Robot-Assisted Laparoscopic Surgery (RALS) is now prevalent in operating rooms. This situation requires future surgeons to learn Classic Laparoscopic Surgery (CLS) and RALS simultaneously. Therefore, along with the investigation of the differences in performance between the two techniques, it is essential to study the impact of training in RALS on the skills mastered in CLS. In this article, we study comanipulated RALS (Co-RALS), one of the two designs for RALS, where the human and the robot share the execution of the task. We use a rarely used in Human–Robot Interaction measuring tool: gaze tracking and time recording to measure for the acquisition of skills in CLS when training in Co-RALS or in CLS and time recording to compare the learning curves between Co-RALS and CLS. These metrics allow us to observe differences in Co-RALS and CLS. Training in Co-RALS develops slightly better but not significantly better hand–eye coordination skills and significantly better timewise performance compared with training in CLS alone. Co-RALS enhances timewise performance in laparoscopic surgery on specific types of tasks that require precision rather than depth perception skills compared with CLS. The results obtained enable us to further define the Human–Robot Interaction quality in Co-RALS.

Classic Minimally Invasive Surgery (MIS) is an ergonomic burden for assistants and surgeons. The former need to adopt uncomfortable positions for hours while holding a camera to track the latter’s gestures inside the patient. This incurs assistant’s muscle fatigue which can lead to tremor or drift of the video feedback. A backdrivable robotic holder can be attached to this device in order to compensate its weight. This allows the user to place the camera at a desired position which the robot will steadily keep once he/she releases it. However, endoscopic cameras present difficult-to-model accessories whose gravity parameters can change during the same surgery. If these changes are not foreseen by the gravity model of the robot this results in a fall of the endoscope each time it is released. Therefore, it is desired to firstly detect if there is a fall in order to be able to correct it. In this article a fall detection method for a comanipulated robotic endoscope holder is proposed. It evaluates smoothness of the robot end effector trajectory to identify whether the user manipulates the instrument or it has been released and poorly compensated. An experiment was carried out with 10 subjects where 240 releases of the endoscope were performed while it was poorly compensated. The algorithm succeeded to detect the falls with sensitivity up to 99.17%.

Surgery may involve precise instrument tip positioning in a minimally invasive way. During these operations, the instrument is inserted in the body through an orifice. The movements of the instrument are constrained by interaction forces arising at the orifice level. The physical constraints may drastically vary depending on the patient’s anatomy. This introduces uncertainties that challenge the positioning task for a robot. Indeed, it raises an antagonism: On one side, the required precision appeals for a rigid behavior. On the other side, forces applied at the entry point should be limited, which requires softness. In this paper we choose to minimize forces at the orifice by using a passive ball joint wrist to manipulate the instrument. From a control perspective, this leads to consider the task as a 3 DOF wrist center positioning problem, whose softness can be achieved through conventional low impedance control. However, positioning the wrist center, even with a high static precision, does not allow to achieve a high precision of the instrument tip positioning when the orifice behavior is not known. To cope with this problem, we implement a controller that servos the tip position by commanding the wrist position. In order to deal with uncertainties, we exploit an adaptive control scheme that identifies in real-time the unknown mapping between the wrist velocity and the tip velocity. Both simulations and in vitro experimental results show the efficiency of the control law.

Robotic-assisted Minimally Invasive Surgery (MIS) is adopted more and more as it overcomes the shortcomings of classic MIS for surgeons while keeping the benefits of small incisions for patients. However, introducing new technology oftentimes affects the work of skilled practitioners. Our goals are to investigate the impacts of telemanipulated surgical robots on the work practices of surgical teams and to understand their cause. We conducted a field study observing 21 surgeries, conducting 12 interviews and performing 3 data validation sessions with surgeons. Using Thematic Analysis, we find that physically separating surgeons from their teams makes them more autonomous, shifts their use of perceptual senses, and turns the surgeon's assistant into the robot's assistant. We open design opportunities for the HCI field by questioning the telemanipulated approach and discussing alternatives that keep surgeons on the surgical field.

The comanipulation paradigm, in which a user and a robot simultaneously hold a tool, allows for gesture guidance. In particular, virtual fixtures, which are geometrical constraints imposed to the tool by the robot, have received great interest in the domain of surgical applications. So far, this concept has been implemented in the context of open surgery. This paper explores the application of virtual fixtures for minimally invasive surgery, in which the tool is inserted in the patient through a fulcrum. Here, a key issue is to return to the surgeon forces that are virtually applied at the instrument distal tip, while the robot is physically attached to the instrument proximal handle. To this aim, two approaches are investigated. A first approach consists of applying a full wrench at the proximal end of the instrument that is equal to the wrench constituted by a pure force applied to the instrument's distal tip. A second approach consists of applying a pure force to the instrument proximal end, thanks to a lever model about the fulcrum. The two approaches are compared through experiments, during which naive subjects blindly perform virtual object palpation and robot-guided movements. During experiments, indicators involving motion and force analysis are computed. The user capacity to distinguish between several virtual objects is evaluated as well. Although drastically different, the two approaches provide assistance with a similar level of efficiency.

A novel robotic device, aimed at assisting a urologist in performing prostate biopsies guided by an endorectal ultrasound probe, is described. The paper describes the robot kinematics and the actuation system. The actuation system combines electromagnetic brakes, balancing springs, and electrical motors with cable transmissions. The robot supports two modes of operation: the free mode, where the entire control of the probe movements is left to the urologist, and the blocked mode, where the robot precisely maintains the probe at a given position and orientation with respect to the prostate. For the blocked mode, the set of specifications is antagonistic: firstly, a security constraint requires a low robot stiffness to allow to compliantly adapt to potential movements from the patient; secondly, a precision constraint requires a high robot stiffness in order to maintain the position and orientation of the probe in the presence of unmodeled external forces, when the robot is switched from the free mode to the blocked mode. A control strategy is developed to obtain this behavior. It combines an inner impedance controller with a relatively low stiffness and an outer intelligent position integrator that operates only during a limited period of time, when switching from the free mode to the blocked mode. Both in vitro and in cadavero experimental results show the efficiency of this approach.

This paper describes results about the development of a repeatable and configurable robotic finger designed to pluck harp strings. Eventually, this device will be a tool to study string instruments in playing conditions. We use a classical robot with two degrees of freedom enhanced with silicone fingertips. The validation method requires a comparison with real harpist performance. A specific experimental setup using a high-speed camera combined with an accelerometer was carried out. It provides finger and string trajectories during the whole plucking action and the soundboard vibrations during the string oscillations. A set of vibrational features are then extracted from these signals to compare robotic finger to harpist plucking actions. These descriptors have been analyzed on six fingertips of various shapes and hardnesses. Results allow to select the optimal shape and hardness among the silicone fingertips according to vibrational features.

The possibility to provide an adequate task assistance using underactuated robots for human-robot tool comanipulation is investigated. This novel approach does not take into account any a priori knowledge about user depending parameters however optimizes the robot-user synergy, for instance during US breast examinations. Results show that the examination time can be reduced and a tendency for increasing scanning accuracy using underactuated robots.

In recent years, there has been an increasing interest in developing systems that couple a robotic device with an ultrasound imager. Applications range from automatic probe positioning to ultrasound image-based guidance of instruments. One issue in these systems is to determine, prior to the intervention, the localization of the probe with respect to the robot. Literature suggests using external localizers, but they add to the complexity of the system, and the resulting precision is usually not good due to the addition of errors in the kinematic chain. In this paper, we study the hand-eye calibration problem without using any additional localizers. A generic system consisting of a fixed probe observing an instrument manipulated by a robot is used. We first derive a simplified model for mapping the image of the instrument with its 3D location w.r.t. the probe and then propose a calibration procedure based on a minimization algorithm. Results show that although very simple models were used for the imaging device, the localization is quite precise, as it results in errors of less than 2 mm, which is enough of a number of ultrasound guided interventions.

This paper presents a robust visual servoing approach for automatic guidance of an instrument. The visual sensor is an ultrasound probe that observes an instrument which is inserted into the beating heart of a patient and manipulated by a robot. The present paper provides stability analysis, robustification of the control law and an in vivo experiment.

Minimally invasive surgery in combination with ultrasound (US) imaging imposes high demands on the sur geon’s hand-eye-coordination capabilities. A possible solution to reduce these requirements is minimally invasive robotic surgery in which the instrument is guided by visual servoing towards the goal defined by the surgeon in the US image. This approach requires robust tracking of the instrument in the US image sequences which is known to be difficult due to poor image quality. This paper presents computer vision algorithms and results of visual servoing experiments. Adaptive thresholding according to Otsu’s method allows to cope with large intensity variations of the instrument echo. Subsequently applied morphological operations suppress noise and echo artefacts. A fast labelling algorithm based on run length coding allows for realtime labelling of the regions. A heuristic exploiting region size and region velocity helps to overcome ambiguities. The overall computation time is less than 10 ms per frame on a standard PC. The tracking algorithm requires no information about texture and shape which are known to be very unreliable in US image sequences. Experimental results for different instrument materials (polyvinyl chloride, polyurethane, nylon, and plexiglas) are given, illustrating the performance of the proposed approach: when chosing the appropriate material the reconstructed trajectories are smooth and only few outliers occur. As a consequence, the visual servoing loop showed to be robust and stable.

Visual servoing is a possible solution to assist the surgeon in performing tasks under ultrasound (US) imaging. To this aim, a system was developed that allows the surgeon to select a desired instrument location on a US image. Then a robot is programmed to automatically move the instrument towards the selected location. This approach requires robust tracking of the instrument in the US image, together with modeling of the overall system and implementation of a visual servoing loop. This paper presents geometrical and kinematic models of the system, as well as the control loop design, which is validated through both numerical simulations, and results of in vitro experiments.

Force-feedback mechanisms have been designed to simplify and enhance the human-vehicle interface. The increase in secondary controls within vehicle cockpits has created a desire for a simpler, more efficient human-vehicle interface. By consolidating various controls into a single, haptic feedback control device, information can be transmitted to the operator, without requiring the driver's visual attention. In this work, the experimental closed loop torque control of electro-rheological fluids (ERF) based actuators for haptic application is performed. ERFs are liquids that respond mechanically to electric fields by changing their properties, such as viscosity and shear stress, electroactively. Using the electrically controlled rheological properties of ERFs, we developed actuators for haptic devices that can resist human operator forces in a controlled and tunable fashion. In this study, the ERF actuator analytical model is derived and experimentally verified and accurate closed loop torque control is experimentally achieved using a non-linear proportional integral controller with a feed-forward loop.