Fetoscopic Laser Coagulation (FLC) for Twin to Twin Transfusion Syndrome is a challenging intervention due to the working conditions: low quality images acquired from a 3 mm fetoscope inside a turbid liquid environment, local view of the placental surface, unstable surgical field and delicate tissue layers. FLC is based on locating, coagulating and reviewing anastomoses over the placenta's surface. The procedure demands the surgeons to generate a mental map of the placenta with the distribution of the anastomoses, maintaining, at the same time, precision in coagulation and protecting the placenta and amniotic sac from potential damages. This paper describes a teleoperated platform with a cognitive-based control that provides assistance to improve patient safety and surgery performance during fetoscope navigation, target re-location and coagulation processes. A comparative study between manual and teleoperated operation, executed in dry laboratory conditions, analyzes basic fetoscopic skills: fetoscope navigation and laser coagulation. Two exercises are proposed: first, fetoscope guidance and precise coagulation. Second, a resolved placenta (all anastomoses are indicated) to evaluate navigation, re-location and coagulation. The results are analyzed in terms of economy of movement, execution time, coagulation accuracy, amount of coagulated placental surface and risk of placenta puncture. In addition, new metrics, based on navigation and coagulation maps evaluate robotic performance. The results validate the developed platform, showing noticeable improvements in all the metrics.

The field of robotic Flexible Endoscopes (FEs) has progressed significantly, offering a promising solution to reduce patient discomfort. However, the limited autonomy of most robotic FEs results in non-intuitive and challenging manoeuvres, constraining their application in clinical settings. While previous studies have employed lumen tracking for autonomous navigation, they fail to adapt to the presence of obstructions and sharp turns when the endoscope faces the colon wall. In this work, we propose a Deep Reinforcement Learning (DRL)-based navigation strategy that eliminates the need for lumen tracking. However, the use of DRL methods poses safety risks as they do not account for potential hazards associated with the actions taken. To ensure safety, we exploit a Constrained Reinforcement Learning (CRL) method to restrict the policy in a predefined safety regime. Moreover, we present a model selection strategy that utilises Formal Verification (FV) to choose a policy that is entirely safe before deployment. We validate our approach in a virtual colonoscopy environment and report that out of the 300 trained policies, we could identify three policies that are entirely safe. Our work demonstrates that CRL, combined with model selection through FV, can improve the robustness and safety of robotic behaviour in surgical applications.

Increased demand for less invasive procedures has accelerated the adoption of Intraluminal Procedures (IP) and Endovascular Interventions (EI) performed through body lumens and vessels. As navigation through lumens and vessels is quite complex, interest grows to establish autonomous navigation techniques for IP and EI for reaching the target area. Current research efforts are directed toward increasing the Level of Autonomy (LoA) during the navigation phase. One key ingredient for autonomous navigation is Motion Planning (MP) techniques. This paper provides an overview of MP techniques categorizing them based on LoA. Our analysis investigates advances for the different clinical scenarios. Through a systematic literature analysis using the PRISMA method, the study summarizes relevant works and investigates the clinical aim, LoA, adopted MP techniques, and validation types. We identify the limitations of the corresponding MP methods and provide directions to improve the robustness of the algorithms in dynamic intraluminal environments. MP for IP and EI can be classified into four subgroups: node, sampling, optimization, and learning-based techniques, with a notable rise in learning-based approaches in recent years. One of the review's contributions is the identification of the limiting factors in IP and EI robotic systems hindering higher levels of autonomous navigation. In the future, navigation is bound to become more autonomous, placing the clinician in a supervisory position to improve control precision and reduce workload.

Flexible Endoscopes (FEs) for colonoscopy present several limitations due to their inherent complexity, resulting in patient discomfort and lack of intuitiveness for clinicians. Robotic FEs with autonomous control represent a viable solution to reduce the workload of endoscopists and the training time while improving the procedure outcome. Prior works on autonomous endoscope FE control use heuristic policies that limit their generalisation to the unstructured and highly deformable colon environment and require frequent human intervention. This work proposes an image-based FE control using Deep Reinforcement Learning, called Deep Visuomotor Control (DVC), to exhibit adaptive behaviour in convoluted sections of the colon. DVC learns a mapping between the images and the FE control signal. A first user study of 20 expert gastrointestinal endoscopists was carried out to compare their navigation performance with DVC using a realistic virtual simulator. The results indicate that DVC shows equivalent performance on several assessment parameters, being more safer. Moreover, a second user study with 20 novice users was performed to demonstrate easier human supervision compared to a state-of-the-art heuristic control policy. Seamless supervision of colonoscopy procedures would enable endoscopists to focus on the medical decision rather than on the control of FE.

Deep Reinforcement Learning (DRL) is a viable solution for automating repetitive surgical subtasks due to its ability to learn complex behaviours in a dynamic environment. This task automation could lead to reduced surgeon’s cognitive workload, increased precision in critical aspects of the surgery, and fewer patient-related complications. However, current DRL methods do not guarantee any safety criteria as they maximise cumulative rewards without considering the risks associated with the actions performed. Due to this limitation, the application of DRL in the safety-critical paradigm of robot-assisted Minimally Invasive Surgery (MIS) has been constrained. In this work, we introduce a Safe-DRL framework that incorporates safety constraints for the automation of surgical subtasks via DRL training. We validate our approach in a virtual scene that replicates a tissue retraction task commonly occurring in multiple phases of an MIS. Furthermore, to evaluate the safe behaviour of the robotic arms, we formulate a formal verification tool for DRL methods that provides the probability of unsafe configurations. Our results indicate that a formal analysis guarantees safety with high confidence such that the robotic instruments operate within the safe workspace and avoid hazardous interaction with other anatomical structures.

This paper addresses the generation of collision-free trajectories for the autonomous execution of assistive tasks in Robotic Minimally Invasive Surgery (R-MIS). The proposed approach takes into account geometric constraints related to the desired task, like for example the direction to approach the final target and the presence of moving obstacles. The developed motion planner is structured as a two-layer architecture: a global level computes smooth spline-based trajectories that are continuously updated using virtual potential fields; a local level, exploiting Dynamical Systems based obstacle avoidance, ensures collision free connections among the spline control points. The proposed architecture is validated in a realistic surgical scenario.

Providing force feedback as a feature in current Robot-Assisted Minimally Invasive Surgery systems still remains a challenge. In recent years, Vision-Based Force Sensing (VBFS) has emerged as a promising approach to address this problem. Existing methods have been developed in a Supervised Learning (SL) setting. Nonetheless, most of the video sequences related to robotic surgery are not provided with ground-truth force data, which can be easily acquired in a controlled environment. A powerful approach to process unlabeled video sequences and find a compact representation for each video frame relies on using an Unsupervised Learning (UL) method. Afterward, a model trained in an SL setting can take advantage of the available ground-truth force data. In the present work, UL and SL techniques are used to investigate a model in a Semi-Supervised Learning (SSL) framework, consisting of an encoder network and a Long-Short Term Memory (LSTM) network. First, a Convolutional Auto-Encoder (CAE) is trained to learn a compact representation for each RGB frame in a video sequence. To facilitate the reconstruction of high and low frequencies found in images, this CAE is optimized using an adversarial framework and a L1-loss, respectively. Thereafter, the encoder network of the CAE is serially connected with an LSTM network and trained jointly to minimize the difference between ground-truth and estimated force data. Datasets addressing the force estimation task are scarce. Therefore, the experiments have been validated in a custom dataset. The results suggest that the proposed approach is promising.

Robotic-assisted surgeries are commonly used today as a more efficient alternative to traditional surgical options. Both surgeons and patients benefit from those systems, as they offer many advantages, including less trauma and blood loss, fewer complications, and better ergonomics. However, a remaining limitation of currently available surgical systems is the lack of force feedback due to the teleoperation setting, which prevents direct interaction with the patient. Once the force information is obtained by either a sensing device or indirectly through vision-based force estimation, a concern arises on how to transmit this information to the surgeon. An attractive alternative is sensory substitution, which allows transcoding information from one sensory modality to present it in a different sensory modality. In the current work, we used visual feedback to convey interaction forces to the surgeon. Our overarching goal was to address the following question: How should interaction forces be displayed to support efficient comprehension by the surgeon without interfering with the surgeon’s perception and workflow during surgery? Until now, the use the visual modality for force feedback has not been carefully evaluated. For this reason, we conducted an experimental study with two aims: (1) to demonstrate the potential benefits of using this modality and (2) to understand the surgeons’ perceptual preferences. The results derived from our study of 28 surgeons revealed a strong positive acceptance of the users (96%) using this modality. Moreover, we found that for surgeons to easily interpret the information, their mental model must be considered, meaning that the design of the visualizations should fit the perceptual and cognitive abilities of the end user. To our knowledge, this is the first time that these principles have been analyzed for exploring sensory substitution in medical robotics. Finally, we provide user-centered recommendations for the design of visual displays for robotic surgical systems.

Robotic-assisted minimally invasive surgical systems suffer from one major limitation which is the lack of interaction forces feedback. The restricted sense of touch hinders the surgeons' performance and reduces their dexterity and precision during a procedure. In this work, we present a sensory substitution approach that relies on visual stimuli to transmit the tool-tissue interaction forces to the operating surgeon. Our approach combines a 3D diffeomorphic deformation mapping with a generative model to precisely label the force level. The main highlights of our approach are that the use of diffeomorphic transformation ensures anatomical structure preservation and the label assignment is based on a parametric form of several mixture elements. We performed experimentations on both ex-vivo and in-vivo datasets and offer careful numerical results evaluating our approach. The results show that our solution has an error measure less than 1mm in all directions and an average labeling error of 2.05%. It can also be applicable to other scenarios that require force feedback such as microsurgery, knot tying or needle-based procedures.

Fuzzy theory was motivated by the need to create human-like solutions that allow representing vagueness and uncertainty that exist in the real-world. These capabilities have been recently further enhanced by deep learning since it allows converting complex relation between data into knowledge. In this paper, we present a novel Deep-Neuro-Fuzzy strategy for unsupervised estimation of the interaction forces in Robotic Assisted Minimally Invasive scenarios. In our approach, the capability of Neuro-Fuzzy systems for handling visual uncertainty, as well as the inherent imprecision of real physical problems, is reinforced by the advantages provided by Deep Learning methods. Experiments conducted in a realistic setting have demonstrated the superior performance of the proposed approach over existing alternatives. More precisely, our method increased the accuracy of the force estimation and compared favorably to existing state of the art approaches, offering a percentage of improvement that ranges from about 35% to 85%.

The performance of interfaces is affected by human factors, which vary from one person to another, and by the inherent characteristics of the various devices involved. A set of techniques has been studied in order to improve the efficiency and efficacy of input interface devices. These techniques are based on the modification of the motor scaling factor, a transformation similar to the known Control-Display ratio (CD ratio). Operation time, the accuracy of the task and user workload are the indicators used in this work. By means of models based on the various human motor behaviors, the improvement of such indicators has been demonstrated. Using some common input interface devices, a number of experiments have been carried out to evaluate the presented methodology. The results show that the overall performance of input interfaces is significantly improved by applying such methodology.

An assistive rehabilitation strategy for a lower-limb wearable robot is proposed and evaluated. The control strategy monitors the human-orthosis interaction torques and modifies the orthosis operation mode depending on its evolution with respect to a normal gait pattern. The control algorithm relies on the adaptation of the joints stiffness in function of these interaction torques and to the deviation from the desired trajectory. A walking pattern, an average of recorded gaits, is used as reference input. The human-orthosis interaction torques are used to define the time instant when robot assistance is needed and its degree. The objective of this work is to demonstrate the feasibility of ensuring a dynamic stability by means of an efficient real-time stiffness adaptation for multiple joints and simultaneously maintaining their synchronization. The algorithm has been tested with five healthy subjects showing its efficient behavior in maintaining the equilibrium while walking in presence of external forces. The work is performed as a preliminary study to assist patients suffering from Spinal cord injury and Stroke.

Teleoperation, by adequately adapting computer interfaces, can benefit from the knowledge on human factors and psychomotor models in order to improve the effectiveness and efficiency in the execution of a task. While scaling is one of the performances frequently used in teleoperation tasks that require high precision, such as surgery, this article presents a scaling method that considers the system dynamics as well. The proposed dynamic scaling factor depends on the apparent position and velocity of the robot and targets. Such scaling improves the performance of teleoperation interfaces, thereby reducing user's workload.

Most exoskeleton designs rely on structures and mechanical joints that do not guarantee the right match between the orthosis and the user. This paper proposes a virtual joint model based on three active degrees of freedom aimed to emulate a human joint. This joint is capable of performing a dynamic servo-adaptation in real-time to avoid misalignments and to provide a flexible adjustment to different users' sizes in order to avoid undesirable interaction forces.

This paper presents a study on how the application of scaling techniques to an interface affects its performance. A progressive scaling factor based on the position and velocity of the cursor and the targets improves the efficiency of an interface, thereby reducing the user's workload. The study uses several human-motor models to interpret human intention and thus contribute to defining and adapting the scaling parameters to the execution of the task. Two techniques addressed to vary the control-display ratio are compared, and a new method for aiding in the task of steering is proposed.

A multi-robot cooperation station for nano-bio characterization of biological specimens is presented. The station is composed of two long travel range and high resolution robots equipped with self-sensing nanoprobes that are able to cooperate with each other and with standard AFM systems, over a common sample. The robots are guided by the use of an upright high-depth-of-field optical microscope to perform complex nano-bio characterization experiments. To achieve the required precision between the two robots reference frames, specific image processing techniques are needed. One of the tips is dedicated to acquire the topography of the sample at nano scale while the second probe performs the biocharacterization experiments. The obtained results show that the two robots can cooperate within the required resolution in bacterial nanomechanical characterization while high resolution topographic images are acquired.

Teleoperation is essential for applications in which, despite the availability of a precise geometrical definition of the working area, a task cannot be explicitly programmed. This paper describes a method of assisted teleoperation that improves the execution of such tasks in terms of ergonomics, precision, and reduction of execution time. The relationships between the operating spaces corresponding to the human-robot interface triangle are analyzed. The proposed teleoperation aid is based on applying adaptive transformations between these spaces.

In order to enable robots to work in close contact with humans or in environments with unknown obstacles, new reactive control strategies based on sensitive bumper skins are proposed. The aim of this work is to provide a robot with contact and force control based strategies that make it dependable, safe and with foreseeable behaviours. The sensory skin is composed of rigid-bumpers provided with deformation sensors incorporated in a flexible substrate. This solution is a compromise between a simple bumper skin and current array-based robot skins, with better potential performances but, more costly and with usability problems. Such sensitive bumpers cover all moveable links of the robot allowing the detection of collisions, measuring the force involved and locating the point of contact with reasonable precision and cost

This paper presents a manipulation and measurement aid for tasks carried out in micro-nano environments operating with scanning AFM. In teleoperated manipulation or measurement over a given point of the target, where a slow and precise movement is necessary, the developed system increases the accuracy in this point producing a space deformation. In automatic scanning, the adjusted selection of the target, through assisted image segmentation, enables to reduce the working time.

Based on the analysis of the parameters considered as more relevant for the evaluation of hand performances, a functionality index is proposed. The aim of this study is the definition of a functionality index that provides a quantitative measure of hand prosthesis functionality. A clinical analysis of the global efficiency of such prosthesis based on their characteristics has enabled us to determine some parameters from which prosthetic hands can be evaluated and compared. This quantitative evaluation could be a tool to learn about prosthesis and help on further designs.

Laboratories are key components in the learning process of applied matters. The laboratory enables students to acquire methodologies, work habitude, knowledge on equipment operation and experience, in conditions as near as possible to their future professional activity. The evolution of communication and information technologies opens new possibilities in educational methods. The purpose of this paper is to present a Web based system for the implementation of a robotics laboratory with didactic finalities. The laboratory is to be accessible indifferently and simultaneously, in situ or via Internet. The system presented aims at providing access to the laboratory at any time, from everywhere, without space problems, paying special attention to safety requirements. For these reasons we call it an Open laboratory.

Teleoperation solves several difficulties that current robots can not overcome. The intervention of a human operator in deciding the strategies required to perform some given tasks allows robots to carry out operations in dangerous environments or in areas inaccessible to humans. Despite such task becomes feasible, it is necessary to use methods that assist the operator in its manipulation. This paper describes a method for assisting teleoperation, which is based on the dynamic variation of the scale of the working space to improve the movement precision near the point of interest.

As robotics evolves towards application fields in which humans cooperate with robots, working closer and closer, the requirements for human robot interactions increase. This paper presents new advances in gesture based guidance and control of a robot in assistant tasks. The work described includes the integration of force and vision for the quantitative and qualitative appreciation of human intention and the robot behaviors to respond in real time to these human orders.

Teleoperation makes possible the execution of many tasks, that otherwise are not feasible when a human operator can not access to the working area due to dangerousness or environmental conditions, and it is not possible as well, to program the task so as to be performed autonomously, due to its complexity. Nevertheless, manipulation tasks require certain ability from the human operator due to the difficulties that produce the need to operate through control devices that does not fit with the structure of the slave arms. With the aim of increasing the precision capabilities provided by such control interfaces, a vision based procedure designed to deform the space around the working point has been developed. The vision system operates from the detection of the relevant scene elements. This space deformation produces automatically a progressive increase in precision when the robot arm approaches the relevant detected elements.

The analysis of the error of stereo measurements by triangulation is revisited from three points of view: geometrical, statistical and visual quality. When the target is visible by a set of distributed cameras in the workspace, there are multiple combinations of camera pairs adequate to be considered for the location, by triangulation, of the target position. Three-camera placements are analysed evaluating their precision in a short-medium distance. The work presented analyses which combination of stereo measurements gives the best results, and proposes a method for the automatic selection of the most adequate cameras pair.

The aim of the work is to determine the best landmark for its use in tasks such as precise positioning of AGVs or mobile robots in front of loading/unloading areas. The work developed aims at determining the precision that can be achieved with different pattern landmarks, of use in positioning systems, based on computer vision. Different computer vision algorithms have been tested from the images acquired from different camera configurations and bending angles with respect to the vertical axis. As a result of the evaluation of these errors, we extract the recommendation of the most favourable bicolour pattern to be used, as well as the bounding errors obtained from the segmentation and recognition methods most frequently used.

This paper presents an interface for the interaction between a human and an adapted kitchen where different elements, including a robot, have to be controlled. After the global structure of the interface is described a more detailed explanation is presented on the way the interface offers the user an easy and friendly way to select the desired objects and to provide their position to the robot for their manipulation.

This paper faces up the problem of carrying out simple tasks, such as to fasten an object, tasks that become relatively complex in operations using robots, specially in underwater environments. The goal of this work is to perform such tasks by using intelligent strategies that plan step by step allowing for compensation of the robots own limitations. The strategy is based on the use of information from stereo vision and the required auxiliary devices or tools to contribute to the development of the task.

This paper describes a method to achieve the most relevant contours of an image. The presented method proposes to integrate the information of the local contours from chromatic components such as H, S and I, taking into account the criteria of coherence of the local contour orientation values obtained from each of these components. The process is based on parametrizing pixel by pixel the local contours (magnitude and orientation values) from the H, S and I images. This process is carried out individually for each chromatic component. If the criterion of dispersion of the obtained orientation values is high, this chromatic component will lose relevance. A final processing integrates the extracted contours of the three chromatic components, generating the so-called integrated contours image.

Behavior-based navigation of autonomous vehicles requires the recognition of the navigable areas and the potential obstacles. In this paper we describe a model-based objects recognition system which is part of an image interpretation system intended to assist the navigation of autonomous vehicles that operate in industrial environments. The recognition system integrates color, shape and texture information together with the location of the vanishing point. The recognition process starts from some prior scene knowledge, that is, a generic model of the expected scene and the potential objects. The recognition system constitutes an approach where different low-level vision techniques extract a multitude of image descriptors which are then analyzed using a rule-based reasoning system to interpret the image content. This system has been implemented using a rule-based cooperative expert system.

Presents a methodology for autonomous navigation in weakly structured outdoor environments such as dirt roads or mountain ways. The main problem to solve is the detection of an ill-defined structure-the way-and the obstacles in the scene, when working in variable lighting conditions. First, we discuss the road description requirements to perform autonomous navigation in this kind of environment and propose a simple sensors configuration based on vision. A simplified road description is generated from the analysis of a sequence of color images, considering the constraints imposed by the model of ill-structured roads. This environment description is done in three steps: region segmentation, obstacle detection and coherence evaluation.

A robotic arm which automatically guides the camera in the laparoscopic surgery is presented. The goal of this work is to generate adequate camera control strategies to track the working scene during a surgical procedure. The system is based on the computer vision analysis of the laparoscopic image that allows the surgeon to identify the scene's relevant point from the surgical instruments.

The vision system developed is based on the addition of 3D information of some preselected points of the 2D image of the scene in such a way that the system is able not only to recognize the part but also to have a rough description of its location in the 3D space which in general means improving the global performance of the vision system for Robotics applications. The method used to get the depth information is "motion stereo". The two techniques proposed to improve resolution of the stereoscopic system developed are based on, on the one hand, increasing the base line (distance between the initial and the final point of view) and on the other hand performing several depth measurements and getting the final value by averaging.

The vision system described below has the aim of recognizing objects on the plane by means of the binary information of their contour. The object's description consist of a sequence of curvatures which is obtained by contour tracking. The recognizing process is based on the measure of similitude between sequences corresponding to the analized object and to models memorized in a previous training phase. The vision system consists of a specialized circuit allowing the obtention of the object contour using a gradient operator, and a low cost microcomputer which is able to recognize the object and to calculate its orientation.