The field of robotic Flexible Endoscopes (FEs) has progressed significantly, offering a promising solution to reduce patient discomfort. However, the limited autonomy of most robotic FEs results in non-intuitive and challenging manoeuvres, constraining their application in clinical settings. While previous studies have employed lumen tracking for autonomous navigation, they fail to adapt to the presence of obstructions and sharp turns when the endoscope faces the colon wall. In this work, we propose a Deep Reinforcement Learning (DRL)-based navigation strategy that eliminates the need for lumen tracking. However, the use of DRL methods poses safety risks as they do not account for potential hazards associated with the actions taken. To ensure safety, we exploit a Constrained Reinforcement Learning (CRL) method to restrict the policy in a predefined safety regime. Moreover, we present a model selection strategy that utilises Formal Verification (FV) to choose a policy that is entirely safe before deployment. We validate our approach in a virtual colonoscopy environment and report that out of the 300 trained policies, we could identify three policies that are entirely safe. Our work demonstrates that CRL, combined with model selection through FV, can improve the robustness and safety of robotic behaviour in surgical applications.

Abstract Natural language annotations and manuals can provide useful procedural information and relations for the highly specialized scenario of autonomous robotic task planning. In this paper, we propose and publicly release AUTOMATE, a pipeline for automatic task knowledge extraction from expert-written domain texts. AUTOMATE integrates semantic sentence classification, semantic role labeling, and identification of procedural connectors, in order to extract templates of Linear Temporal Logic (LTL) relations that can be directly implemented in any sufficiently expressive logic programming formalism for autonomous reasoning, assuming some low-level commonsense and domain-independent knowledge is available. This is the first work that bridges natural language descriptions of complex LTL relations and the automation of full robotic tasks. Unlike most recent similar works that assume strict language constraints in substantially simplified domains, we test our pipeline on texts that reflect the expressiveness of natural language used in available textbooks and manuals. In fact, we test AUTOMATE in the surgical robotic scenario, defining realistic language constraints based on a publicly available dataset. In the context of two benchmark training tasks with texts constrained as above, we show that automatically extracted LTL templates, after translation to a suitable logic programming paradigm, achieve comparable planning success in reduced time, with respect to logic programs written by expert programmers.

Automatic recognition of fine-grained surgical activities, called steps, is a challenging but crucial task for intelligent intra-operative computer assistance. The development of current vision-based activity recognition methods relies heavily on a high volume of manually annotated data. This data is difficult and time-consuming to generate and requires domain-specific knowledge. In this work, we propose to use coarser and easier-to-annotate activity labels, namely phases, as weak supervision to learn step recognition with fewer step annotated videos. We introduce a step-phase dependency loss to exploit the weak supervision signal. We then employ a Single-Stage Temporal Convolutional Network (SS-TCN) with a ResNet-50 backbone, trained in an end-to-end fashion from weakly annotated videos, for temporal activity segmentation and recognition. We extensively evaluate and show the effectiveness of the proposed method on a large video dataset consisting of 40 laparoscopic gastric bypass procedures and the public benchmark CATARACTS containing 50 cataract surgeries.

Increased demand for less invasive procedures has accelerated the adoption of Intraluminal Procedures (IP) and Endovascular Interventions (EI) performed through body lumens and vessels. As navigation through lumens and vessels is quite complex, interest grows to establish autonomous navigation techniques for IP and EI for reaching the target area. Current research efforts are directed toward increasing the Level of Autonomy (LoA) during the navigation phase. One key ingredient for autonomous navigation is Motion Planning (MP) techniques. This paper provides an overview of MP techniques categorizing them based on LoA. Our analysis investigates advances for the different clinical scenarios. Through a systematic literature analysis using the PRISMA method, the study summarizes relevant works and investigates the clinical aim, LoA, adopted MP techniques, and validation types. We identify the limitations of the corresponding MP methods and provide directions to improve the robustness of the algorithms in dynamic intraluminal environments. MP for IP and EI can be classified into four subgroups: node, sampling, optimization, and learning-based techniques, with a notable rise in learning-based approaches in recent years. One of the review's contributions is the identification of the limiting factors in IP and EI robotic systems hindering higher levels of autonomous navigation. In the future, navigation is bound to become more autonomous, placing the clinician in a supervisory position to improve control precision and reduce workload.

Autonomous robotic surgery requires deliberation, i.e. the ability to plan and execute a task adapting to uncer-tain and dynamic environments. Uncertainty in the surgical domain is mainly related to the partial pre-operative knowledge about patient-specific anatomical properties. In this paper, we introduce a logic-based framework for surgical tasks with deliberative functions of monitoring and learning. The DE-liberative Framework for Robot-Assisted Surgery (DEFRAS) estimates a pre-operative patient-specific plan, and executes it while continuously measuring the applied force obtained from a biomechanical pre-operative model. Monitoring module compares this model with the actual situation reconstructed from sensors. In case of significant mismatch, the learning module is invoked to update the model, thus improving the estimate of the exerted force. DEFRAS is validated both in simulated and real environment with da Vinci Research Kit executing soft tissue retraction. Compared with state-of-the-art related works, the success rate of the task is improved while minimizing the interaction with the tissue to prevent unintentional damage.

Flexible Endoscopes (FEs) for colonoscopy present several limitations due to their inherent complexity, resulting in patient discomfort and lack of intuitiveness for clinicians. Robotic FEs with autonomous control represent a viable solution to reduce the workload of endoscopists and the training time while improving the procedure outcome. Prior works on autonomous endoscope FE control use heuristic policies that limit their generalisation to the unstructured and highly deformable colon environment and require frequent human intervention. This work proposes an image-based FE control using Deep Reinforcement Learning, called Deep Visuomotor Control (DVC), to exhibit adaptive behaviour in convoluted sections of the colon. DVC learns a mapping between the images and the FE control signal. A first user study of 20 expert gastrointestinal endoscopists was carried out to compare their navigation performance with DVC using a realistic virtual simulator. The results indicate that DVC shows equivalent performance on several assessment parameters, being more safer. Moreover, a second user study with 20 novice users was performed to demonstrate easier human supervision compared to a state-of-the-art heuristic control policy. Seamless supervision of colonoscopy procedures would enable endoscopists to focus on the medical decision rather than on the control of FE.

Robot-Assisted minimally invasive robotic surgery is the gold standard for the surgical treatment of many pathological conditions, and several manuals and academic papers describe how to perform these interventions. These high-quality, often peer-reviewed texts are the main study resource for medical personnel and consequently contain essential procedural domain-specific knowledge. The procedural knowledge therein described could be extracted, e.g., on the basis of semantic parsing models, and used to develop clinical decision support systems or even automation methods for some procedure’s steps. However, natural language understanding algorithms such as, for instance, semantic role labelers have lower efficacy and coverage issues when applied to domain others than those they are typically trained on (i.e., newswire text). To overcome this problem, starting from PropBank frames, we propose a new linguistic resource specific to the robotic-surgery domain, named Robotic Surgery Procedural Framebank (RSPF). We extract from robotic-surgical texts verbs and nouns that describe surgical actions and extend PropBank frames by adding any of new lemmas, frames or role sets required to cover missing lemmas, specific frames describing the surgical significance, or new semantic roles used in procedural surgical language. Our resource is publicly available and can be used to annotate corpora in the surgical domain to train and evaluate Semantic Role Labeling (SRL) systems in a challenging fine-grained domain setting.

Optical Coherence Tomography (OCT) is increasingly used in endoluminal procedures since it provides high-speed and high resolution imaging. Distortion and instability of images obtained with a proximal scanning endoscopic OCT system are significant due to the motor rotation irregularity, the friction between the rotating probe and outer sheath and synchronization issues. On-line compensation of artefacts is essential to ensure image quality suitable for real-time assistance during diagnosis or minimally invasive treatment. In this paper, we propose a new online correction method to tackle both B-scan distortion, video stream shaking and drift problem of endoscopic OCT linked to A-line level image shifting. The proposed computational approach for OCT scanning video correction integrates a Convolutional Neural Network (CNN) to improve the estimation of azimuthal shifting of each A-line. To suppress the accumulative error of integral estimation we also introduce another CNN branch to estimate a dynamic overall orientation angle. We train the network with semi-synthetic OCT videos by intentionally adding rotational distortion into real OCT scanning images. The results show that networks trained on this semi-synthetic data generalize to stabilize real OCT videos, and the algorithm efficacy is demonstrated on both ex vivo and in vivo data, where strong scanning artifacts are successfully corrected.

Surgical robots have been widely adopted with over 4000 robots being used in practice daily. However, these are telerobots that are fully controlled by skilled human surgeons. Introducing "surgeon-assist"-some forms of autonomy-has the potential to reduce tedium and increase consistency, analogous to driver-assist functions for lanekeeping, cruise control, and parking. This article examines the scientific and technical backgrounds of robotic autonomy in surgery and some ethical, social, and legal implications. We describe several autonomous surgical tasks that have been automated in laboratory settings, and research concepts and trends.

In this paper, we propose a novel bilateral teleoperation architecture that allows to optimally render the remote interaction force at the local side while guaranteeing a robustly stable behaviour. Stability is guaranteed by ensuring a proper energy exchange between the local and the remote sides. Desired performance is obtained by optimizing the way energy is exploited for generating the behaviour at each side. The effectiveness of the proposed architecture is experimentally validated on a torque-controlled manipulator and in a surgical scenario, using the da Vinci®Research Kit (dVRK).

Deep Reinforcement Learning (DRL) is a viable solution for automating repetitive surgical subtasks due to its ability to learn complex behaviours in a dynamic environment. This task automation could lead to reduced surgeon’s cognitive workload, increased precision in critical aspects of the surgery, and fewer patient-related complications. However, current DRL methods do not guarantee any safety criteria as they maximise cumulative rewards without considering the risks associated with the actions performed. Due to this limitation, the application of DRL in the safety-critical paradigm of robot-assisted Minimally Invasive Surgery (MIS) has been constrained. In this work, we introduce a Safe-DRL framework that incorporates safety constraints for the automation of surgical subtasks via DRL training. We validate our approach in a virtual scene that replicates a tissue retraction task commonly occurring in multiple phases of an MIS. Furthermore, to evaluate the safe behaviour of the robotic arms, we formulate a formal verification tool for DRL methods that provides the probability of unsafe configurations. Our results indicate that a formal analysis guarantees safety with high confidence such that the robotic instruments operate within the safe workspace and avoid hazardous interaction with other anatomical structures.

Abstract The quality of robot-assisted surgery can be improved and the use of hospital resources can be optimized by enhancing autonomy and reliability in the robot’s operation. Logic programming is a good choice for task planning in robot-assisted surgery because it supports reliable reasoning with domain knowledge and increases transparency in the decision making. However, prior knowledge of the task and the domain is typically incomplete, and it often needs to be refined from executions of the surgical task(s) under consideration to avoid sub-optimal performance. In this paper, we investigate the applicability of inductive logic programming for learning previously unknown axioms governing domain dynamics. We do so under answer set semantics for a benchmark surgical training task, the ring transfer. We extend our previous work on learning the immediate preconditions of actions and constraints, to also learn axioms encoding arbitrary temporal delays between atoms that are effects of actions under the event calculus formalism. We propose a systematic approach for learning the specifications of a generic robotic task under the answer set semantics, allowing easy knowledge refinement with iterative learning. In the context of 1000 simulated scenarios, we demonstrate the significant improvement in performance obtained with the learned axioms compared with the hand-written ones; specifically, the learned axioms address some critical issues related to the plan computation time, which is promising for reliable real-time performance during surgery.

Automated surgical gestures classification and recognition are important precursors for achieving the goal of objective evaluation of surgical skills. Many works have been done to discover and validate metrics based on the motion of instruments that can be used as features for automatic classification of surgical gestures. In this work, we present a series of angular metrics that can be used together with Cartesian-based metrics to better describe different surgical gestures. These metrics can be calculated both in Cartesian and joint space, and they are used in this work as features for automatic classification of surgical gestures. To evaluate the proposed metrics, we introduce a novel surgical dataset that contains both Cartesian and joint spaces data acquired with da Vinci Research Kit (dVRK) while a single expert operator is performing 40 subsequent suturing exercises. The obtained results confirm that the application of metrics in the joint space improves the accuracy of automatic gesture classification.

The use of robots in minimally invasive surgery has improved the quality of standard surgical procedures. So far, only the automation of simple surgical actions has been investigated by researchers, while the execution of structured tasks requiring reasoning on the environment and the choice among multiple actions is still managed by human surgeons. In this paper, we propose a framework to implement surgical task automation. The framework consists of a task-level reasoning module based on answer set programming, a low-level motion planning module based on dynamic movement primitives, and a situation awareness module. The logic-based reasoning module generates explainable plans and is able to recover from failure conditions, which are identified and explained by the situation awareness module interfacing to a human supervisor, for enhanced safety. Dynamic Movement Primitives allow to replicate the dexterity of surgeons and to adapt to obstacles and changes in the environment. The framework is validated on different versions of the standard surgical training peg-and-ring task.

Reinforcement Learning (RL) methods have demonstrated promising results for the automation of subtasks in surgical robotic systems. Since many trial and error attempts are required to learn the optimal control policy, RL agent training can be performed in simulation and the learned behavior can be then deployed in real environments. In this work, we introduce an open-source simulation environment providing support for position based dynamics soft bodies simulation and state-of-the-art RL methods. We demonstrate the capabilities of the proposed framework by training an RL agent based on Proximal Policy Optimization in fat tissue manipulation for tumor exposure during a nephrectomy procedure. Leveraging on a preliminary optimization of the simulation parameters, we show that our agent is able to learn the task on a virtual replica of the anatomical environment. The learned behavior is robust to changes in the initial end-effector position. Furthermore, we show that the learned policy can be directly deployed on the da Vinci Research Kit, which is able to execute the trajectories generated by the RL agent. The proposed simulation environment represents an essential component for the development of next-generation robotic systems, where the interaction with the deformable anatomical environment is involved.

Autonomy in robotic surgery will significantly improve the quality of interventions in terms of safety and recovery time for the patient, and reduce fatigue of surgeons and hospital costs. A key requirement for such autonomy is the ability of the surgical system to encode and reason with commonsense task knowledge, and to adapt to variations introduced by the surgical scenarios and the individual patients. However, it is difficult to encode all the variability in surgical scenarios and in the anatomy of individual patients a priori, and new knowledge often needs to be acquired and merged with the existing knowledge. At the same time, it is not possible to provide a large number of labeled training examples in the robotic surgery. This paper presents a framework based on inductive logic programming and answer set semantics for incrementally learning domain knowledge from a limited number of executions of basic surgical tasks. As an illustrative example, we focus on the peg transfer task, and learn state constraints and the preconditions of actions starting from different levels of prior knowledge. We do so using a small dataset comprising human and robotic executions with the da Vinci surgical robot in a challenging simulated scenario.

We propose an Active Object Recognition (AOR) strategy explicitly suited to work with robotic arms in human-robot cooperation scenarios. So far, AOR policies on robotic arms have focused on heterogeneous constraints, most of them related to classification accuracy, classification confidence, number of moves etc., discarding physical and energetic constraints a real robot has to fulfill. Our strategy overcomes this weakness by exploiting a POMDP-based AOR algorithm that explicitly considers manipulability and energetic terms in the planning optimization. The manipulability term avoids the robotic arm to get close to singularities, which require expensive and straining backtracking steps; the energetic term deals with the arm gravity compensation when in static conditions, which is crucial in AOR policies where time is spent in the classifier belief update, before doing the next movement. Several experiments have been carried out on a redundant, 7-DoF Panda arm manipulator, on a multi-object recognition task. This allows to appreciate the improvement of our solution with respect to other competitors evaluated on simulations only.

Robust representation of actions and its sequences for complex robotic tasks would transform robot's understanding to execute robotic tasks efficiently. The challenge is to understand action sequences for highly unstructured environments and to represent and construct action and action sequences. In this manuscript, we present a review of literature dealing with representation of action and action sequences for robot task planning and execution. The methodological review was conducted using Google Scholar and IEEE Xplore, searching the specific keywords. This manuscript gives an overview of current approaches for representing action sequences in robotics. We propose a classification of different methodologies used for action sequences representation and describe the most important aspects of the reviewed publications. This review allows the reader to understand several options that do exist in the research community, to represent and deploy such action representations in real robots.

The use of robots in operating rooms improves safety and decreases patient recovery time and surgeon fatigue, but it introduces new potential hazards that can lead to severe injury or even the loss of human life. Thus, safety has been perceived as a crucial system property since the early days by the industry, the medical community, and the regulatory agents. In this article, we discuss the application of the mathematically rigorous technique known as Formal Verification to analyze the safety properties of a laser incision case study, and we assess its safe and predictable operation. Like all formal methods approaches, our analysis has three distinct components: a method to create a model of the system, a language to specify the properties, and a strategy to prove rigorously that the behavior of the model fulfills the desired properties. The model of the system takes the form of a hybrid automaton consisting of a discrete control part that operates in a continuous environment. The safety constraints are formalized as reachability properties of the hybrid automaton model, while the verification strategy exploits the capabilities of the tool A riadne to address the verification problem and answer the related questions ranging from safety to efficiency and effectiveness.

Series elastic actuators (SEAs) have become fundamental components in robots that physically interact with unstructured environments and humans. Force control of SEAs is indeed an active area of research. This paper proposes a theoretical foundation for acceleration feedback (AF) in SEA force control. Even if AF already appeared in early works on SEAs, its advantages have not been properly highlighted in the literature. In particular, this paper formally motivates improved performance robustness and transparency exactly as if using a softer and lighter actuator. Taking advantage of AF, we propose a generic control architecture characterized by impressive performance robustness in spite of even high environment uncertainties. A comparison with state-of-the-art force control solutions such as disturbance observers and adaptive controllers is reported using a comprehensive set of simulations and experiments. As a result, AF methods exhibit the higher performance robustness and accuracy. Beside this outcome, AF controllers are extremely easy to implement and the rise of low-cost miniaturized accelerometers based on micro electro-mechanical systems (MEMS) represents an additional motivations for their use.

Deformations are an essential aspect of our interaction with real bodies, prompting the development of many modelling and simulation methods for virtual environments. Some of the methods address specific classes of interaction, e.g. pushing, grabbing, cutting or needle insertion, whereas hybrid approaches have been proposed to deal with more complex scenarios. However, a general strategy to combine different simulation methods is still not available. This paper presents a unified approach to combine different methods, each optimised for a specific interaction and object type. Our approach is tailored to the needs of simulating haptic Human-Robot Interactions and allows abstracting from the implementation details of different methods for modelling deformable objects. It has been integrated with collision detection and friction models, ported to a graphic processing unit (GPU), and demonstrated with realistic simulations and experiments.

Teleoperation systems allow humans to interact with remote environments by providing the operator with similar feedback as those s/he would experience at the remote site. Moreover, teleoperators may communicate contact force/torque information from the slave to the master side thus increasing the sense of telepresence of the human operator and improving task performance. When the kinesthetic coupling between operator and environment is enhanced by dynamic coupling, we refer to bilateral teleoperation. Unfortunately force feedback could destabilize a teleoperated system if the communication delay is not properly managed. For this reason many researchers are focusing nowadays on cutaneous feedback that does not affect stability but can still provide useful information to the operator. In this paper we design a cutaneous-feedback teleoperation system where the slave robot is a robotic hand. The software architecture is developed using the Robot Operating System (ROS). In ROS it is easy to integrate the different hardware components in a seamless way. The cutaneous feedback is provided by mini-motors whose vibration intensities are related with the forces measured by the pressure sensors embedded in the hand. Mini-motors are cheap devices that can be easily fastened to the operator's fingers via Velcro straps. The motion of the operator hand is calculated by the Leap Motion controller and mapped into the motion of the robotic hand. The integration of these devices allows to overcome the problem of developing complex and expensive haptic devices for the human hand.

Abstract Following the continuous increase in computational power of consumer hardware, interactive virtual environments have been recently enriched with more and more complex deformable objects. However, many physics engines are still very limited in the way they handle interacting rigid and deformable objects. This paper proposes a constraint‐based approach to real‐time simulation of coupled rigid and deformable objects capable of providing two‐way interactions. Similar techniques have seen widespread usage for either rigid or deformable objects, but not for the simultaneous simulation of both. By extending such approaches, we show not only how interaction is possible but also how it can be performed at real‐time rates. We address contact response and also show how to implement typical constraints to enforce limitations in the degrees of freedom and to enhance the dynamical properties of deformable objects. The method is easily integrated into existing physics engines that use similar constraint solvers and is independent on the kind of deformable object paradigm chosen. The provided simulation results show that the method is fast and effective in handling contacts between rigid and deformable objects and in simulating friction and other kinds of constraints. Copyright © 2015 John Wiley & Sons, Ltd.

Introducing some form of autonomy in robotic surgery is being considered by the medical community to better exploit the potential of robots in the operating room. However, significant technological steps have to occur before even the smallest autonomous task is ready to be presented to the regulatory authorities. In this paper, we address the initial steps of this process, in particular the development of control concepts satisfying the basic safety requirements of robotic surgery, i.e., providing the robot with the necessary dexterity and a stable and smooth behavior of the surgical tool. Two specific situations are considered: the automatic adaptation to changing tissue stiffness and the transition from autonomous to teleoperated mode. These situations replicate real-life cases when the surgeon adapts the stiffness of her/his arm to penetrate tissues of different consistency and when, due to an unexpected event, the surgeon has to take over the control of the surgical robot. To address the first case, we propose a passivity-based interactive control architecture that allows us to implement stable time-varying interactive behaviors. For the second case, we present a two-layered bilateral control architecture that ensures a stable behavior during the transition between autonomy and teleoperation and, after the switch, limits the effect of initial mismatch between master and slave poses. The proposed solutions are validated in the realistic surgical scenario developed within the EU-funded I-SUR project, using a surgical robot prototype specifically designed for the autonomous execution of surgical tasks like the insertion of needles into the human body.

In this paper we discuss the application of formal methods for the verification of properties of control systems designed for autonomous robotic systems. We illustrate our proposal in the context of surgery by considering the automatic execution of a simple action such as puncturing. To prove that a sequence of subtasks planned on pre-operative data can successfully accomplish the surgical operation despite model uncertainties, we specify the problem by using hybrid automata. We express the requirements of interest as questions about reachability properties of the hybrid automaton model. Then, we compare the different performance of current state-of-the art tools for reachability analysis of hybrid automata.

Networked control systems are a well-known sub-set of cyber-physical systems in which the plant is controlled by sending commands through a digital packet-based network. Current control networks provide advanced channel access mechanisms to guarantee low delay on a limited fraction of packets (low-delay class) while the other packets (un-protected class) experience a higher delay which increases with channel utilization. We investigate the extension of model predictive control to choose both the command value and its assignment to one of the two classes according to the predicted state of the plant and the knowledge of network condition. Experimental results show that more commands are assigned to the low-delay class when either the tracking error is high or the network condition is bad.

This paper proposes a novel architecture for networked embedded systems which exploits a differentiated services approach to guarantee control performance even in case of time-varying network condition. Control commands are transmitted as high-priority packets when the plant behavior is far from the desired target or network condition does not assure the reliable and prompt reception of commands and measurements. The assignment of different priorities to packets belonging to the same flow (i.e., commands or measurements) may lead to out-of-sequence forwarding which may compromise the plant stability and the estimation of the state at the controller side. As far as we know this is the first work which solves such important issues by combining priority-based forwarding with the passivity mechanism to ensure stability. Moreover, we adopt an optimized filter for plant state estimation which takes into account the vector of the last commands used by the plant (updated through information received from the plant together with measurements) and the vector of the last measurements. Different packet marking strategies are compared: the best one leads to the same performance of random marking by using a smaller fraction of the high-priority bandwidth.

Ultrasound (US) guided biopsy is a medical procedure routinely performed in clinical practice. This task could be performed by robotic systems to improve the precision in the execution and then the safety for the patient. Both robotic and human procedures could greatly benefit from real-time localization of the needle in US images. This information could guide the robot or the specialists to the correct target point avoiding critical structures. Unfortunately US data provide very low quality images of the needle making this task quite complex, even more if you want to perform the localization on-line during the image acquisition. In this work we present a needle localization method able to extract the needle orientation and the tip position in real time from B-mode US images. To evaluate the performance of the algorithm in a precise way we use an optical tracking system to measure the position and the orientation of the needle and the US probe. In such a way the comparison is not human dependent (i.e. there are no radiologists manually selecting the needle tip) and fully repeatable. The results show an improvement in term of localization accuracy compared to previous works in literature.

Networked control systems are feedback systems where plant and controller are connected through lossy wired/wireless networks. To mitigate communication delays and packet losses different control solutions have been proposed. In this work the model predictive control (MPC) has been improved by introducing transmission options offering different probabilities of packet drops (high priority service and low priority service). This Differentiated Services architecture introduces Quality-of-Service (QoS) guarantees and can be used to jointly design the control command and the transmission strategy. A novel MPC-QoS controller is proposed and its design is obtained by solving a mixed integer quadratic problem.

The relevance of formal verification methods is widely recognized in the computer science and embedded systems community. Recently, such methods have been introduced also within the control community, to help designers in developing control architectures for complex robotics systems. Robotic systems typically mix continuous and discrete behaviors that cannot be modeled faithfully using neither continuous-only nor discrete-only formalisms. The interaction of continuous and discrete dynamics makes the formal treatment of this kind of systems computationally very demanding, and justifies the need of studying new methods and algorithms. In this paper, we outline the current state-of-the-art, and describe some open problems in verification, refinement and implementation of autonomous robotic systems. We motivate the relevance of our analysis by means of an Autonomous Robotic Surgery test case.

This paper addresses the problem of training of operators for telemanipulation tasks. In particular, it describes the development of a physics based virtual environment that allows a user to train in the control of an innovative robotic tool designed for the retrieval of spent nuclear fuels.

In this work we have designed and developed a new navigation system for interventional radiology, implemented in a light and compact device. The system attached to the needle is composed by a small screen that gives hints about the position and the orientation, a controller that commands the screen and interfaces with the computer, and a marker that communicates with a tracking system. By using a real time software the user is guided to move the needle along the desired position and orientation. To the best of our knowledges, this is the first system to have the navigation display integrated directly on the tool. The in-vitro tests we have performed, show how such a system yields a higher precision in the execution of the task and a reduction of the time required to complete the procedure.

This paper proposes a new criterion, called absolute transparency, to design control schemes applied to bilateral teleoperation of mobile robots with time-varying delay. The absolute transparency measures how and how fast the human operator and the remote system interact with each other through a teleoperation system. The absolute transparency of different control schemes is analyzed and tested through teleoperation experiments where a human operator drives a mobile robot and receives both visual and force feedback.

Within the broader scope of underwater biomimetics, in this paper we address the relevance of factors such as shape and elasticity distribution in the ability of a compliant device to imitate the kinematic behaviour of a fish. We assess the viability of myometry as a tool to determine candidate mechanical parameters without relying solely on analytical models; we show that we can obtain elasticity distributions that are both consistent with previous theoretical investigations and experimentally better adherent to the passive kinematics of a biological embodiment (rainbow trout).

In this paper the problem of computing a rigid object trajectory in an environment populated with deformable objects is addressed. The problem arises in Minimally Invasive Robotic Surgery (MIRS) from the needs of reaching a point of interest inside the anatomy with rigid laparoscopic instruments. We address the case of abdominal surgery. The abdomen is a densely populated soft environment and it is not possible to apply classical techniques for obstacle avoidance because a collision free solution is, most of the time, not feasible. In order to have a convergent algorithm with, at least, one possible solution we have to relax the constraints and allow collision under a specific contact threshold to avoid tissue damaging. In this work a new approach for trajectory planning under these peculiar conditions is implemented. The method computes offline the path which is then tested in a surgical simulator as part of a pre-operative surgical plan.

This article reports on experiments about haptic perception aimed at measuring the force/torque differential thresholds applied to the hand-arm system. The experimental work analyzes how force is sent back to the user by means of a 6 degrees-of-freedom haptic device. Our findings on force perception indicate that the just-noticeable-difference is generally higher than previously reported in the literature and not constant along the stimulus continuum. We found evidence that the thresholds change also among the different directions. Furthermore, asymmetries in force perceptions, which were not described in previous reports, can be evinced for most of the directions. These findings support our claim that human beings perceive forces differently along different directions, thus suggesting that perception can also be enhanced by suitable signal processing, that is, with a manipulation of the force signal before it reaches the haptic device. We think that the improvement of the user perception can have a great impact in many applications and in particular we are focusing on surgical teleoperation scenarios.

Interactive simulations of deformable bodies are a growing research area with possible applications in several fields, i.e. computer aided surgery. The main implementation issue is to mimic the real behavior of the body at the extremely high rates required by haptic devices. Since even high-end computers have inadequate performance, one possible solution is to exploit the parallelism of modern Graphics Processing Units. In this paper we present our research aiming at moving the whole computational process from theCPUto theGPUtaking advantage of the computational power of the graphics hardware. We use a mass-spring model, augmented with local damping coefficients and volume preservation forces. Collision detection is performed against external rigid bodies with high complexity mesh, such as the skeletonpsilas one. The user interacts with the model by controlling virtual tools, i.e. probes or tweezers. Haptic forces are computed onGPUand the results are asyncronously transferred to theCPU. Our approach can simulate the deformation of complex models with gravity and interaction with environment and tools at a frame rate higher than 1 KHz, making it suitable for visual rendering and haptic feedback.

This special section editorial defines home automation and investigates the various approaches to home automation which can be used to facilitate independent living. The paper concludes by introducing the three technical papers that are part of the special section.

The two main categories of deformable models used in surgical simulators are Mass Spring Models (MSM) and Finite Element Models (FEM). Mass spring models are often preferred due to their simplicity and low computational cost and because they allow to perform topology changes on the modeled body without significant computational overhead. The principal drawback of the mass spring model is the need of complex calibration procedure since they don't have a clear physical meaning. In this paper we propose a new method to calibrate mass spring models. Our method uses CAT data to identify mass values and deformation measures to define elastic coefficient and damping ratio for the springs of the model. Spring parameters are obtained through a genetic algorithm that minimizes the difference between the model and the measured behavior. The algorithm we developed to compute the masses was tested with medical CAT data whereas the spring algorithm correctness was tested with synthetic models. Simulation verifications are presented.

In the last few years, robotics education has been proposed in Universities and High Schools, to take advantage of the appeal of this discipline to teach scientific subjects. Robotics education is greatly improved when classroom teaching is supported by adequate laboratory courses and experiments following the "learning by doing" paradigm. However, cost of robotic laboratory equipment is an important issue, and several low cost instruments have been developed to permit an effective teaching in robotics at an affordable cost. These learning tools have demonstrated the importance and effectiveness of hand-on experiences and of group work in robotics learning. Among the best known and most used tools for robotics teaching, there is the Legocopy Mindstormtrade set, which is normally used to develop and program mobile robots, often biologically inspired and cooperating. In this paper however, we describe our experience with using LEGO kits in the development of teaching curricula for fixed robot manipulators. We found that the LEGO set is also an excellent tool to analyze robot kinematics and trajectory planning, and we describe our experience with a laboratory course designed to address kinematic properties of fixed robots

Teleoperation with force feedback is a complex task and a good haptic device is a key element. In this paper we present an innovative hardware/software structure used to control an actuated 6 dof joystick in a teleoperation task. To increase speed and reliability, parts of the kinematic calculation are embedded into the joystick controller. To implement this idea we used an FPGA to handle both the low level tasks and the algorithmic part of the approach. This speedup does not preclude the possibility of changing the control strategies, the parameters and the feedback calculations as well, necessary to carry out experiments about human perception. Thus the need of flexibility and performance, and the choice of an FPGA. We tested the setup proposed in a real teleoperation task within our software framework (Penelope) collecting data of speed, precision and reliability

In this paper we report on the efforts carried out at the Robotics Laboratory ALTAIR of the University of Verona (Italy) towards the development of a high performance architecture for bilateral, i.e. force reflecting, teleoperation system. This architecture, called Penelope, takes into account the main features of a haptics system: real-time behavior, distributed resources, general purpose structure and safe exchange of data. The emphasis here is on heterogeneous hardware and software within the same structure

This paper gives an overview of results of the Human++ research program. This research aims to achieve highly miniaturized and autonomous transducer systems that assist our health and comfort. It combines expertise in wireless ultra-low power communications, 3D integration technologies, MEMS energy scavenging techniques and low-power design techniques

This paper presents theoretical and experimental results of robot interaction control schemes for a teleoperation system whose slave end effector comes in contact with a compliant surface. A salient feature of this work is a comparison of different control schemes implemented on a system whose slave consists of an industrial robot with an open software architecture and equipped with a wrist force sensor. Three control strategies are considered; one based on a single regulator PID giving a nominal performance, the second based on impedance control, and the last one based on a hybrid system approach. The performance of the various schemes is compared on a contact and puncturing task of an elastic membrane. The comparison is first carried out in simulation and then the simulations are validated with experiments. Quantitative as well as subjective performance measures are compared to assess the performance of the different control schemes.

The automatic execution of a complex task requires the identification of an underlying mental model to derive a possible task control sequence. The model aims at analysing and segmenting the task in simpler sub-tasks. As an example of a complex task, in this paper we consider teleoperation where a person commands a remote robot. This paper presents a new modeling approach using hidden Markov models (HMM) and support vector machines (SVM) to analyse the force/torque signals of a teleoperation task. The task is divided into simpler sub-tasks and the model is used to segment the signals in each sub-task. The segmentation gives informations on the system behavior identifying the changes of the model states. Peg in hole force/torque data are used for testing the model. The results are consistent with the literature with respect to off-line analysis, whereas a significant increase of performance is achieved for on-line analysis.

This paper presents a comparison of four control strategies for a teleoperation system with a flexible link slave. In the classic literature, slave devices are considered only with rigid links whereas applications such as space robotics include flexible links. Thus, a more realistic model of a telemanipulator to compare and evaluate the performance of control architectures for teleoperation needs to be developed. The simulation is based on a one degree of freedom telemanipulation system available in our laboratory. Results indicate that position control modes could yield better performance than rate control when considering point-to-point tests.

In this paper we describe some of the key technologies of a mobile manipulator that are used for package transportation in a pharmaceutical warehouse. The paper presents, beyond the functional aspects of the system, the main modules of the software that controls the mobile manipulator. The robot is a demonstrator of technologies for transporting safely and efficiently goods in partially structured, dynamic and public environments. Several issues are discussed such as the design and integration of mechanical elements, the development of non invasive localization and guidance procedures, the design and control of grasping devices for specific box/storage combinations, and the development of testing, verification and validation procedures satisfying strict pharmaceutical regulations. Results of the laboratory tests demonstrate the capability of the prototype.

This paper proposes a new geometrical structure for stereoscopic vision using omnidirectional cameras. The motivation of this work comes from the desire to equip a small hopping robot with an efficient and robust vision system to perform self localization during exploration missions. Because of size and weight constraints, we selected the Panoramic Annular Lens, for which no geometric model of stereo configuration was available. The paper describes the geometrical optical properties of the single lens and proposes a configuration for doing stereo vision with this lens. The analytical properties as well as the requirements of the complete system are discussed in the paper.

This paper describes our efforts and plans to develop a Virtual Laboratory for the education in Robotics and Automation. These efforts are characterized by the need of blending R&A subjects into a traditional Computer Science curriculum, thus forcing a specific selection of development topics. In this context, the Robotics Laboratory must provide the basic as well as advanced experiments, to address the needs of students at different education levels. In this paper, we present the development of three main applications, to support Control Systems and Robotics classes, as well as the thesis and dissertation research. Of particular interest is the effort in the area of teleoperation, preliminary to the opening (next year) of a new curriculum on Medical Informatics, in which Computer Assisted Surgery will play an important role.

This paper describes a new calibration method for robot manipulators using a mobile TV camera and 3D digitizer. This method is motivated by the need of precisely calibrating a large work space without using special instruments or calibration fixtures. The method uses an innovative kinematic model of the robot-camera system, which permits one to combine, without loss of precision, measurements taken from different camera positions of different work volumes. The method includes algorithms for the selection of robot poses and kinematic parameter identification. Results of simulation tests of a surgical robot are presented.

We propose a new method for real time obstacle avoidance using visual information. It is based on our on-going research on vector fields and potential functions associated to successful motion planning algorithms. Specifically, this paper addresses the avoidance of dynamic obstacles which have the property of producing instabilities in the navigation vector field associated to a planning algorithm. We show how avoidance maneuvers can be seen as a reaction to potential field instabilities measured by the moving robot. The paper briefly summarizes the analytical derivation of this approach, and discusses the results of planning experiments carried out using a Nomad200 robot with a landmark-based planning and navigation systems.

To tackle the singularity problem, the SC (singularity-consistent) approach was introduced. It achieves very stable control at and around a singularity with feasible joint velocities and no directional error in the end-effector velocity. This approach is very suited for a direct manual teleoperation system because of its errorless character for the direction. Until now, it was applied for the teleoperation of a nonredundant manipulator. In this paper, the SC approach for a 7-DOF manipulator will be addressed. To derive the whole properties of the SC approach, analytical studies for both the adjoint and the determinant of the Jacobian are necessary. However, it is very difficult to realize this goal, since the kinematics of a 7-DOF manipulator is quite complicated. Therefore, here, we establish a method to apply the SC approach to a 7-DOF manipulator numerically without analyzing the kinematic properties. This method, though, cannot realize the whole properties of the SC approach but a stable control at and around the singularities is achieved which is the most important and demanded property of the SC approach. Moreover, this method can also be applied for any type of articulated manipulator if its Jacobian can be defined. The results of our approach have been confirmed by experiments with graphics model.

Describes an improved architecture, for the control of force-feedback teleoperation in the presence of variable communication time-delay between the master and the slave. It is based on wave variable transformations and identification of time delay properties. This scheme achieves better position and force tracking than similar architectures, by estimating the current value of time delay and compensating its variations by adjusting a single control parameter. We derive some of the analytical properties of this scheme, and we show system performance by simulating a simple Internet-based teleoperation system.

Different working principles are often considered when different visual behaviors are implemented in an agent. This occurs basically because the physical interaction between the behavior and the environment is not studied in depth. The paper shows how apparently different visual behaviors share common theoretical principles for their working mechanism. In particular properties related to the navigation vector field they compute in the environment, provide a base to explain visual learning, guidance, topological navigation, sub goal placement, obstacle avoidance and navigation enhancement. To handle the mathematics of a vector field robust tools are needed. Techniques borrowed from computer vision literature provide the necessary mathematical tools. All behaviors described have been tested in real robots. On going research is still in progress for topological navigation and subgoal placement.

Surveys past and current designs of surface mobility systems for planetary exploration robots developed at JPL/Caltech. Wheeled rovers are discussed in some detail and compared to new designs, such as legged and hopping robots, which are emerging as viable alternatives to wheeled mobility for specific applications. The paper discusses the main features of mobility designs and summarizes some of the experimental results.

The development of a vision-only intelligent interface for dexterous robots is described, capable of tracking operator motions: evaluating trajectory feasibility, and providing visual feedback to the operator. This interface analyses the operator's arm motion for safety, and then converts it into trajectory commands for a mechanical arm. Potentially dangerous situations, such as singularity and self collision, are displayed to the operator as graphical icons superimposed over the robot video images. The paper summarizes the main features of the current implementation, presents the approach developed for singularity identification and describes the performance of the demonstration prototype.

This paper describes a minimalist hopping robot that can perform basic exploration tasks on Mars or other moderate gravity bodies. We show that a single actuator can control the vehicle's jumping and steering operations, as well as the panning of an on-board camera. Our novel thrusting linkage also leads to good system efficiency. The inherent minimalism of our hopping paradigm offers interesting advantages over wheeled and legged mobility concepts for some types of planetary exploration. The paper summarizes the evolutionary development of the system, issues relevant to the design of such jumping systems, and experimental results obtained with system prototypes.

This paper describes a PC-based controller for robot-assisted minimally invasive surgery. The development is motivated by the need of reducing the exposure of operating room personnel to X-rays during surgical procedures such as percutaneous discectomy. The approach taken consists of upgrading the hardware architecture and the control software of a clean-room PUMA 260 manipulator, and by developing a new vision-based operator interface. The original PUMA controller is interfaced to a PC for trajectory generation and force/torque data acquisition. Kinematic and communication functions are ported to the PC from the RCCL package. The operator interface is used for robot calibration and motion commands. In this implementation, the position control and force/torque data acquisition are executed every 28 ms, thus preserving the original robot performance. Preliminary tests have shown the validity of this approach and readiness of the system for more realistic experiments in the operating room.

This paper describes the approach followed in the design of a service robot for health care applications. Under the auspices of the NASA Technology Transfer Program, a partnership was established between JPL and RWI, a manufacturer of mobile robots, to design and evaluate a mobile robot for health care assistance to the elderly and the handicapped. The activities of the first phase of the project include the development of a multi-modal operator interface, and the design and fabrication of a manipulator arm for the mobile robot. This paper describes the architecture of the system, the features of the manipulator arm, and the operator interface.

This paper presents a method for motion planning in dynamic environments, subject to robot dynamics and actuator constraints. The time optimal trajectory is computed by first generating an initial guess using the concept of velocity obstacle. The initial guess, computed by a global search over a tree of avoidance maneuvers, is then optimized using a dynamic optimization. This method is applicable to repetitive tasks in known dynamic environments, as is demonstrated for a planar robot manipulator.

This paper describes the architecture and performance of a PC-based configuration controller for dexterous 7-DOF manipulators. The computing platform is a 486-based personal computer equipped with a bus extender to access the robot Multibus controller, together with a single-board computer as the graphical engine, and a parallel I/O board to interface with a force-torque sensor mounted on the manipulator wrist. The Windows environment is enhanced by the iRMX real-time operating system that runs the configuration control algorithms for redundancy resolution. The position control algorithm is executed every 2.5 ms; motions can be simulated and displayed in real-time by the graphical engine on a separate monitor. The results of several experiments carried out with a Robotics Research manipulator have shown motion control capabilities comparable to those obtained with more extensive computing systems, thus validating the use of PCs for dexterous manipulator control. PC-based motion controllers for RRC arms, incorporating the configuration control software, are currently produced by the Robotics Research Corporation.

This paper briefly describes an advanced teleoperator (ATOP) system and its control station where a variety of computer-based operator interface devices and techniques are integrated into a functional setting, accommodating a primary operator and secondary operators. Computer graphics is a key operator interface component in the control station where new types of manual interface devices also are employed. The results of some generic and application task experiments are summarized, including the performance of a simulated remote satellite servicing task, carried out under four to eight seconds communication time delay, using satellite TV and Internet computer communication links. In conclusion, the paper highlights the lessons learned so far.>

In the last few years, teleoperated tasks have been the subject of extensive research to determine the best combination of control modalities according to specific criteria. The operator's performance were compared on the basis of task completion time and of force and torque measurements during the tasks. This paper proposes a procedure for the spectral analysis of force and torque signals generated during teleoperation experiments. There are two main reasons for examining teleoperation data in the frequency domain: a spectral analysis of different tasks can validate the assumptions made in the design of the teleoperator, and a task's frequency signature can be a valuable measure of the operator's performance.

By means of a system of parallel coordinates, a nonprojective mapping from R/sup N/ to R/sup 2/ is obtained for any positive integer N. In this way multivariate data and relations can be represented in the Euclidean plane (embedded in the projective plane). Basically, R/sup 2/ with Cartesian coordinates is augmented by N parallel axes, one for each variable. The N joint variables of a robotic device can be represented graphically by using parallel coordinates. Two key reasons motivate this kind of approach: first, some properties of the relation are better perceived visually from the parallel coordinate representation; and secondly, new algorithms and data structures can be obtained from this representation. The main features of parallel coordinates are described, and an example is presented of their use for configuration space representation of a mechanical are. The purpose of the example is to show the potential of the representation for the cases where Cartesian coordinates cannot be used.>

A new device for human operator control of a robotic gripper has been developed and preliminary evaluation has been performed. The JPL Force Reflecting Hand Trigger system features: an instrumented index finger trigger with load cell detection of finger force. A servo controlled, lead screw driven backdrive capability by which the trigger's position can be made to follow that of the remotely controlled gripper. And a novel feedback mechanism by which clamping force or some other signal can be fed back via a swiveling motion, also servo controlled, of the trigger surface (force reflection). This system has undergone preliminary testing in which the amount of force reflection is varied and dynamic force tracking response is observed.