Domain-specific ontologies support system design and can establish a framework for fulfilling user-level, safety, or ethical requirements. The IEEE 7007-2021 Ontological Standard for ethically driven robotics and automation systems is the first industry standard to introduce a structure of ontologies concerning robot ethics and related fields, such as data privacy, transparency, responsibility, and accountability, offering a systems science approach to support the ethically aligned design of complex cyber–physical systems (CPSs) and robots particularly. This article provides a comprehensive overview of the main ontological commitments composing the foundation of the standard, the rationale behind their development, together with use cases of applications. Future directions for ethically aligned robotics and artificial intelligence (AI)-based systems along IEEE 7007-2021 are outlined, taking into account the exponentially growing fields of service and medical robotics.

Interventional robotic systems have been deployed with all existing imaging modalities in an expansive portfolio of therapies and surgeries. Over the years, literature reviews have painted a comprehensive portrait of the translation of the underlying technology from research to practice. While many of these robots performed promisingly in preclinical settings, only a handful of them managed to evolve further, break through the commercialization boundary, and even fewer reached a wide-scale adoption. Despite the undeniable success of service robotics in general and particularly in some sophisticated medical applications, image-guided robotics’ impact remained modest compared to other surgical areas, especially laparoscopic minimally invasive surgery. This article aims to embrace the state of the art on the one hand, provide a comprehensive narrative of the situation described, support future system developers, and facilitate the translation from scientific research to applied clinical technology development.

Telesurgical robotics, as a technical solution for robot-assisted minimally invasive surgery (RAMIS), has become the first domain within medicosurgical robotics that achieved a true global clinical adoption. Its relative success (still at a low single-digit percentile total market penetration) roots in the particular human-in-the-loop control, in which the trained surgeon is always kept responsible for the clinical outcome achieved by the robot-actuated invasive tools. Nowadays, this paradigm is challenged by the need for improved surgical performance, traceability, and safety reaching beyond the human capabilities. Partially due to the technical complexity and the financial burden, the adoption of telesurgical robotics has not reached its full potential, by far. Apart from the absolutely market-dominating da Vinci surgical system, there are already 60+ emerging RAMIS robot types, out of which 15 have already achieved some form of regulatory clearance. This article aims to connect the technological advancement with the principles of commercialization, particularly looking at engineering components that are under development and have the potential to bring significant advantages to the clinical practice. Current RAMIS robots often do not exceed the functionalities deriving from their mechatronics, due to the lack of data-driven assistance and smart human–machine collaboration. Computer assistance is gradually gaining more significance within emerging RAMIS systems. Enhanced manipulation capabilities, refined sensors, advanced vision, task-level automation, smart safety features, and data integration mark together the inception of a new era in telesurgical robotics, infiltrated by machine learning (ML) and artificial intelligence (AI) solutions. Observing other domains, it is definite that a key requirement of a robust AI is the good quality data, derived from proper data acquisition and sharing to allow building solutions in real time based on ML. Emerging RAMIS technologies are reviewed both in a historical and a future perspective.

The segmentation of moving objects from video sequences is an intensely researched topic. The two leading application domains are video surveillance systems and self-driving vehicles. Many of the current approaches use some deep learning architecture to accomplish the task. For the training of such models, an extensive collection of input data is necessary. However, it is hard to find an openly accessible, properly annotated dataset for this task in the field of mobile robotics, so one has to create their own. Deep learning models of high complexity easily overfit small datasets if they are trained from scratch, therefore readily available pre-trained networks should be better used. The pretrained deep learning models usually expect RGB images as their input, meanwhile most of the state-of-the-art approaches for moving object segmentation incorporate optical flow data in the prediction process as well. In this paper, we propose a moving object segmentation deep neural network for avoiding dynamic obstacles in the case of indoor mobile robot navigation. The network was built on the top of a deep learning model that was pre-trained for RGB image recognition. We encode the optical flow data and the video frames together into a format that can be processed by the pre-trained network. We introduce a compound loss and training strategy for the model. The empirical evaluation results show that our network predicts satisfactory segmentation masks on the test data for the task of dynamic obstacle avoidance in indoor mobile robot navigation while operating in real-time ($\sim 30$ fps) and being robust to camera movement and the appearance of moving objects.

A key research domain within self-driving vehicle technology is system safety. Current development efforts target Level 3 autonomy, where the vehicle controls both lateral and transversal motion of the dynamic driving task, while the driver is permitted to divert its attention, as long as s/he is able to react properly to a handover request. Situation awareness, describing the cognitive capabilities of the driver in that given instant, has become a key metric of a safe handover process. This paper proposes an experimental setup for handover studies using the CARLA driving simulator alongside the master console of the da Vinci Surgical System. This surgical master console offers a built-in display in a setup particularly suitable for the analysis of situation awareness, controllers and foot pedals. Steering wheel-like behavior of the arms was achieved using impedance control. The presented solution is open-source, and available at https://github.com/ABC-iRobotics/dvrk_carla.

Medical imaging introduced the greatest paradigm change in the history of modern medicine, and particularly ultrasound (US) is becoming the most widespread imaging modality. The integration of digital imaging into the surgical domain opens new frontiers in diagnostics and intervention, and the combination of robotics leads to improved accuracy and targeting capabilities. This paper reviews the state-of-the-art in US-based robotic platforms, identifying the main research and clinical trends, reviewing current capabilities and limitations. The focus of the study includes non-autonomous US-based systems, US-based automated robotic navigation systems and US-guided autonomous tools. These areas outline future development, projecting a swarm of new applications in the computer-assisted surgical domain.

Advanced surgical robotic systems aim to offer improved capabilities through automated low-level functions. In the applications, soft tissue mechanics and tool-tissue interaction modeling play an important role in achieving optimal control, relying on model-based control methods. This approach allows for addressing crucial issues during teleoperation, such as time-delay, state observation or stability. This paper presents a novel approach for modeling the behavior of soft tissue during surgical interventions, relying on the widely-employed concept of rheological models. The nonlinear Wiecher model is used for reaction force estimation during tissue indentation, tested on beef liver samples for acquiring mechanical parameters from experimental data. Curve fitting methods were used in both stress relaxation and constant indentation speed compression phases. Reaction forces are estimated using the proposed model, followed by verification tests on ex-vivo beef liver samples. The results of this research showed that the proposed novel rheological soft tissue model is capable of estimating the reaction forces acting on the tool, if the shape of the deformed tissue is known in time. This model can be successfully integrated into closed-loop surgical robot controllers.

Reliable force control is one of the key components of modern robotic teleoperation. The performance of these systems, in terms of safety and stability, largely depends on the controller design, as it is desired to deal with various disturbing conditions, such as uncertainties of the model parameters or latency-induced problems. This work presents a polytopic quasi-linear parameter-varying (qLPV) model derived from a previously verified nonlinear soft tissue model, along with a model-based force control scheme that involves a tensor product polytopic state feedback controller. The derivation is based on the Tensor Product (TP) Model Transformation. The proposed force control scheme is verified and evaluated through numerical simulations.

Robotic telesurgery application could greatly benefit from force control, providing safe and stable application even under time delays and other disturbing conditions. This paper describes a soft tissue model which can be integrated into the control loop of a teleoperational master -- slave robot. Three alternations of the Wiechert model have been implemented and matched against experimental data. A nonlinear model has been proposed to realistically model the properties of liver-type tissues. Based on the model, adequate force control can be designed in the future.

Benchmark problems continue to represent an actively studied domain, focusing on application-based situations, where controllers have to deal with typical real environments. In this paper, a Robust Fixed Point Transformations (RFPT)-based Model Reference Adaptive Controller (MRAC) is designed for a modified Translational Oscillations by a Rotational Actuator (TORA) system, which is an indirectly driven, underactuated classical mechanical system with peculiar properties. The RFPT-based design has the advantage of working only with three free parameters, and does not need complex a priori calculations. It is founded on the idea that at the cost of replacing the requirement for global stability with local stability, a mathematically very simple and geometrically lucid, well interpreted methodology can be developed. The resulting structure directly concentrates on the primary design intent, i.e., on the realization of a purely kinematically prescribed trajectory tracking. Examples and simulation results are presented in this paper, demonstrating that the RFPT-based design can provide an efficient MRAC controller for a very special physical system.

Model Reference Adaptive Controllers (MRAC) have dual functionality: besides guaranteeing precise trajectory tracking of the controlled system, they have to provide an “external control loop” with the illusion that it controls a physical system of prescribed dynamic properties, i.e., the “reference system”. The MRACs are designed traditionally by Lyapunov's 2ndmethod that is mathematically complicated, requiring strong skills from the designer. Adaptive controllers alternatively designed by the use of Robust Fixed Point Transformations (RFPT) operate according to Banach's Fixed Point Theorem, and are normally simple iterative constructions that also have a standard variant for MRAC design. This controller assumes a single actuator that is driven adaptively. Master-Slave Systems form a distinct class of practical applications, in which two arms-the master and the slave-operate simultaneously. The movement of the master must be tracked precisely by the slave in spite of the quite different forces exerted by them. In the present paper, a soft tissue-cutting operation by a master-slave structure is simulated. The master arm has a simple torque-reference friction model, and is driven by the surgeon. The obtained master arm trajectory has to be precisely tracked by the electric DC motor driven slave system, which is in dynamic interaction with the actual tissue under operation. It is shown via simulations that the RFPT-based design can efficiently solve such tasks without considerable mathematical complexity.

Object tracking is a key enabling technology in the context of computer-assisted medical interventions. Allowing the continuous localization of medical instruments and patient anatomy, it is a prerequisite for providing instrument guidance to subsurface anatomical structures. The only widely used technique that enables real-time tracking of small objects without line-of-sight restrictions is electromagnetic (EM) tracking. While EM tracking has been the subject of many research efforts, clinical applications have been slow to emerge. The aim of this review paper is therefore to provide insight into the future potential and limitations of EM tracking for medical use. We describe the basic working principles of EM tracking systems, list the main sources of error, and summarize the published studies on tracking accuracy, precision and robustness along with the corresponding validation protocols proposed. State-of-the-art approaches to error compensation are also reviewed in depth. Finally, an overview of the clinical applications addressed with EM tracking is given. Throughout the paper, we report not only on scientific progress, but also provide a review on commercial systems. Given the continuous debate on the applicability of EM tracking in medicine, this paper provides a timely overview of the state-of-the-art in the field.

Unambiguous definition of spatial position and orientation has crucial importance for robotics. In this paper we propose an ontology about positioning. It is part of a more extensive core ontology being developed by the IEEE RAS Working Group on ontologies for robotics and automation. The core ontology should provide a common ground for further ontology development in the field. We give a brief overview of concepts in the core ontology and then describe an integrated approach for representing quantitative and qualitative position information.

Service robotics is becoming a leading application area of human-centered technologies, and the rise of household and personal assistance robots forecasts an utopist world of human--robot collaborative society. Along the road, one of the robotics community's major tasks is to work on the harmonization of trends, standardization of terms, interfaces and technologies. It is important to keep the scientific and social progress under control through sufficient public outreach and technology dissemination. Along those lines, the IEEE Robotics & Automation Society is sponsoring a working group entitled Ontologies for Robotics and Automation that will bridge cutting edge technology to users of the services---the general public. In this paper, the background of the project is presented, the definitions and examples of descriptive systems of relations are described, showing how ontologies can help to address the above challenges.

Image-guided surgery offers great advantages to surgeons through the possibility to track tools in 3D space and to navigate based on the virtual model of the patient. In the case of robot-assisted procedures, both the inherent accuracy of the system components and the quality of the registration procedures are critical to provide high precision treatment delivery. One of the major barriers towards more technology-integrated procedures is the fact that alterations in the operating room environment can fundamentally change the performance of the system, decrease the accuracy, and therefore pose significant danger to the patient. Surgical events from the control point of view may include motion of the robot, motion of the camera, or motion of the patient. The paper describes a new concept to treat these events, to track and automatically compensate for abrupt changes that may affect the accuracy of a robot-integrated interventional system. Our solution is to use all available information at a given time, including the intra-operative tracker's internal base frame, to distinguish between different surgical events. The concept has been developed and tested on the neurosurgical robot system at the Johns Hopkins University. Initial experiments performed on data recordings from simulated scenarios showed that the algorithm was able to correctly identify the cases.