Bilateral teleoperation systems are often used in safety–critical scenarios where human operators may interact with the environment remotely, as in robotic-assisted surgery or nuclear plant maintenance. Teleoperation's stability and transparency are the two most important properties to be satisfied, but they cannot be optimized independently since they are in contrast. This article presents a passive linear MPC control scheme to implement bilateral teleoperation that optimizes the tradeoff between stability and transparency (a.k.a. performance). First, we introduce a linear virtual energy tank with a novel energy-sharing policy, allowing us to define a passive linear model predictive control (MPC). Second, we provide conditions to guarantee the stability of the nonlinear closed-loop system. We validate the proposed approach in a teleoperation scheme using two 7-degree of freedom manipulators while performing an assembly task. This novel passivity-based bilateral teleoperation using linear MPC and linearized energy tank reduces the computational effort of existing passive nonlinear MPC controllers.

Reinforcement Learning (RL) algorithms can learn robotic control tasks from visual observations, but they often require a large amount of data, especially when the visual scene is complex and unstructured. In this paper, we explore how the agent’s knowledge of its shape can improve the sample efficiency of visual RL methods. We propose a novel method, Disentangled Environment and Agent Representations (DEAR), that uses the segmentation mask of the agent as supervision to learn disentangled representations of the environment and the agent through feature separation constraints. Unlike previous approaches, DEAR does not require reconstruction of visual observations. These representations are then used as an auxiliary loss to the RL objective, encouraging the agent to focus on the relevant features of the environment. We evaluate DEAR on two challenging benchmarks: Distracting DeepMind control suite and Franka Kitchen manipulation tasks. Our findings demonstrate that DEAR surpasses state-of-the-art methods in sample efficiency, achieving comparable or superior performance with reduced parameters. Our results indicate that integrating agent knowledge into visual RL methods has the potential to enhance their learning efficiency and robustness.

Path re-planning and repairing are key topics for robust planning and navigation in open dynamic environments, finding applications in various domains such as fleet control of Unmanned Ground Vehicles (UGVs) in warehouses. The use of UGVs in open and dynamic environments requires flexible cooperation between human operators and the UGV fleet within a shared environment. In this paper, we propose a local strategy to re-plan the path of robots encountering unexpected and dynamic obstacles. Specifically, starting from a given Multi-Agent path, we model the re-planning problem as a Markov Decision Process (MDP) considering a stochastic obstacle lifespan, and we propose two local approaches based on Monte-Carlo Tree Search to re-plan the path of the robots that encounter obstacles. We compare these approaches with traditional Multi-Agent Path Finding (MAPF) algorithms to obtain new collision-free paths when an obstacle is detected. The evaluation is performed in simulation using benchmarking instances of warehouses and experimentally in a research facility with a scaled-down Industry 4.0 production line.

Within the context of Robotic Minimally Invasive Surgery (R-MIS), we propose a novel linear model predictive controller formulation for the coordination of multiple autonomous robotic arms. The controller is synthesized by formulating a linear approximation of non-linear constraints, which allows the controller to be both computationally faster and better performing due to the increased prediction horizon allowed within the real-time control requirements for the proposed surgical application. The solution is validated under the expected constraints of a surgical scenario in which multiple laparoscopic tools must move and coordinate in a shared environment.

In this paper, we propose a novel bilateral teleoperation architecture that allows to optimally render the remote interaction force at the local side while guaranteeing a robustly stable behaviour. Stability is guaranteed by ensuring a proper energy exchange between the local and the remote sides. Desired performance is obtained by optimizing the way energy is exploited for generating the behaviour at each side. The effectiveness of the proposed architecture is experimentally validated on a torque-controlled manipulator and in a surgical scenario, using the da Vinci®Research Kit (dVRK).

We present a toolehain based on Docker and KubeEdge that enables containerization and orchestration of ROS-based robotic SW applications on heterogeneous and hierarchical HW architectures. The toolehain allows for verification of functional and real-time constraints through HW-in-the-loop simulation, and for automatic mapping exploration of the SW across Cloud-Server-Edge architectures. We present the results obtained for the deployment of a real case of study composed by an ORB-SLAM application combined to local/global planners with obstacle avoidance for a mobile robot navigation.

This paper addresses the generation of collision-free trajectories for the autonomous execution of assistive tasks in Robotic Minimally Invasive Surgery (R-MIS). The proposed approach takes into account geometric constraints related to the desired task, like for example the direction to approach the final target and the presence of moving obstacles. The developed motion planner is structured as a two-layer architecture: a global level computes smooth spline-based trajectories that are continuously updated using virtual potential fields; a local level, exploiting Dynamical Systems based obstacle avoidance, ensures collision free connections among the spline control points. The proposed architecture is validated in a realistic surgical scenario.

In this paper we present a novel strategy for motion planning of autonomous robotic arms in Robotic Minimally Invasive Surgery (R-MIS). We consider a scenario where several laparoscopic tools must move and coordinate in a shared environment. The motion planner is based on a Model Predictive Controller (MPC) that predicts the future behavior of the robots and allows to move them avoiding collisions between the tools and satisfying the velocity limitations. In order to avoid the local minima that could affect the MPC, we propose a strategy for driving it through a sequence of waypoints. The proposed control strategy is validated on a realistic surgical scenario.

Bilateral teleoperation systems allow the telepresence of an operator while working remotely. Such ability becomes crucial when dealing with critical environments like space, nuclear plants, rescue, and surgery. The main properties of a teleoperation system are the stability and the transparency which, in general, are in contrast and they cannot be fully achieved at the same time. In this paper, we will present a novel model predictive controller that implements a passivity-based bilateral teleoperation algorithm. Our solution mitigates the chattering issue arising when resorting to the energy tank (or reservoir) mechanism by forcing the passivity as a hard constraint on the system evolution.

In automated warehouses, path planning is a crucial topic to improve automation and efficiency. This kind of planning is usually computed off-line knowing the planimetry of the warehouse and the starting and target points of each agent. However, this global approach is not able to manage unexpected static/dynamic obstacles and other agents moving in the same area. For this reason in multi-robot systems global planners are usually integrated with local collision avoidance algorithms. In this paper we use the Voronoi diagram as global planner and the Velocity Obstacle (VO) method as collision avoidance algorithm. The goal of this paper is to extend such hybrid motion planner by enforcing mechanical constraints between agents in order to execute a task that cannot be performed by a single agent. We will focus on the cooperative task of carrying a payload, such as a bar. Two agents are constrained to move at the end points of the bar. We will improve the original algorithms by taking into account dynamically the constrained motion both at the global and at the collision avoidance level.

In this paper, a two-layer architecture for the bilateral teleoperation of multi-arms systems with communication delay is presented. We extend the single-master-single-slave two layer approach proposed in [1] by connecting multiple robots to a single energy tank. This allows to minimize the conservativeness due to passivity preservation and to increment the level of transparency that can be achieved. The proposed approach is implemented on a realistic surgical scenario developed within the EU-funded SARAS project.

Performing ultrasound procedures from a remote site is a challenging task since both a stable behavior, for the safety of the patient, and a high-level of usability, to exploit the sonographer's expertise, need to be guaranteed. Furthermore, a teleoperation system that provides such requirements has to deal with communication delays as well. To address this issue, we use the two-layer algorithm: a passivity-based bilateral teleoperation architecture able to guarantee stability despite unknown and time-varying delay. Its flexibility allows to implement different kinds of control laws. In a Tele-Echography system, the slave manipulator has to apply significant forces needed by the procedure whereas the haptic device at the master side should be very light to avoid tiring the operator. Therefore, the energy needed by these two robots to perform their movements is very different and the energy injected into the system by the operator is often not sufficient to implement the desired action at the slave side. Methods to overcome this problem require to perfectly know the dynamical models of the robots. The solution proposed in this paper does not require such knowledge and is based on properly scaling the energy exchanged between the master and the slave side. We show the effectiveness of this approach in a real setup using a TOUCH haptic device and a WAM Barrett robot holding an ultrasound probe.

The development of robotic systems with a certain level of autonomy to be used in critical scenarios, such as an operating room, necessarily requires a seamless integration of multiple state-of-the-art technologies. In this paper we propose a cognitive robotic architecture that is able to help an operator accomplish a specific task. The architecture integrates an action recognition module to understand the scene, a supervisory control to make decisions, and a model predictive control to plan collision-free trajectory for the robotic arm taking into account obstacles and model uncertainty. The proposed approach has been validated on a simplified scenario involving only a da VinciO surgical robot and a novel manipulator holding standard laparoscopic tools.

We propose an Active Object Recognition (AOR) strategy explicitly suited to work with robotic arms in human-robot cooperation scenarios. So far, AOR policies on robotic arms have focused on heterogeneous constraints, most of them related to classification accuracy, classification confidence, number of moves etc., discarding physical and energetic constraints a real robot has to fulfill. Our strategy overcomes this weakness by exploiting a POMDP-based AOR algorithm that explicitly considers manipulability and energetic terms in the planning optimization. The manipulability term avoids the robotic arm to get close to singularities, which require expensive and straining backtracking steps; the energetic term deals with the arm gravity compensation when in static conditions, which is crucial in AOR policies where time is spent in the classifier belief update, before doing the next movement. Several experiments have been carried out on a redundant, 7-DoF Panda arm manipulator, on a multi-object recognition task. This allows to appreciate the improvement of our solution with respect to other competitors evaluated on simulations only.

The use of robots in operating rooms improves safety and decreases patient recovery time and surgeon fatigue, but it introduces new potential hazards that can lead to severe injury or even the loss of human life. Thus, safety has been perceived as a crucial system property since the early days by the industry, the medical community, and the regulatory agents. In this article, we discuss the application of the mathematically rigorous technique known as Formal Verification to analyze the safety properties of a laser incision case study, and we assess its safe and predictable operation. Like all formal methods approaches, our analysis has three distinct components: a method to create a model of the system, a language to specify the properties, and a strategy to prove rigorously that the behavior of the model fulfills the desired properties. The model of the system takes the form of a hybrid automaton consisting of a discrete control part that operates in a continuous environment. The safety constraints are formalized as reachability properties of the hybrid automaton model, while the verification strategy exploits the capabilities of the tool A riadne to address the verification problem and answer the related questions ranging from safety to efficiency and effectiveness.

An adaptive optics (AO) system is composed of three key elements: a wave-front sensor (WFS) that detects the aberrations, a deformable mirror (DM) that provides the wavefront correction, and a closed-loop control system that elaborates the measurements acquired by the sensor and sends commands to the mirror. The control system can be implemented on a dedicated platform (e.g., field programmable gate array) or on general-purpose platforms (e.g., central processing unit (CPU), graphics processing unit). Dedicated hardware guarantees high performance but needs more development time and programming skills than general-purpose hardware, leading to a less maintainable system for the end user. The proposed solution aims to be a cost-effective multiplatform CPU-based flexible framework. The software, developed in C++ and using Eigen and Qt libraries, provides the tools to tune and control the AO system from wavefront measurement settings to controller parameters. A logging feature allows in-depth offline data analysis, while scripting enables execution of batch experiments. The AO system is tuned and evaluated by interfacing the WFS and DM with our software architecture. The results show that the proposed solution is able to correct the aberrations of a low- to medium-size single conjugate AO system, with a control frequency up to $\text{500}\,{\text{Hz}}$ and computational latency of $\text{40}\,\mu {\text{s}}$, using a consumer-grade notebook.

We apply formal methods to lay and streamline theoretical foundations to reason about Cyber-Physical Systems (CPSs) and cyber-physical attacks. We focus on integrity and DoS attacks to sensors and actuators of CPSs, and on the timing aspects of these attacks. Our contributions are threefold: (1) we define a hybrid process calculus to model both CPSs and cyber-physical attacks. (2) we define a threat model of cyber-physical attacks and provide the means to assess attack tolerance/vulnerability with respect to a given attack. (3) we formalise how to estimate the impact of a successful attack on a CPS and investigate possible quantifications of the success chances of an attack. We illustrate definitions and results by means of a non-trivial engineering application.

Use of cooperative multi vehicle team including aerial and ground vehicles has been growing rapidly over the last years, ranging from search and rescue to logistics. In this paper, we consider a cooperative landing task problem, where an unmanned aerial vehicle (UAV) must land on an unmanned ground vehicle (UGV) while such ground vehicle is moving in the environment to execute its own mission. To solve this challenging problem we consider the Petri Net Plans (PNPs) framework, an advanced planning specification framework, to effectively use different controllers in different conditions and to monitor the evolution of the system during mission execution so that the best controller is always used even in face of unexpected situations. Empirical simulation results show that our system can properly monitor the joint mission carried out by the UAV/UGV team, hence confirming that the use of a formal planning language significantly helps in the design of such complex scenarios.

Teleoperation systems allow humans to interact with remote environments by providing the operator with similar feedback as those s/he would experience at the remote site. Moreover, teleoperators may communicate contact force/torque information from the slave to the master side thus increasing the sense of telepresence of the human operator and improving task performance. When the kinesthetic coupling between operator and environment is enhanced by dynamic coupling, we refer to bilateral teleoperation. Unfortunately force feedback could destabilize a teleoperated system if the communication delay is not properly managed. For this reason many researchers are focusing nowadays on cutaneous feedback that does not affect stability but can still provide useful information to the operator. In this paper we design a cutaneous-feedback teleoperation system where the slave robot is a robotic hand. The software architecture is developed using the Robot Operating System (ROS). In ROS it is easy to integrate the different hardware components in a seamless way. The cutaneous feedback is provided by mini-motors whose vibration intensities are related with the forces measured by the pressure sensors embedded in the hand. Mini-motors are cheap devices that can be easily fastened to the operator's fingers via Velcro straps. The motion of the operator hand is calculated by the Leap Motion controller and mapped into the motion of the robotic hand. The integration of these devices allows to overcome the problem of developing complex and expensive haptic devices for the human hand.

This paper proposes an energy-efficient security-aware architecture for wireless control systems to be used in factory automation. We face deception attacks that corrupt commands and measurements in a smart way and with intermittent behavior to produce the highest damage without being discovered. The intrusion is hard to distinguish from normal disturbance. Furthermore, protection against attacks is energy-consuming and it would be desirable to activate protection only when needed. We propose packet-based selective encryption to reduce energy consumption, and to detect when an attack starts and ends. Since energy consumption depends also on packet transmission rate, especially during attacks, we also propose to adapt it according to instantaneous control performance.

Introducing some form of autonomy in robotic surgery is being considered by the medical community to better exploit the potential of robots in the operating room. However, significant technological steps have to occur before even the smallest autonomous task is ready to be presented to the regulatory authorities. In this paper, we address the initial steps of this process, in particular the development of control concepts satisfying the basic safety requirements of robotic surgery, i.e., providing the robot with the necessary dexterity and a stable and smooth behavior of the surgical tool. Two specific situations are considered: the automatic adaptation to changing tissue stiffness and the transition from autonomous to teleoperated mode. These situations replicate real-life cases when the surgeon adapts the stiffness of her/his arm to penetrate tissues of different consistency and when, due to an unexpected event, the surgeon has to take over the control of the surgical robot. To address the first case, we propose a passivity-based interactive control architecture that allows us to implement stable time-varying interactive behaviors. For the second case, we present a two-layered bilateral control architecture that ensures a stable behavior during the transition between autonomy and teleoperation and, after the switch, limits the effect of initial mismatch between master and slave poses. The proposed solutions are validated in the realistic surgical scenario developed within the EU-funded I-SUR project, using a surgical robot prototype specifically designed for the autonomous execution of surgical tasks like the insertion of needles into the human body.

In this paper we discuss the application of formal methods for the verification of properties of control systems designed for autonomous robotic systems. We illustrate our proposal in the context of surgery by considering the automatic execution of a simple action such as puncturing. To prove that a sequence of subtasks planned on pre-operative data can successfully accomplish the surgical operation despite model uncertainties, we specify the problem by using hybrid automata. We express the requirements of interest as questions about reachability properties of the hybrid automaton model. Then, we compare the different performance of current state-of-the art tools for reachability analysis of hybrid automata.

Several embedded system applications are used to control physical processes. Sensing, computation and actuation are combined thus involving a set of highly heterogeneous components, i.e., digital and analog hardware, software, energy sources, and external environment. Moreover, the growing use of networks contributes to introduce a further level of heterogeneity. All these aspects should be taken into account in the design process to find highly optimized solutions, therefore a Cyber-Physical System approach is needed. In particular, simulation is a key technique in the different design stages. However, the heterogeneity of components, together with the presence of the network, forces to adopt complex and slow co-simulation techniques to carry on the simulation of the entire system. This work aims at proposing SystemC as unified framework to model and simulate Networked Cyber-Physical Systems. Concerning the modeling of continuous-time components and a specific class of discrete-time components, the different Models of Computation provided by the Analog/Mixed-Signal (AMS) extension of SystemC are used. Regarding the network, SystemC and the SystemC Network Simulation Library are used to model communications at different abstraction levels. The accuracy and speed of different simulation alternatives are compared by the application to a networked control system.

Networked control systems are a well-known sub-set of cyber-physical systems in which the plant is controlled by sending commands through a digital packet-based network. Current control networks provide advanced channel access mechanisms to guarantee low delay on a limited fraction of packets (low-delay class) while the other packets (un-protected class) experience a higher delay which increases with channel utilization. We investigate the extension of model predictive control to choose both the command value and its assignment to one of the two classes according to the predicted state of the plant and the knowledge of network condition. Experimental results show that more commands are assigned to the low-delay class when either the tracking error is high or the network condition is bad.

This paper proposes a novel architecture for networked embedded systems which exploits a differentiated services approach to guarantee control performance even in case of time-varying network condition. Control commands are transmitted as high-priority packets when the plant behavior is far from the desired target or network condition does not assure the reliable and prompt reception of commands and measurements. The assignment of different priorities to packets belonging to the same flow (i.e., commands or measurements) may lead to out-of-sequence forwarding which may compromise the plant stability and the estimation of the state at the controller side. As far as we know this is the first work which solves such important issues by combining priority-based forwarding with the passivity mechanism to ensure stability. Moreover, we adopt an optimized filter for plant state estimation which takes into account the vector of the last commands used by the plant (updated through information received from the plant together with measurements) and the vector of the last measurements. Different packet marking strategies are compared: the best one leads to the same performance of random marking by using a smaller fraction of the high-priority bandwidth.

Ultrasound (US) guided biopsy is a medical procedure routinely performed in clinical practice. This task could be performed by robotic systems to improve the precision in the execution and then the safety for the patient. Both robotic and human procedures could greatly benefit from real-time localization of the needle in US images. This information could guide the robot or the specialists to the correct target point avoiding critical structures. Unfortunately US data provide very low quality images of the needle making this task quite complex, even more if you want to perform the localization on-line during the image acquisition. In this work we present a needle localization method able to extract the needle orientation and the tip position in real time from B-mode US images. To evaluate the performance of the algorithm in a precise way we use an optical tracking system to measure the position and the orientation of the needle and the US probe. In such a way the comparison is not human dependent (i.e. there are no radiologists manually selecting the needle tip) and fully repeatable. The results show an improvement in term of localization accuracy compared to previous works in literature.

Networked control systems are feedback systems where plant and controller are connected through lossy wired/wireless networks. To mitigate communication delays and packet losses different control solutions have been proposed. In this work the model predictive control (MPC) has been improved by introducing transmission options offering different probabilities of packet drops (high priority service and low priority service). This Differentiated Services architecture introduces Quality-of-Service (QoS) guarantees and can be used to jointly design the control command and the transmission strategy. A novel MPC-QoS controller is proposed and its design is obtained by solving a mixed integer quadratic problem.

The relevance of formal verification methods is widely recognized in the computer science and embedded systems community. Recently, such methods have been introduced also within the control community, to help designers in developing control architectures for complex robotics systems. Robotic systems typically mix continuous and discrete behaviors that cannot be modeled faithfully using neither continuous-only nor discrete-only formalisms. The interaction of continuous and discrete dynamics makes the formal treatment of this kind of systems computationally very demanding, and justifies the need of studying new methods and algorithms. In this paper, we outline the current state-of-the-art, and describe some open problems in verification, refinement and implementation of autonomous robotic systems. We motivate the relevance of our analysis by means of an Autonomous Robotic Surgery test case.