Safe and intuitive telemanipulation of multiple microrobots is limited by visual occlusions that compromise control and task success. This paper introduces a shared control framework that guarantees safety and visibility by formulating the control problem as a constrained optimization problem. Our framework combines Control Lyapunov Functions (CLFs) for operator-driven stability with High-Order Control Barrier Functions (HOCBFs) to enforce collision and occlusion avoidance, all resolved within a real-time Quadratic Program (QP). A key innovation is a rendering technique based on the QP’s Lagrange multipliers, which are used to decompose the optimal control solution. This method isolates the effects of individual constraints to provide the human operator with distinct, interpretable visuohaptic cues for navigation and avoidance. The architecture also includes a viewpoint optimization system with a virtual autonomous camera to maximize task visibility. The framework is validated via two user studies in simulation and a demonstration on a real-world electromagnetic microrobotics actuation system. Results confirm that our visuo-haptic shared control strategies significantly improve user performance and task completion rates, while the control design provably prevents the microrobots from colliding or entering occluded regions.

This paper proposes a method to robustify model predictive path integral (MPPI) control by directly taking into account the effects of parameter uncertainty into the controller formulation. Leveraging the recent notion of closed-loop state sensitivity, the proposed MPPI can consider the state sensitivity against parameter mismatch as a part of the system state, and consequently exploit this additional information to address the challenge of model mismatch in sampling-based model predictive control. Using an obstacle avoidance scenario, we demonstrate the use of our approach to control an aerial robot. We present an embedded implementation of our method, utilizing parallelization of computations on a GPU. Finally, we show the increased robustness of our approach over a standard MPPI controller through hardware-in-the-loop simulations and validate its embedded real-time properties.

Microrobotics implies actuation-related constraints that make safe telemanipulation particularly challenging. We present a haptic shared control system for electromagnetic-based telemanipulation of a pair of microrobots using a constrained optimization framework. Our contributions include: (1) a Quadratic Programming formulation with Control Lyapunov Functions and Control Barrier Functions, for safe and stable navigation in cluttered environments; (2) a shared control architecture, combining a haptic interface and simulation environment, to teleoperate the microrobots and enable micromanipulation capabilities; and (3) haptic shared control strategies offering visuo-haptic cues for task execution. The approach is validated through a user study, highlighting better navigation accuracy, control stability and task efficiency.

The precise manipulation of microrobots presents challenges arising from their small size and susceptibility to external disturbances. To address these challenges, we present the experimental evaluation of a haptic shared control teleoperation framework for the locomotion of multiple microrobots, relying on a kinesthetic haptic interface and a custom electromagnetic system. Six combinations of haptic and shared control strategies are evaluated during a safe 3D navigation scenario in a cluttered environment. 18 participants are asked to steer two spherical magnetic microrobots among obstacles to reach a predefined goal, under different conditions. For each condition, participants are provided with different obstacle avoidance and navigation guidance cues. Results show that providing assistance in avoiding obstacles guarantees safer performance, regardless if the assistance is autonomous or delivered through a haptic repulsive force. Moreover, autonomous obstacle avoidance also reduces the completion time by 30% compared to haptic obstacle avoidance and no obstacle avoidance cases, although haptic feedback is preferred by the users. Finally, providing haptic guidance towards the target improves by the 65% the positioning accuracy of the microrobots with respect to not providing this guidance. We also present some illustrative scenarios to generalize the presented haptic shared control strategies to arbitrary formations of N microrobots, while showing the effectiveness of the method for a clinical use-case of endovascular navigation in simulated environment. Note to Practitioners—The recent increasing interest in microrobotics arises from its potential applications in fields like medicine, manufacturing, and environmental monitoring, enabling highly precise control of minimally invasive tools. By enabling users to teleoperate microscale tools with partial autonomous support, these systems facilitate safe access to confined spaces, enhance task efficiency, and enable navigation in otherwise inaccessible environments. Our presented solution serves as an experimental platform to evaluate the efficacy of different combinations of tactile feedback and partial autonomy during safe navigation tasks, with potential applications spanning microsurgery, drug delivery, microscale manufacturing, and environmental remediation. Further practical adaptation of the system will require defining specific application objectives and specifications, along with potential modifications to the actuation system to accommodate environmental constraints of targeted scenarios.

Dynamic average estimation is a critical problem in multi-agent systems, enabling agents to collaboratively estimate time-varying signals using only local information exchange. Traditional model-based approaches often face challenges related to convergence speed and sensitivity to network topology changes. This article introduces a novel learning-based solution leveraging Gated Graph Neural Networks (GGNNs) for fast-convergent dynamic average estimation in a fully distributed manner. Taking advantage of the inherent structure of GGNNs, the proposed method models the estimation process as a distributed autoregressor, ensuring rapid convergence while maintaining stability. We incorporate a regularization term during training to enforce convergence guarantees and introduce an encoding–decoding mechanism to reduce communication overhead without sacrificing accuracy compared to standard GGNNs. Extensive numerical experiments demonstrate that our approach significantly outperforms conventional model-based estimators in terms of both convergence speed and precision, making it a promising alternative for multi-agent applications that require dynamic average estimation.

This article introduces a computationally efficient robust model predictive control (MPC) scheme for controlling nonlinear systems affected by parametric uncertainties in their models. The approach leverages the recent notion ofclosed-loop state sensitivityand the associated ellipsoidal tubes of perturbed trajectories for taking into account online time-varying restrictions on state and input constraints. This makes the MPC controller “aware” of potential additional requirements needed to cope with parametric uncertainty, thus significantly improving the tracking performance and success rates during navigation in constrained environments. One key contribution lies in the introduction of a computationally efficient robust MPC formulation with acomparable computational complexityto a standard MPC (i.e., an MPC not explicitly dealing with parametric uncertainty). An extensive simulation campaign is presented to demonstrate the effectiveness of the proposed approach in handling parametric uncertainties and enhancing task performance, safety, and overall robustness. Furthermore, we also provide an experimental validation that shows the feasibility of the approach in real-world conditions and corroborates the statistical findings of the simulation campaign. The versatility and efficiency of the proposed method make it therefore a valuable tool for real-time control of robots subject to nonnegligible uncertainty in their models.

In this work, we propose a framework for the distributed implementation of Quadratic Programs-based controllers, building upon and rectifying a significant limitation in a previously presented approach. The proposed framework is primarily motivated by the distributed implementation of Control Barrier Functions (CBFs), whose primary objective is to make minimal adjustments to a nominal controller while ensuring constraint satisfaction. By improving over some limitations in the current state-of-the-art, we are able to apply distributed CBFs to the problem of global connectivity maintenance in presence of communication and sensing constraints. Specifically, we consider the problem of preserving connectivity for a group of quadrotors with onboard sensors under distance and field of view constraints. Leveraging distributed control barrier functions, our approach maintains global graph connectivity while optimizing the performance of the desired task. Numerical simulations validate its effectiveness.

Nowadays ubiquitous robots must be adaptive and easy to use. To this end, dynamical system-based imitation learning plays an important role. In fact, it allows to realize stable and complex robotic tasks without explicitly coding them, thus facilitating the robot use. However, the adaptation capabilities of dynamical systems have not been fully exploited due to the lack of closed-loop implementations making use of visual feedback. In this regard, the integration of visual information allows higher flexibility to cope with environmental changes. This work presents a dynamical system-based imitation learning for visual servoing, based on the large projection task priority formulation. The proposed scheme enables complex and stable visual tasks, as demonstrated by a simulation analysis and experiments with a robotic manipulator.

Closed-loop state sensitivity [1], [2] is a recently introduced notion that can be used to quantify deviations of the closed-loop trajectory of a robot/controller pair against variations of uncertain parameters in the robot model. While local optimization techniques are used in [1], [2] to generate reference trajectories minimizing a sensitivity-based cost, no global planning algorithm considering this metric to compute collision-free motions robust to parametric uncertainties has yet been proposed. The contribution of this paper is to propose a global control-aware motion planner for optimizing a state sensitivity metric and producing collision-free reference motions that are robust against parametric uncertainties for a large class of complex dynamical systems. Given the prohibitively high computational cost of directly minimizing the state sensitivity using asymptotically optimal sampling-based tree planners, the proposed RRT*-based SAMP planner uses an appropriate steering method to first compute a (near) time-optimal and kinodynamically feasible trajectory that is then locally deformed to improve robustness and decrease its sensitivity to uncertainties. The evaluation performed on planar/full-3D quadrotor UAV models shows that the SAMP method produces low sensitivity robust solutions with a much higher performance than a planner directly optimizing the sensitivity.

We present a decentralized connectivity-maintenance algorithm for controlling a group of quadrotor UAVs with limited field of view (FOV) and not sharing a common reference frame for collectively expressing measurements and commands. This is in contrast to the vast majority of previous works on this topic which, instead, make the (simplifying) assumptions of omnidirectional sensing and presence of a common shared frame. For achieving this goal, we design a gradient-based connectivity-maintenance controller able to take into account the presence of a limited FOV. We also propose a novel (to our knowledge) decentralized estimator of the relative orientation among neighboring robots, which is a necessary quantity for correctly implementing the connectivity-maintenance action. We validate the framework in realistic simulations that show the effectiveness of our approach.

Energy tanks have gained popularity inside the robotics and control communities over the last years, since they represent a formidable tool to enforce passivity (and, thus, input/output stability) of a controlled robot, possibly interacting with uncertain environments. One weak point of passification strategies based on energy tanks concerns, however, their initialization. Indeed, a too large initial energy can cause practical unstable behaviors, while a too low initial energy level can prevent the correct execution of the task. This shortcoming becomes even more relevant in presence of uncertainties in the robot model and/or environment, since it may be hard to predict in advance the correct (safe) amount of initial tank energy for a successful task execution. In this paper we then propose a new strategy for addressing this issue. The recent notion of closed-loop state sensitivity is exploited to derive precise bounds (tubes) on the tank energy behavior by assuming parametric uncertainty in the robot model. These tubes are then exploited in a novel nonlinear optimization problem aiming at finding both the best trajectory and the minimal initial tank energy that allow executing a positioning task for any value of the uncertain parameters in a given range. The approach is finally validated via a statistical analysis in simulation and experiments on real robot hardware.

In this work, we exploit the recent notion of closed-loop state sensitivity to critically compare three typical controllers for a quadrotor UAV with the goal of evaluating the impact of controller choice, gain tuning and shape of the reference trajectory in minimizing the sensitivity of the closed-loop system against uncertainties in the model parameters. To this end, we propose a novel optimization problem that takes into account both the shape of the reference trajectory and the controller gains. We then run a large statistical campaign for comparing the performance of the three controllers which provides some interesting insight for the goal of increasing closed-loop robustness against parametric uncertainties.

We present in this paper a new open-source simulator based on CoppeliaSim and ROS for the popular Franka Emika Robot (FER) fully integrated in the ViSP ecosystem, a powerful library for Visual-Servoing. The simulator features a dynamic model that has been accurately identified from a real robot, leading to more realistic simulations. The C++ Api closely follows the ViSP class of the real robot allowing to narrow the gap between simulation code and real control software deployment. Conceived as a multipurpose research simulation platform, it is well suited for visual servoing applications as well as, in general, for any pedagogical purpose in robotics. All the software, models and CoppeliaSim scenes presented in this work are publicly available under free GPL-2.0 license.

In this research, we aim to answer the question: How to combine Closed-Loop State and Input Sensitivity-based with Observability-aware trajectory planning? These possibly op-posite optimization objectives can be used to improve trajectory control tracking and, at the same time, estimation performance. Our proposed novel Control & Observability-aware Planning (COP) framework is the first that uses these possibly opposing objectives in a Single-Objective Optimization Problem (SOOP) based on the Augmented Weighted Tchebycheff method to perform the balancing of them and generation of Bézier curve-based trajectories. Statistically relevant simulations for a 3D quadrotor unmanned aerial vehicle (UAV) case study produce results that support our claims and show the negative correlation between both objectives. We were able to reduce the positional mean integral error norm as well as the estimation uncertainty with the same trajectory to comparable levels of the trajectories optimized with individual objectives.

Neural Style Transfer (NST) refers to a class of algorithms able to manipulate an element, most often images, to adopt the appearance or style of another one. Each element is defined as a combination of Content and Style: the Content can be conceptually defined as the “what” and the Style as the “how” of said element. In this context, we propose a custom NST framework for transferring a set of styles to the motion of a robotic manipulator, e.g., the same robotic task can be carried out in an “angry”, “happy”, “calm”, or “sad” way. An autoencoder architecture extracts and defines the Content and the Style of the target robot motions. A Twin Delayed Deep Deterministic Policy Gradient (TD3) network generates the robot control policy using the loss defined by the autoencoder. The proposed Neural Policy Style Transfer TD3 NPST3 alters the robot motion by introducing the trained style. Such an approach can be implemented either offline, for carrying out autonomous robot motions in dynamic environments, or online, for adapting at runtime the style of a teleoperated robot. The considered styles can be learned online from human demonstrations. We carried out an evaluation with human subjects enrolling 73 volunteers, asking them to recognize the style behind some representative robotic motions. Results show a good recognition rate, proving that it is possible to convey different styles to a robot using this approach.

This paper considers a multi-robot team tasked with monitoring an environmental field of interest over long time horizons. The approach is based on a control-theoretic measure of the information collected by the robots, namely a norm of the constructability Gramian. This measure is leveraged in order to learn a distributed multi-robot control policy using the reinforcement learning paradigm. The learned policy is then combined with energy constraints using the constraint-driven control framework in order to achieve persistent environmental monitoring. The proposed approach is tested in a simulated multi-robot persistent environmental monitoring scenario where a team of robots with limited availability of energy is to be controlled in a coordinated fashion in order to estimate the concentration of a gas diffusing in the environment.

In this work we present our results on dynamic visual servoing for the case of moving targets while also exploring the possibility of using such a controller for interaction with the environment. We illustrate the derivation of a feature space impedance controller for tracking a moving object as well as an Extended Kalman Filter based on the visual servoing kinematics for increasing the rate of the visual information and estimating the target velocity for both the cases of PBVS and IBVS with image point features. Simulations are carried out to validate the estimator performance during a Peg-in-Hole insertion task with a moving part. Experiments are also conducted on a real redundant manipulator with a low-cost wrist-mounted camera. Details on several implementation issues encountered during implementation are also discussed.

We present a decentralized connectivity-maintenance control framework for a heterogeneous human–robot team. The algorithm is able to manage a team composed of an arbitrary number of mobile robots (drones and ground robots in our case) and humans, for collaboratively achieving exploration and patrolling tasks. Differently from other works on the subject, here the human user physically becomes part of the team, moving in the same environment of the robots and receiving information about the team connectivity through wearable haptics or audio feedback. Although human explores the environment, robots move so as to keep the team connected via a connectivity-maintenance algorithm; at the same time, each robot can also be assigned with a specific target to visit. We carried out three human subject experiments, both in virtual and real environments. Results show that the proposed approach is effective in a wide range of scenarios. Moreover, providing either haptic or audio feedback for conveying information about the team connectivity significantly improves the performance of the considered tasks, although users significantly preferred receiving haptic stimuli w.r.t. the audio ones. Note to Practitioners—Exploration, patrolling, and search-and-rescue are highly dynamic and unstructured scenarios. When considering the operative conditions of such environments, the benefits of multirobot systems are evident. Most tasks can be carried out faster and more robustly by a team of robots with respect to a single unit. There are also situations explicitly requiring the presence of a multirobot team, e.g., using one drone for surveillance of the ground team and one ground mobile robot for carrying supplies. Of course, if the operator(s) in charge of the operation could share the same environment of the robots (i.e., be together with the robots in the field), they would be provided with a level of situational awareness that no teleoperation technology can match as of today. This work presents a framework for controlling heterogeneous teams composed of one human operator and an arbitrary number of aerial and ground mobile robots. The operator moves together with the robotic team and, at the same time, he or she receives meaningful information about the status of the formation. The algorithm only uses the relative position of the drones and humans with respect to each other, and all computations are designed in a decentralized fashion. Decentralization avoids relying on any absolute positioning system (e.g., GPS) or centralized command centers. These features make the proposed framework ready for deployment in different high-impact applications, such as in surveillance, search-and-rescue, and disaster response scenarios.

This article proposes a shared-control teleoperation architecture for robot manipulators transporting an object on a tray. Differently from many existing studies about remotely operated robots with firm grasping capabilities, we consider the case in which, in principle, the object can break its contact with the robot end-effector. The proposed shared-control approach automatically regulates the remote robot motion commanded by the user and the end-effector orientation to prevent the object from sliding over the tray. Furthermore, the human operator is provided with haptic cues informing about the discrepancy between the commanded and executed robot motion, which assist the operator throughout the task execution. We carried out trajectory tracking experiments employing an autonomous 7-degree-of-freedom (DoF) manipulator and compared the results obtained using the proposed approach with two different control schemes (i.e., constant tray orientation and no motion adjustment). We also carried out a human-subjects study involving 18 participants in which a 3-DoF haptic device was used to teleoperate the robot linear motion and display haptic cues to the operator. In all experiments, the results clearly show that our control approach outperforms the other solutions in terms of sliding prevention, robustness, commands tracking, and user’s preference.

Research on robotic manipulation of fragile, compliant objects, such as food items, is gaining traction due to its game-changing potential within the food production and retailing sectors, currently characterized by manually intensive and highly repetitive tasks. Food products exhibit high levels of frailness, biological variation, and complex 3-D shapes and textures. For these reasons, introducing greater levels of robotic automation in the food and agricultural sectors remains an important challenge. This article addresses this challenge by developing a human-centered, haptic-based, learning from demonstration (LfD) policy that enables pretrained autonomous grasping of food items using an anthropomorphic robotic system. The policy combines data from teleoperation and direct human manipulation of objects, embodying human intent and interaction areas of significance. We evaluated the proposed solution against a recent state-of-the-art LfD policy as well as against two standard impedance controller techniques. Results show that the proposed policy performs significantly better than the other considered techniques, leading to high grasping success rates while guaranteeing the integrity of the food at hand.

In this paper we extend the previously introduced notion of closed-loop state sensitivity by introducing the concept of input sensitivity and by showing how to exploit it in a trajectory optimization framework. This allows to generate an optimal reference trajectory for a robot that minimizes the state and input sensitivities against uncertainties in the model parameters, thus producing inherently robust motion plans. We parametrize the reference trajectories with Béziers curves and discuss how to consider linear and nonlinear constraints in the optimization process (e.g., input saturations). The whole machinery is validated via an extensive statistical campaign that clearly shows the interest of the proposed methodology.

This article presents the design of a decentralized connectivity-maintenance algorithm for the teleoperation of a team of multiple UAVs, together with an extensive human subject evaluation in virtual and real environments. The proposed connectivity-maintenance algorithm enhances earlier works by improving their applicability, safety, effectiveness, and ease of use, by including: 1) an airflow-avoidance behavior that avoids stack downwash phenomena in rotor-based aerial robots; 2) a consensus-based action for enabling fast displacements with minimal topology changes by having all follower robots moving at the leader’s velocity; 3) an automatic decrease of the minimum degree of connectivity, enabling an intuitive and dynamic expansion/compression of the formation; and 4) an automatic detection and resolution of deadlock configurations, i.e., when the robot leader cannot move due to counterbalancing connectivity- and external-related inputs. We also devised and evaluated different interfaces for teleoperating the team as well as different ways of receiving information about the connectivity force acting on the leader. The results of two human subject experiments show that the proposed algorithm is effective in various situations. Moreover, using haptic feedback to provide information about the team connectivity outperforms providing both no feedback at all and sensory substitution via visual feedback.Note to Practitioners—The control of one drone is usually performed with a remote controller (similar to a joypad) that uses radio-wave signals. When controlling more than one drone, even a simple task, such as moving the whole team around, becomes very challenging. Developing an easy, yet efficient way to impart commands to a formation of drones is necessary to achieve any complex task. This article proposes a framework to control a fleet of drones (quadrotors) in an intuitive way while receiving meaningful and effective information on the state of the formation. The proposed technique does not rely on any absolute positioning system (e.g., GPS) or centralized command center. Instead, it only uses the relative position of the drones with respect to each other, and all computations are designed in a decentralized fashion. These features make the proposed framework ready for deployment in different high-impact applications, such as in surveillance, search-and-rescue, and disaster response scenarios.

Robot teleoperation is widely used for several hazardous applications. To increase teleoperator capabilities shared-control methods can be employed. In this paper, we present a passive task-prioritized shared-control method for remote telemanipulation of redundant robots. The proposed method fuses the task-prioritized control architecture with haptic guidance techniques to realize a shared-control framework for teleoperation systems. To preserve the semi-autonomous telerobotic system safety, passivity is analyzed and an energy-tanks passivity-based controller is developed. The proposed theoretical results are validated through experiments involving a real haptic device and a simulated slave robot.

Robot-assisted cutting is considered an important task in several fields, such as robotic surgery, nuclear decommissioning, waste management, and manufacturing. Despite the complex dexterity requirements of cutting tasks, very simple mechanically-linked master-slave manipulators still dominate many of the above fields (e.g., nuclear robotics). Moreover, even when more dexterous manipulators are available (e.g., in robot-assisted surgery), the employed systems show little or no autonomy, delegating all control to the experience of the human operator. To ameliorate this situation, we present two haptic shared-control approaches for robotic cutting. They are designed to assist the human operator by enforcing different nonholonomic-like constraints representative of the cutting kinematics. To validate our approach, we carried out a human-subject experiment in a real cutting scenario. We compared our shared-control techniques with each other and with a standard haptic teleoperation scheme. Results show the usefulness of assisted control schemes in complex applications such as cutting. However, they also show a discrepancy between objective and subjective metrics.

This article proposes an online optimal active perception strategy for differentially flat systems meant to maximize the information collected via the available measurements along the planned trajectory. The goal is to generate online a trajectory that minimizes the maximum state estimation uncertainty provided by the employed observer. To quantify the richness of the acquired information about the current state, the smallest eigenvalue of the constructibility Gramian is adopted as a metric. In this article, we use B-splines for parametrizing the trajectory of the flat outputs and we exploit a constrained gradient descent strategy for optimizing online the location of the B-spline control points in order to actively maximize the information gathered over the whole planning horizon. To show the effectiveness of our method in maximizing the estimation accuracy, we consider two case studies involving a unicycle and a quadrotor that need to estimate their poses while measuring two distances w.r.t. two fixed landmarks. Concurrent estimation of calibration/environment parameters is also considered for illustrating how the proposed method copes with instances of active self-calibration and map building.

The goal of this paper is to increase the estimation performance of an Extended Kalman Filter for a nonlinear differentially flat system by planning trajectories able to maximize the amount of information gathered by onboard sensors in presence of both process and measurement noises. In a previous work, we presented an online gradient descent method for planning optimal trajectories along which the smallest eigenvalue of the Observability Gramian (OG) is maximized. As the smallest eigenvalue of the OG is inversely proportional to the maximum estimation uncertainty, its maximization reduces the maximum estimation uncertainty of any estimation algorithm employed during motion. However, the OG does not consider the process noise that, instead, in several applications is far from being negligible. For this reason, this paper proposes a novel solution able to cope with non-negligible process noise: this is achieved by minimizing the largest eigenvalue of the a posteriori covariance matrix obtained by solving the Continuous Riccati Equation as a measure of the total available information. This minimization is expected to maximize the information gathered by the outputs while, at the same time, limiting as much as possible the negative effects of the process noise. We apply our method to a unicycle robot. The comparison between the novel method and the one of our previous work (which did not consider process noise) shows significant improvements in the obtained estimation accuracy.

In this paper we propose a novel general method to let a dynamical system fulfil at best a control task when the nominal parameters are not perfectly known. The approach is based on the introduction of the novel concept of closed-loop sensitivity, a quantity that relates parameter variations to deviations of the closed-loop trajectory of the system/controller pair. This new definition takes into account the dependency of the control inputs from the system states and nominal parameters as well as from the controller dynamics. The reference trajectory to be tracked is taken as optimization variable, and the dynamics of both the sensitivity and of its gradient are computed analytically along the system trajectories. We then show how this computation can be effectively exploited for solving trajectory optimization problems aimed at generating a reference trajectory that minimizes a norm of the closed-loop sensitivity. The theoretical results are validated via an extensive campaign of Monte Carlo simulations for two relevant robotic systems: a unicycle and a quadrotor UAV.

Robotic telemanipulators are already widely used in nuclear decommissioning sites for handling radioactive waste. However, currently employed systems are still extremely primitive, making the handling of these materials prohibitively slow and ineffective. As the estimated cost for the decommissioning and clean-up of nuclear sites keeps rising, it is clear that one would need faster and more effective approaches. Towards this goal, in this paper we present the user evaluation of a recently proposed haptic-enabled shared-control architecture for telemanipulation. An autonomous algorithm regulates a subset of the slave manipulator degrees of freedom (DoF) in order to help the human operator in grasping an object of interest. The human operator can then steer the manipulator along the remaining null-space directions with respect to the main task by acting on a grounded haptic interface. The haptic cues provided to the operator are designed in order to inform about the feasibility of the user's commands with respect to possible constraints of the robotic system. In this paper we compared this shared-control architecture against a classical 6-DOF teleoperation approach in a real scenario by running experiments with 10 subjects. The results clearly show that the proposed shared-control approach is a viable and effective solution for improving currently-available teleoperation systems in remote telemanipulation tasks.

Shared control is a key technology for various robotic applications in which a robotic system and a human operator are meant to collaborate efficiently. In order to achieve efficient task execution in shared control, it is essential to predict the desired behavior for a given situation or context in order to simplify the control task for the human operator. This prediction is obtained by exploiting Learning from Demonstration (LfD), which is a popular approach for transferring human skills to robots. We encode the demonstrated behavior as trajectory distributions and generalize the learned distributions to new situations. The goal of this paper is to present a shared control framework that uses learned expert distributions to gain more autonomy. Our approach controls the balance between the controller's autonomy and the human preference based on the distributions of the demonstrated trajectories. Moreover, the learned distributions are autonomously refined from collaborative task executions, resulting in a master-slave system with increasing autonomy that requires less user input with an increasing number of task executions. We experimentally validated that our shared control approach enables efficient task executions. Moreover, the conducted experiments demonstrated that the developed system improves its performances through interactive task executions with our shared control.

We propose in this paper a new active perception scheme based on Model Predictive Control under constraints for generating a sequence of visual servoing tasks. The proposed control scheme is used to compute the motion of a camera whose task is to successively observe a set of robots for measuring their position and improving the accuracy of their localization. This method is based on the prediction of an uncertainty model (due to actuation and measurement noise) for determining which robot has to be observed by the camera. Simulation results are presented for validating the approach.

Nowadays, one of the largest environmental challenges that European countries must face consists in dealing with the past half century of nuclear waste. In order to optimize maintenance costs, nuclear waste must be sorted, segregated and stored according to its radiation level. Towards this end, in [1] we have recently proposed a visual-based shared control architecture meant to facilitate a human operator in controlling two remote robotic arms (one equipped with a gripper and another with a camera) during remote manipulation tasks of nuclear waste via a master device. The operator could then receive force cues informative of the feasibility of her/his motion commands during the task execution. The strategy presented in [1], albeit effective, suffers however from a locality issue since the operator can only provide instantaneous velocity commands (in a suitable task space), and receive instantaneous force feedback cues. On the other hand, the ability to `steer' a whole future trajectory in task space, and to receive a corresponding integral force feedback along the whole planned trajectory (because of any constraint of the considered system), could significantly enhance the operator's performance, especially when dealing with complex manipulation tasks. The aim of this work is to then extend [1] towards a planning-based shared control architecture able to take into account the mentioned requirements. A human/hardware-in-the-loop experiment with simulated slave robots and a real master device is reported for demonstrating the feasibility and effectiveness of the proposed approach.

This paper deals with the problem of active sensing control for nonlinear differentially flat systems. The objective is to improve the estimation accuracy of an observer by determining the inputs of the system that maximise the amount of information gathered by the outputs over a time horizon. In particular, we use the Observability Gramian (OG) to quantify the richness of the acquired information. First, we define a trajectory for the flat outputs of the system by using B-Spline curves. Then, we exploit an online gradient descent strategy to move the control points of the B-Spline in order to actively maximise the smallest eigenvalue of the OG over the whole planning horizon. While the system travels along its planned (optimized) trajectory, an Extended Kalman Filter (EKF) is used to estimate the system state. In order to keep memory of the past acquired sensory data for online re-planning, the OG is also computed on the past estimated state trajectories. This is then used for an online replanning of the optimal trajectory during the robot motion which is continuously refined by exploiting the state estimation obtained by the EKF. In order to show the effectiveness of our method we consider a simple but significant case of a planar robot with a single range measurement. The simulation results show that, along the optimal path, the EKF converges faster and provides a more accurate estimate than along other possible (non-optimal) paths.

This paper considers the problem of controlling a formation of quadrotor UAVs equipped with onboard cameras with the goal of maintaining bearing rigidity during motion despite the presence of several sensing constraints, that is, minimum/maximum range, limited camera field of view, and possible occlusions caused by the agents of the formation. To this end, a decentralized gradient-based control action is developed, based on a suitable ‘degree of infinitesimal rigidity’ linked to the spectral properties of the bearing rigidity matrix. The approach is then experimentally validated with five quadrotor UAVs.

Teleoperating a quadrotor equipped with a monocular camera for exploring a wide area in search of something has become a common practice in many application scenarios (e.g. search and rescue). In order to efficiently plan operations, estimating the 3D pose of a point of interest is as important as detecting it. In this paper we propose a novel bilateral teleoperation architecture where an estimation scheme is exploited for recovering the position of a set of visual features while an operator steers the motion of the quadrotor UAV. The operator acts on a force-feedback master device that produces force cues meant to suggest where to drive the quadrotor for improving the convergence rate of the estimation process. The effectiveness of the proposed teleoperation strategy is validated by means of hardware in the loop simulations.

This paper presents image-based navigation from an image memory using a combination of line segments and feature points. The environment is represented by a set of key images, which are acquired during a prior mapping phase that defines the path to be followed during the navigation. The switching of key images is done exploiting the common line segments and feature points between the current acquired image and the nearby key images. Based on the key images and the current image, a control law is derived for computing the rotational velocity of a mobile robot during its visual navigation. Using our approach, real-time navigation has been performed in real indoor environment with a Pioneer 3-DX equipped with an on-board perspective camera and the humanoid robot Pepper without the need of accurate mapping and localization nor of 3D reconstruction. We also show that the combination of points and lines increases the number of features that helps in robust and successful navigation especially in those regions where few points or lines can be detected and tracked/matched.

This paper addresses the problem of mixed initiative, shared control for master-slave grasping and manipulation. We propose a novel system, in which an autonomous agent assists a human in teleoperating a remote slave arm/gripper, using a haptic master device. Our system is designed to exploit the human operator's expertise in selecting stable grasps (still an open research topic in autonomous robotics). Meanwhile, a-priori knowledge of: i) the slave robot kinematics, and ii) the desired post-grasp manipulative trajectory, are fed to an autonomous agent which transmits force cues to the human, to encourage maximally manipulable grasp pose selections. Specifically, the autonomous agent provides force cues to the human, during the reach-to-grasp phase, which encourage the human to select grasp poses which maximise manipulation capability during the post-grasp object manipulation phase. We introduce a task-oriented velocity manipulability cost function (TOV), which is used to identify the maximum kinematic capability of a manipulator during post-grasp motions, and feed this back as force cues to the human during the pre-grasp phase. We show that grasps which minimise TOV result in significantly reduced control effort of the manipulator, compared to other feasible grasps. We demonstrate the effectiveness of our approach by experiments with both real and simulated robots.

In this paper, we address the problem of using a camera with limited field of view for controlling the motion of a quadrotor in aggressive flight regimes. We present a minimum time trajectory planning method that guarantees visibility of the image features while allowing the robot to undertake aggressive motions for which the usual near-hovering assumption is violated. We exploit differential flatness and B-Splines to parametrize the system trajectories in terms of a finite number of control points, which can then be optimized by Sequential Quadratic Programming (SQP). The control strategy is similar to a Receding Horizon Control (RHC) approach for modifying online the reference trajectory in order to account for noise, disturbances and any non-modeled effect. The algorithm is validated in a physically realistic simulation environment.

This paper presents a complete framework for image-based navigation from an image memory that exploits mutual information and does not need any feature extraction, matching or any 3D information. The navigation path is represented by a set of automatically selected key images obtained during a prior learning phase. The shared information (entropy) between the current acquired image and nearby key images is exploited to switch key images during navigation. Based on the key images and the current image, the control law proposed by [1] is used to compute the rotational velocity of a mobile robot during its qualitative visual navigation. Using our approach, real-time navigation has been performed inside a corridor and inside a room with a Pioneer 3-DX equipped with an on-board perspective camera without the need of accurate mapping and localization.

Cleaning up the past half century of nuclear waste represents the largest environmental remediation project in the whole Europe. Nuclear waste must be sorted, segregated and stored according to its radiation level in order to optimize maintenance costs. The objective of this work is to develop a shared control framework for remote manipulation of objects using visual information. In the presented scenario, the human operator must control a system composed of two robotic arms, one equipped with a gripper and the other one with a camera. In order to facilitate the operator's task, a subset of the gripper motion are assumed to be regulated by an autonomous algorithm exploiting the camera view of the scene. At the same time, the operator has control over the remaining null-space motions w.r.t. the primary (autonomous) task by acting on a force feedback device. A novel force feedback algorithm is also proposed with the aim of informing the user about possible constraints of the robotic system such as, for instance, joint limits. Human/hardware-in-the-loop experiments with simulated slave robots and a real master device are finally reported for demonstrating the feasibility and effectiveness of the approach.

This paper considers the problem of controlling a formation of quadrotor UAVs equipped with onboard cameras able to measure relative bearings in their local body frames w.r.t. neighboring UAVs. The control goal is twofold: (i) steering the agent group towards a formation defined in terms of desired bearings, and (ii) actuating the group motions in the `null-space' of the current bearing formation. The proposed control strategy relies on an extension of the rigidity theory to the case of directed bearing frameworks in ℝ3×S1. This extension allows to devise a decentralized bearing controller which, unlike most of the present literature, does not need presence of a common reference frame or of reciprocal bearing measurements for the agents. Simulation and experimental results are then presented for illustrating and validating the approach.

In this paper, we propose a novel decentralized active perception strategy that maximizes the convergence rate in estimating the (unmeasurable) formation scale in the context of bearing-based formation localization for robots evolving in ℝ3× S1. The proposed algorithm does not assume presence of a global reference frame and only requires bearing-rigidity of the formation (for the localization problem to admit a unique solution), and presence of (at least) one pair of robots in mutual visibility. Two different scenarios are considered in which the active scale estimation problem is treated either as a primary task or as a secondary objective with respect to the constraint of attaining a desired bearing formation. The theoretical results are validated by realistic simulations.

One way to deal with occlusions or loss of tracking of the visual features used for visual servoing tasks is to predict the feature behavior in the image plane when the measurements are missing. Different prediction and correction methods have already been proposed in the literature. The purpose of this paper is to compare and experimentally validate some of these methods for eye-in-hand and eye-to-hand configurations. In particular, we show that a correction based both on the image and the camera/target pose provides the best results.

The selection of a suitable set of visual features for an optimal performance of closed-loop visual control or Structure from Motion (SfM) schemes is still an open problem in the visual servoing community. For instance, when considering integral region-based features such as image moments, only heuristic, partial, or local results are currently available for guiding the selection of an appropriate moment set. The goal of this paper is to propose a novel learning strategy able to automatically optimize online the shape of a given class of image moments as a function of the observed scene for improving the SfM performance in estimating the scene structure. As case study, the problem of recovering the (unknown) 3D parameters of a planar scene from measured moments and known camera motion is considered. The reported simulation results fully confirm the soundness of the approach and its superior performance over more consolidated solutions in increasing the information gain during the estimation task.

In this paper we experimentally validate and compare three different methods for estimating the 3D parameters of a planar scene from a (possibly time-varying) set of feature points acquired by a moving monocular camera. The first method, based on the classical decomposition of the homography matrix, is meant to serve as a baseline condition classically used in many previous works. The other two methods exploit an active Structure from Motion (SfM) scheme for either extracting the plane from the reconstructed 3D position of all the tracked points, or for directly estimating the plane parameters by considering a set of discrete image moments as visual input. The possible loss/gain of point features during the camera motion is considered in all three methods by, in particular, introducing a suitable weighting strategy for the image moment case. Finally, the results of an experimental validation are presented with a comparative discussion of the pros/cons of the three methods.

Incremental Structure from Motion (SfM) algorithms require, in general, precise knowledge of the camera linear and angular velocities in the camera frame for estimating the 3D structure of the scene. Since an accurate measurement of the camera own motion may be a non-trivial task in several robotics applications (for instance when the camera is onboard a UAV), we propose in this paper an active SfM scheme fully independent from the camera angular velocity. This is achieved by considering, as visual features, some rotational invariants obtained from the projection of the perceived 3D points onto a virtual unitary sphere (unified camera model). This feature set is then exploited for designing a rotation-free active SfM algorithm able to optimize online the direction of the camera linear velocity for improving the convergence of the structure estimation task. As case study, we apply our framework to the depth estimation of a set of 3D points and discuss several simulations and experimental results for illustrating the approach.

Plane detection and estimation from visual data is a classical problem in robotic vision. In this work we propose a novel active strategy in which a monocular camera tries to determine whether a set of observed point features belongs to a common plane, and, if so, what are the associated plane parameters. The active component of the strategy imposes an optimized camera motion (as a function of the observed scene) able to maximize the convergence in estimating the scene structure. Based on this strategy, two methods are then proposed to solve the plane estimation task: a classical solution exploiting the homography constraint (and, thus, almost completely based on image correspondances across distant frames), and an alternative method fully taking advantage of the scene structure estimated incrementally during the camera motion. The two methods are extensively compared in several case studies by discussing the various pros/cons.

A new framework for semi-autonomous path planning for mobile robots that extends the classical paradigm of bilateral shared control is presented. The path is represented as a B-spline and the human operator can modify its shape by controlling the motion of a finite number of control points. An autonomous algorithm corrects in real time the human directives in order to facilitate path tracking for the mobile robot and ensures i) collision avoidance, ii) path regularity, and iii) attraction to nearby points of interest. A haptic feedback algorithm processes both human's and autonomous control terms, and their integrals, to provide an information of the mismatch between the path specified by the operator and the one corrected by the autonomous algorithm. The framework is validated with extensive experiments using a quadrotor UAV and a human in the loop with two haptic interfaces.

In this paper we propose a solution for coupling the execution of a visual servoing task with a recently developed active Structure from Motion strategy able to optimize online the convergence rate in estimating the (unknown) 3D structure of the scene. This is achieved by suitably modifying the robot trajectory in the null-space of the servoing task so as to render the camera motion `more informative' w.r.t. the states to be estimated. As a byproduct, the better 3D structure estimation also improves the evaluation of the servoing interaction matrix which, in turn, results in a better closed-loop convergence of the task itself. The reported experimental results support the theoretical analysis and show the benefits of the method.

Structure estimation from motion (SfM) is a classical and well-studied problem in computer and robot vision, and many solutions have been proposed to treat it as a recursive filtering/estimation task. However, the issue of actively optimizing the transient response of the SfM estimation error has not received a comparable attention. In this paper, we provide an experimental validation of a recently proposed nonlinear active SfM strategy via two concrete applications: 3D structure estimation for a spherical and a cylindrical target. The experimental results fully support the theoretical analysis and clearly show the benefits of the proposed active strategy. Indeed, by suitably acting on the camera motion and estimation gains, it is possible to assign the error transient response and make it equivalent to that of a reference linear second-order system with desired poles.

Predicting the behavior of visual features on the image plane over a future time horizon is an important possibility in many different control problems. For example when dealing with occlusions (or other constraints such as joint limits) in a classical visual servoing loop, or also in the more advanced model predictive control schemes recently proposed in the literature. Several possibilities have been proposed to perform the initial correction step for then propagating the visual features by exploiting the measurements currently available by the camera. But the predictions proposed so far are inaccurate in situations where the depths of the tracked points are not correctly estimated. We then propose in this paper a new correction strategy which tries to directly correct the relative pose between the camera and the target instead of only adjusting the error on the image plane. This correction is then analysed and compared by evaluating the corresponding improvements in the feature prediction phase.

In this paper, we illustrate the application of a nonlinear active structure estimation from motion (SfM) strategy to three problems, namely 3-D structure estimation for 1) a point, 2) a sphere, and 3) a cylinder. In all three cases, an appropriate parameterization reduces the problem to the estimation of a single quantity. Knowledge of this estimated quantity and of the available measurements allows for then retrieving the full 3-D structure of the observed objects. Furthermore, in the point feature case, two different parameterizations based on either a planar or a spherical projection model are critically compared. Indeed, the two models yield, somehow unexpectedly, to different convergence properties for the SfM estimation task. The reported simulative and experimental results fully support the theoretical analysis and clearly show the benefits of the proposed active estimation strategy, which is in particular able to impose a desired transient response to the estimation error equivalent to that of a reference linear second-order system with assigned poles.

In this work we present a novel concept of a quadrotor UAV with tilting propellers. Standard quadrotors are limited in their mobility because of their intrinsic underactuation (only 4 independent control inputs vs. their 6-dof pose in space). The quadrotor prototype discussed in this paper, on the other hand, has the ability to also control the orientation of its 4 propellers, thus making it possible to overcome the aforementioned underactuation and behave as a fully-actuated flying vehicle. We first illustrate the hardware/software specifications of our recently developed prototype, and then report the experimental results of some preliminary, but promising, flight tests which show the capabilities of this new UAV concept.

This paper presents a novel bilateral controller that allows to stably teleoperate the degree of connectivity in the mutual interaction between a remote group of mobile robots considered as the slave-side. A distributed leader-follower scheme allows the human operator to command the overall group motion. The group autonomously maintains the connectivity of the interaction graph by using a decentralized gradient descent approach applied to the Fiedler eigenvalue of a properly weighted Laplacian matrix. The degree of connectivity, and then the flexibility, of the interaction graph can be finely tuned by the human operator through an additional bilateral teleoperation channel. Passivity of the overall system is theoretically proven and extensive human/hardware in-the-loop simulations are presented to empirically validate the theoretical analysis.

In this paper, an adaptive trajectory tracking controller for quadrotor MAVs is presented. The controller exploits the common assumption of a faster orientation dynamics w.r.t. the translational one, and is able to asymptotically compensate for parametric uncertainties (e.g., displaced center of mass), as well as external disturbances (e.g., wind). The good performance of the proposed controller is then demonstrated by means of an extensive experimental evaluation performed with a commercially-available quadrotor MAV.

For the purpose of autonomous UAV flight control, cameras are ubiquitously exploited as a cheap and effective onboard sensor for obtaining non-metric position or velocity measurements. Since the metric scale cannot be directly recovered from visual input only, several methods have been proposed in the recent literature to overcome this limitation by exploiting independent `metric' information from additional onboard sensors. The flexibility of most approaches is, however, often limited by the need of constantly tracking over time a certain set of features in the environment, thus potentially suffering from possible occlusions or loss of tracking during flight. In this respect, in this paper we address the problem of estimating the scale of the observed linear velocity in the UAV body frame from direct measurement of the instantaneous (and non-metric) optical flow, and the integration of an onboard Inertial Measurement Unit (IMU) for providing (metric) acceleration readings. To this end, two different estimation techniques are developed and critically compared: a standard Extended Kalman Filter (EKF) and a novel nonlinear observer stemming from the adaptive control literature. Results based on simulated and real data recorded during a quadrotor UAV flight demonstrate the effectiveness of the approach.

In this paper, we investigate the effect of haptic cueing on a human operator's performance in the field of bilateral teleoperation of multiple mobile robots, particularly multiple unmanned aerial vehicles (UAVs). Two aspects of human performance are deemed important in this area, namely, the maneuverability of mobile robots and the perceptual sensitivity of the remote environment. We introduce metrics that allow us to address these aspects in two psychophysical studies, which are reported here. Three fundamental haptic cue types were evaluated. The Force cue conveys information on the proximity of the commanded trajectory to obstacles in the remote environment. The Velocity cue represents the mismatch between the commanded and actual velocities of the UAVs and can implicitly provide a rich amount of information regarding the actual behavior of the UAVs. Finally, the Velocity+Force cue is a linear combination of the two. Our experimental results show that, while maneuverability is best supported by the Force cue feedback, perceptual sensitivity is best served by the Velocity cue feedback. In addition, we show that large gains in the haptic feedbacks do not always guarantee an enhancement in the teleoperator's performance.

Robot vision became a field of increasing importance in micro aerial vehicle robotics with the availability of small and light hardware. While most approaches rely on external ground stations because of the need of high computational power, we will present a full autonomous setup using only on-board hardware. Our work is based on the continuous homography constraint to recover ego-motion from optical flow. Thus we are able to provide an efficient fall back routine for any kind of UAV (Unmanned Aerial Vehicles) since we rely solely on a monocular camera and on on-board computation. In particular, we devised two variants of the classical continuous 4-point algorithm and provided an extensive experimental evaluation against a known ground truth. The results show that our approach is able to recover the ego-motion of a flying UAV in realistic conditions and by only relying on the limited on-board computational power. Furthermore, we exploited the velocity estimation for closing the loop and controlling the motion of the UAV online.

Standard quadrotor UAVs possess a limited mobility because of their inherent underactuation, i.e., availability of 4 independent control inputs (the 4 propeller spinning velocities) vs. the 6 dofs parameterizing the quadrotor position/ orientation in space. As a consequence, the quadrotor pose cannot track an arbitrary trajectory over time (e.g., it can hover on the spot only when horizontal). In this paper, we propose a novel actuation concept in which the quadrotor propellers are allowed to tilt about their axes w.r.t. the main quadrotor body. This introduces an additional set of 4 control inputs which provides full actuation to the quadrotor position/orientation. After deriving the dynamical model of the proposed quadrotor, we formally discuss its controllability properties and propose a nonlinear trajectory tracking controller based on dynamic feedback linearization techniques. The soundness of our approach is validated by means of simulation results.

In this paper, we present a passivity-based decentralized approach for bilaterally teleoperating a group of UAVs composing the slave side of the teleoperation system. In particular, we explicitly consider the presence of time delays, both among the master and slave, and within UAVs composing the group. Our focus is on analyzing suitable (passive) strategies that allow a stable teloperation of the group despite presence of delays, while still ensuring high flexibility to the group topology (e.g., possibility to autonomously split or join during the motion). The performance and soundness of the approach is validated by means of human/hardware-in-the-loop simulations (HHIL).

Robotic vision has become an important field of research for micro aerial vehicles in the recent years. While many approaches for autonomous visual control of such vehicles rely on powerful ground stations, the increasing availability of small and light hardware allows for the design of more independent systems. In this context, we present a robust algorithm able to recover the UAV ego-motion using a monocular camera and on-board hardware. Our method exploits the continuous homography constraint so as to discriminate among the observed feature points in order to classify those belonging to the dominant plane in the scene. Extensive experiments on a real quadrotor UAV demonstrate that the estimation of the scaled linear velocity in a cluttered environment improved by a factor of 25% compared to previous approaches.

This work extends the framework of bilateral shared control of mobile robots with the aim of increasing the robot autonomy and decreasing the operator commitment. We consider persistent autonomous behaviors where a cyclic motion must be executed by the robot. The human operator is in charge of modifying online some geometric properties of the desired path. This is then autonomously processed by the robot in order to produce an actual path guaranteeing: i) tracking feasibility, ii) collision avoidance with obstacles, iii) closeness to the desired path set by the human operator, and iv) proximity to some points of interest. A force feedback is implemented to inform the human operator of the global deformation of the path rather than using the classical mismatch between desired and executed motion commands. Physically-based simulations, with human/hardware-in-the-loop and a quadrotor UAV as robotic platform, demonstrate the feasibility of the method.

For a quadrotor aircraft, we study the problem of planning a trajectory that connects two arbitrary states while allowing the UAV to grasp a moving target at some intermediate time. To this end, two classes of canonical grasping maneuvers are defined and characterized. A planning strategy relying on differential flatness is then proposed to concatenate one or more grasping maneuvers by means of spline-based subtrajectories, with the additional objective of minimizing the total transfer time. The proposed planning algorithm is not restricted to pure hovering-to-hovering motions and takes into account practical constraints, such as the finite duration of the grasping phase. The effectiveness of the proposed approach is shown by means of physically-based simulations.

In this paper, a novel decentralized control strategy for bilaterally teleoperating heterogeneous groups of mobile robots from different domains (aerial, ground, marine, and underwater) is proposed. By using a decentralized control architecture, the group of robots, which is treated as the slave side, is made able to navigate in a cluttered environment while avoiding obstacles, interrobot collisions, and following the human motion commands. Simultaneously, the human operator acting on the master side is provided with a suitable force feedback informative of the group response and of the interaction with the surrounding environment. Using passivity-based techniques, we allow the behavior of the group to be as flexible as possible with arbitrary split and join events (e.g., due to interrobot visibility/packet losses or specific task requirements) while guaranteeing the stability of the system. We provide a rigorous analysis of the system stability and steady-state characteristics and validate performance through human/hardware-in-the-loop simulations by considering a heterogeneous fleet of unmanned aerial vehicles (UAVs) and unmanned ground vehicles as a case study. Finally, we also provide an experimental validation with four quadrotor UAVs.

The use of multiple unmanned aerial vehicles (UAVs) is increasingly being incorporated into a wide range of teleoperation applications. To date, relevant research has largely been focused on the development of appropriate control schemes. In this paper, we extend previous research by investigating how control performance could be improved by providing the teleoperator with haptic feedback cues. First, we describe a control scheme that allows a teleoperator to manipulate the flight of multiple UAVs in a remote environment. Next, we present three designs of haptic cue feedback that could increase the teleoperator's environmental awareness of such a remote environment. These cues are based on the UAVs' i) velocity information, ii) proximity to obstacles, and iii) a combination of these two sources of information. Finally, we present an experimental evaluation of these haptic cue designs. Our evaluation is based on the teleoperator's perceptual sensitivity to the physical environment inhabited by the multiple UAVs. We conclude that a teleoperator's perceptual sensitivity is best served by haptic feedback cues that are based on the velocity information of multiple UAVs.

In this paper, a novel distributed control strategy for teleoperating a fleet of Unmanned Aerial Vehicles (UAVs) is proposed. Using passivity based techniques, we allow the behavior of the UAVs to be as flexible as possible with arbitrary split and join decisions while guaranteeing stability of the system. Furthermore, the overall teleoperation system is also made passive and, therefore, characterized by a stable behavior both in free motion and when interacting with unknown passive obstacles. The performance of the system is validated through semi-experiments.

We propose a novel haptic teleoperation control framework for multiple unmanned aerial vehicles (UAVs) over the Internet, consisting of the three control layers: 1) UAV control layer, where each UAV is abstracted by, and is controlled to follow the trajectory of, its own kinematic virtual point (VP); 2) VP control layer, which modulates each VP's motion according to the teleoperation commands and local artificial potentials (for inter-VP/VP-obstacle collision avoidance and inter-VP connectivity preservation); and 3) teleoperation layer, through which a remote human user can command all (or some) of the VPs' velocity while haptically perceiving the state of all (or some) of the UAVs over the Internet. Master passivity/slave-stability and some asymptotic performance measures are proved. Semi-experiment results are presented to validate the theory.

This paper describes the mechanical and control design of the new 7-DOF CyberMotion Simulator, a redundant industrial manipulator arm consisting of a standard 6 DOF anthropomorphic manipulator plus an actuated cabin attached to the end-effector. Contrarily to Stewart platforms, an industrial manipulator offers several advantages when used as motion simulator: larger motion envelope, higher dexterity, and possibility to realize any end-effector posture within the workspace. In addition to this, the new actuated cabin acts as an additional joint and provides the needed kinematic redundancy to cope with the robot actuator and joint range constraints, which in general can significantly deteriorate the desired motion cues the robot is reproducing. In particular, we will show that, by suitably exploiting the redundancy better results can be obtained in reproducing sustained acceleration cues, a relevant problem when implementing vehicle simulators.

We present a decentralized system for the bilateral teleoperation of groups of UAVs which only relies on relative bearing measurements, i.e., without the need of distance information or global localization. The properties of a 3D bearing-formation are analyzed, and a minimal set of bearings needed for its definition is provided. We also design a novel decentralized formation control almost globally convergent and able to maintain bounded and non-vanishing inter-distances among the agents despite the absence of direct distance measurements. Furthermore, we develop a multi-master/ multi-slave teleoperation setup in order to control the overall behavior of the group and to convey to the human operator suitable force cues, while ensuring stability in presence of delays and packet losses over the master-slave communication channel. The theoretical framework is validated by means of extensive human/hardware in-the-loop simulations using two force-feedback devices and a group of quadrotors.

In this paper, we present an experimental validation of a novel decentralized passivity-based control strategy for teleoperating a group of Unmanned Aerial Vehicles (UAVs): the slave side, consisting of the UAVs, is endowed with large group autonomy by allowing time-varying topology and interrobot/obstacle collision avoidance. The master side, represented by a human operator, controls the group motion and receives suitable force feedback cues informing her/him about the remote slave motion status. Passivity theory is exploited for guaranteeing stability of the slave side and of the overall teleoperation channel. Results of experiments involving the use of 4 quadcopters are reported and discussed, confirming the soundness of the paper theoretical claims.

In this paper, we investigate the maneuverability performance of human teleoperators on multi-robots. First, we propose that maneuverability performance can be assessed by a frequency response function that jointly considers the input force of the operator and the position errors of the multi-robot system that is being maneuvered. Doing so allows us to evaluate maneuverability performance in terms of the human teleoperator's interaction with the controlled system. This allowed us to effectively determine the suitability of different haptic cue algorithms in improving teleoperation maneuverability. Performance metrics based on the human teleoperator's frequency response function indicate that maneuverability performance is best supported by a haptic feedback algorithm which is based on an obstacle avoidance force.

Despite many recent developments in virtual reality, an effective locomotion interface which allows for normal walking through large virtual environments was until recently still lacking. Here, we describe the new CyberWalk omnidirectional treadmill system, which makes it possible for users to walk endlessly in any direction, while never leaving the confines of the limited walking surface. The treadmill system improves on previous designs, both in its mechanical features and in the control system employed to keep users close to the center of the treadmill. As a result, users are able to start walking, vary their walking speed and direction, and stop walking as they would on a normal, stationary surface. The treadmill system was validated in two experiments, in which both the walking behavior and the performance in a basic spatial updating task were compared to that during normal overground walking. The results suggest that walking on the CyberWalk treadmill is very close to normal walking, especially after some initial familiarization. Moreover, we did not find a detrimental effect of treadmill walking in the spatial updating task. The CyberWalk system constitutes a significant step forward to bringing the real world into the laboratory or workplace.

This paper considers the problem of realizing a 6-DOF closed-loop motion simulator by exploiting an anthropomorphic serial manipulator as motion platform. Contrary to standard Stewart platforms, an industrial anthropomorphic manipulator offers a considerably larger motion envelope and higher dexterity that let envisage it as a viable and superior alternative. Our work is divided in two papers. In this Part I, we discuss the main challenges in adopting a serial manipulator as motion platform, and thoroughly analyze one key issue: the design of a suitable inverse kinematics scheme for online motion reproduction. Experimental results are proposed to analyze the effectiveness of our approach. Part II [1] will address the design of a motion cueing algorithm tailored to the robot kinematics, and will provide an experimental evaluation on the chosen scenario: closed-loop simulation of a Formula 1 racing car.

This paper, divided in two Parts, considers the problem of realizing a 6-DOF closed-loop motion simulator by exploiting an anthropomorphic serial manipulator as motion platform. After having proposed a suitable inverse kinematics scheme in Part I [1], we address here the other key issue, i.e., devising a motion cueing algorithm tailored to the specific robot motion envelope. An extension of the well-known classical washout filter designed in cylindrical coordinates will provide an effective solution to this problem. The paper will then present a thorough experimental evaluation of the overall architecture (inverse kinematics + motion cueing) on the chosen scenario: closed-loop simulation of a Formula 1 racing car. This will prove the feasibility of our approach in fully exploiting the robot motion capabilities as a motion simulator.

We consider the kinematic control problem for nonholonomic mobile manipulators (NMMs) whose base contains steering wheels. For all typical tasks, the steering velocity inputs of such systems do not appear in the differential relationship between the first-order time derivative of the task output and the available NMM inputs. As a consequence, these inputs are not used by velocity-level control laws based on simple (pseudo)inversion of the task Jacobian, leading in general to the impossibility of completing the task. We propose two control solutions to this open problem based on the framework of input-output feedback linearization. First, a static feedback law is presented that defines the unspecified steering velocities via an optimization action in the null space of the task Jacobian. A dynamic feedback law is then proposed based on the input-output differential map obtained by considering the task acceleration. In this case, the velocity of the steering wheels becomes an active input for task execution, together with the manipulator joint accelerations and the driving accelerations of the base. The feasibility and performance of the two kinematic controllers are compared in simulation for a car-like base carrying a planar manipulator.

Creation of a virtual interactive and highly evolved environment with Surprises characters.

For us humans, walking is our most natural way of moving through the world. One of the major challenges in present research on navigation in virtual reality is to enable users to physically walk through virtual environments. Although treadmills, in principle, allow users to walk for extended periods of time through large virtual environments, existing setups largely fail to produce a truly immersive sense of navigation. Partially, this is because of inadequate control of treadmill speed as a function of walking behavior. Here, we present a new control algorithm that allows users to walk naturally on a treadmill, including starting to walk from standstill, stopping, and varying walking speed. The treadmill speed control consists of a feedback loop based on the measured user position relative to a given reference position, plus a feed-forward term based on online estimation of the user's walking velocity. The purpose of this design is to make the treadmill compensate fully for any persistent walker motion, while keeping the accelerations exerted on the user as low as possible. We evaluated the performance of the algorithm by conducting a behavioral experiment in which we varied its most important parameters. Participants walked at normal walking speed and then, on an auditory cue, abruptly stopped. After being brought back to the center of the treadmill by the control algorithm, they rated how smoothly the treadmill had changed its velocity in response to the change in walking speed. Ratings, in general, were quite high, indicating good control performance. Moreover, ratings clearly depended on the control algorithm parameters that were varied. Ratings were especially affected by the way the treadmill reversed its direction of motion. In conclusion, controlling treadmill speed in such a way that changes in treadmill speed are unobtrusive and do not disturb VR immersiveness is feasible on a normal treadmill with a straightforward control algorithm.

Research on humanoid robots for use in servicing tasks, e.g. fetching and delivery, attracts steadily more interest. With “Rollin' Justin” a mobile robotic system and research platform is presented that allows sophisticated control algorithms and dexterous manipulation. This video gives an overview of the mobile humanoid robotic system “Rollin' Justin” with special emphasis on mechanical design features, control issues and high-level system capabilities such as human robot interaction.

Research on humanoid robots for use in servicing tasks, e.g. fetching and delivery, attracts steadily more interest. With Rollin' Justin a mobile robotic system and research platform is presented that allows the implementation and demonstration of sophisticated control algorithms and dexterous manipulation. Important problems of service robotics such as mobile manipulation and strategies for using the increased workspace and redundancy in manipulation task can be studied in detail. This paper gives an overview of the design considerations for a mobile platform and their realizations to transform the formerly table-mounted humanoid upper body system Justin into Rollin' Justin, a fully self-sustaining mobile research platform.

Mobile platforms equipped with several steering wheels are known to be omnidirectional, i.e., able to independently translate and rotate on the plane. As an improvement to this design, the Justin mobile platform also possesses the ability to vary its footprint over time by extending/retracting the wheel legs during motion. In this paper, we discuss the kinematic modeling and control issues for such a platform. The goal is to obtain a tracking controller which is able to realize an arbitrary linear/angular platform motion while, at the same time, independently expanding/retracting each leg. Experimental results support the proposed approach.

The CyberWalk is a large size 2D omni-directional platform that allows unconstrained locomotion possibilities to a walking user for VR exploration. In this paper we present the motion control design for the platform, which has been developed within the homonymous European research project. The objective is to compensate the intentional motion of the user, so as to keep her/him always close to the platform center while limiting the perceptual effects due to actuation commands. The controller acts at the acceleration level, using suitable observers to estimate the unmeasurable intentional walker's velocity and acceleration. A moving reference position is used to limit the accelerations felt by the user in critical transients, e.g., when the walker suddenly stops motion. Experimental results are reported that show the benefit of designing separate control gains in the two orthogonal directions (lateral and sagittal) of a frame attached to the walker.

In this paper, we characterize the time-optimal trajectories leading a Dubins car in collision with the obstacles in its workspace. Due to the constant velocity constraint characterizing the Dubins car model, these trajectories form a sufficient set of shortest paths between any robot configuration and the obstacles in the environment. Based on these paths, we define and give the algorithm for computing a distance function that takes into account the nonholonomic constraints and captures the nonsymmetric nature of the system. The developments presented here assume that the obstacles and the robot are polygons although the methodology can be applied to different shapes.

In the image-based visual servoing framework, image moments provide an appealing choice as visual features since they can be easily evaluated on any shape on the image plane, and do not require tracking and matching of individual geometric structures between distinct image frames (i.e., the so-called correspondence problem). However, computation of the moment interaction matrix still requires the knowledge of specific unmeasurable 3D quantities relative to the target object, quantities that are usually approximated in practical implementations. Therefore, in this paper we analyze the possibility to estimate on-line the value of such 3D quantities during the camera motion with the only assumption of a target shape with planar limb surface. The proposed estimation scheme builds upon the theory of nonlinear observers, and in particular exploits the basic formulation of the persistency of excitation Lemma. Simulation results are then presented in order to support the effectiveness of the proposed approach.

Within the standard IBVS framework for control of generic robotic systems, a suitable exploitation of redundancy w.r.t. the given visual task can significantly improve the overall task execution. Indeed, redundancy can be used to avoid occlusions, joint limits, or to realize tasks that would be ill-conditioned if addressed altogether. In this respect, we propose an experimental evaluation of the performance of two redundancy resolution schemes, namely Task Priority and Task Sequencing, when adopted to realize IBVS tasks on a mobile robot equipped with a pan-tilt camera onboard.

In this paper we present an experimental evaluation of automatic robotic assembly of complex planar parts. The torque-controlled DLR light-weight robot, equipped with an on-board camera (eye-in-hand configuration), is committed with the task of looking for given parts on a table, picking them, and inserting them inside the corresponding holes on a movable plate. Visual servoing techniques are used for fine positioning over the selected part/hole, while insertion is based on active compliance control of the robot and robust assembly planning in order to align the parts automatically with the hole. Execution of the complete task is validated through extensive experiments, and performance of humans and robot are compared in terms of overall execution time.

The CyberCarpet is an actuated platform that allows unconstrained locomotion of a walking user for VR exploration. The platform has two actuating devices (linear and angular) and the motion control problem is dual to that of nonholonomic wheeled mobile robots. The main control objective is to keep the walker close to the platform center. We first recall global kinematic control schemes developed at the velocity level, i.e., with the linear and angular velocities of the platform as input commands. Then, we use backstepping techniques and the theory of cascaded systems to move the design to control laws at the acceleration level. Acceleration control is more suitable to take into account the limitations imposed to the platform motion by the actuation system and/or the physiological bounds on the human walker. In particular, the availability of platform accelerations allows the analytical computation of the apparent accelerations felt by the user.

In the image-based visual servoing framework, error signals are directly computed from image feature parameters, thus obtaining control schemes which do not need neither a 3-D model of the scene, nor a perfect knowledge of the camera calibration matrix. However, the current value of the depth Z for each considered feature must be known. We propose a method to estimate on-line the value of Z for point features while the camera is moving through the scene, by using tools from nonlinear observer theory. By interpreting Z as a continuous unknown state with known dynamics, we build an estimator which asymptotically recovers the actual depth value for the selected feature.

Exploration of virtual worlds with unconstrained locomotion possibilities for the user is the main objective of the European research project CyberWalk. This should be achieved through the use of an actuated platform (the CyberCarpet) that compensates for the walker's locomotion in such a way to keep her/him close to the platform center. This paper presents the control problem for the platform motion, including objectives and constraints, overall control architecture, and kinematic modeling. Since the platform has only two actuating devices (linear and angular), the control problem is similar to that of output regulation for nonholonomic wheeled mobile robots in the presence of an unpredictable disturbance due to walker's locomotion. Based on the kinematic model, a velocity control design achieving input-output decoupling and linearization is proposed and its performance is verified by simulations

We consider robotic systems made of a nonholonomic mobile platform carrying a manipulator (nonholonomic mobile manipulator, NMM). By combining the manipulator differential kinematics with the admissible differential motion of the platform, a simple and general kinematic model for NMMs is derived. Assuming that the robotic system is kinematically redundant for a given task, we present the extension of redundancy resolution schemes originally developed for standard manipulators, in particular the projected gradient (PG) and the reduced gradient (RG) optimization-based methods. The case of a configuration-dependent task specification is also discussed. The proposed modeling approach is illustrated with reference to representative NMMs, and the performance of the PG and RG methods for redundancy resolution is compared on a series of numerical case studies

This paper shows how to compute the nonholonomic distance between a polygonal car-like robot and polygonal obstacles. The solution extends previous work of Reeds and Shepp by finding the shortest path to a manifold (rather than to a point) in configuration space. Based on optimal control theory, the proposed approach yields an analytic solution to the problem

An experimental study is presented on vision-based feedback control methods for the nonholonomic wheeled mobile robot SuperMARIO. The robot posture is measured via a camera fixed on the ceiling of an indoor environment. To this end, a simple localization algorithm has been developed. Performance on trajectory following and parking tasks is compared under different controllers and using either odometric or visual feedback. The improvement with the latter is obtained at the expense of a limited increase in sampling time.