Human-multi-robot teaming in precision agriculture presents a promising approach to addressing labor shortages and managing the complexities of agricultural practices. An effective coordination of these teams, including task allocation and scheduling strategies while accounting for the inherent unpredictability of human behavior, is crucial for maximizing system productivity and ensuring user comfort. In this study, we introduce a Mixed-Integer Linear Programming (MILP) approach that aims to minimize workers’ waiting times, robots’ energy consumption during the different phases of the robots’ motions, and the overall makespan. To enhance the robustness of our framework and consider human preferences, a user interface is designed to capture real-time human feedback; then, an adaptive online updating strategy that dynamically adjusts plans responding to variations in human operators’ parameters is devised. To handle large-scale problems, we extend the solution approach by leveraging Constraint Programming (CP) combined with a batch decomposition strategy. The approach is validated through extensive simulations in a Unity-based realistic virtual reality environment and laboratory experiments using two TurtleBot2 robots and two human operators performing grape harvesting tasks.

This paper introduces a control architecture that enables a robotic system to ensure the safety of human operators entering its workspace. The proposed method utilizes an appropriate metric to measure safety levels and adjusts the robot’s motion to maintain this metric above a minimum threshold. To guarantee safety, the robot scales down and deviates from its intended path. For redundant robots, internal motion is exploited to enhance safety levels further. The approach is incorporated into a Hierarchical Quadratic Programming control framework, allowing the robot to address other control objectives simultaneously, such as handling joint limits. Experimental results with a dual-arm mobile robot developed as part of the EU-funded CANOPIES project demonstrate the effectiveness of the proposed method.Note to Practitioners—This paper was motivated by the problem of ensuring human safety in unstructured environments shared with human operators. We propose a control architecture that allows complex dual-arm robotic systems to operate effectively in such scenarios. The devised architecture gives the robot the capability to slow down a trajectory to follow as well as to deviate from a nominal path to keep a human operator safe. We tested the devised approach in a precision farming setting; however, it can be adopted in any human-robot interaction scenario.

Weeds management is a repetitive and crucial task for agricultural settings. This paper considers a four-wheeled robot, called Vitirover, designed for grass-cutting and weed management tasks in vineyards. The robot steering mechanism employs differential rotation of rear wheels, mounted on a universal joint. First, the kinematic model of the robot is derived. Next, based on the kinematic model, a Model Predictive Control (MPC) formulation is designed to encourage the robot to follow a desired path while targeting weeds in the environment using dynamic weights. Simulation results in Gazebo simulator are provided to validate the overall system.

This paper introduces a shared control framework designed specifically for agricultural mobile manipulators engaged in harvesting operations. The shared control strategy allows for achieving such operations by dynamically exchanging the control between the robotic system and a human operator depending on the uncertainty in the environment perception. For this purpose, the robot’s behavior is dynamically adapted to switch between two control modes with a different level of autonomy of the robot. The level of autonomy is encoded in two different admittance behaviors which are included in a first-order Hierarchical Quadratic Programming (HQP) control framework, that allows the robot to simultaneously address other control objectives at the same time. Experimental results with a dual-arm mobile robot, developed as part of the EU-funded CANOPIES project, demonstrate the effectiveness of the proposed method in real conditions.

Visual planning methods are promising to handle complex settings where extracting the system state is challenging. However, none of the existing works tackles the case of multiple heterogeneous agents which are characterized by different capabilities and/or embodiment. In this work, we propose a method to realize visual action planning in multi-agent settings by exploiting a roadmap built in a low-dimensional structured latent space and used for planning. To enable multi-agent settings, we infer possible parallel actions from a dataset composed of tuples associated with individual actions. Next, we evaluate feasibility and cost of them based on the capabilities of the multi-agent system and endow the roadmap with this information, building a capability latent space roadmap (C-LSR). Additionally, a capability suggestion strategy is designed to inform the human operator about possible missing capabilities when no paths are found. The approach is validated in a simulated burger cooking task and a real-world box packing task.

Redundancy in robot structures allows the implementation of control algorithms in which it is possible to add secondary control objectives. Those are typically functions to be minimized/maximized and projected onto the null-space of the primary control objectives. As an example, typical metrics to maximize are the robot manipulability or the distance from its mechanical joint limits. Usually, designer's heuristics is used to decide which function eventually to optimize. This paper shows that heuristics may lead to counter-intuitive results such as, for example, reducing the dexterous workspace with respect to, e.g., avoiding optimization at all. A learning algorithm is proposed to allow the robot to dynamically select the function to optimize in a way to increase the overall dexterous workspace with respect to the static, heuristic choice. As a result, the robot will be able to increase its dexterous workspace by selecting the proper lower-priority task via the use of a neural network trained during a proper supervised learning process. A 3-link planar manipulator is used as numerical case study.

The requirements of modern production systems together with more advanced robotic technologies have fostered the integration of teams comprising humans and autonomous robots. While this integration has the potential to provide various benefits, it also raises questions about how to effectively manage these teams, taking into account the different characteristics of the agents involved. This paper presents a framework for task allocation in a human multi-robot collaborative scenario. The proposed solution combines an optimal offline allocation with an online reallocation strategy which accounts for inaccuracies of the offline plan and/or unforeseen events, human subjective preferences and cost of task switching. Experiments with two manipulators cooperating with a human operator in a box filling task are presented.

The use of heterogeneous human-multi-robot teams enables the combination of complementary skills of these two different types of agents. To have an effective collaboration, it is necessary to define a strategy for allocating and scheduling tasks among them. In this work, we distinguish robots in working robots and service ones: working robots and human operators can perform similar tasks in the environment and both are assisted by service robots. We propose a Mixed-Integer Linear Programming approach that aims to minimize the waiting times of the working agents, the energy consumption of the service robots, and the makespan while ensuring that the velocity constraints of the robots are met and the task ordering is correct. Furthermore, we propose an online updating strategy that tackles changes in the parameters of working agents and adapts the plan accordingly based on a heuristic algorithm. To validate our framework, we analyze a precision agriculture harvesting application with two human operators, two working robots, and two service robots.

In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.

Visual action planning particularly excels in applications where the state of the system cannot be computed explicitly, such as manipulation of deformable objects, as it enables planning directly from raw images. Even though the field has been significantly accelerated by deep learning techniques, a crucial requirement for their success is the availability of a large amount of data. In this work, we propose the Augment-Connect-Explore (ACE) paradigm to enable visual action planning in cases of data scarcity. We build upon the Latent Space Roadmap (LSR) framework which performs planning with a graph built in a low dimensional latent space. In particular, ACE is used to i) Augment the available training dataset by autonomously creating new pairs of datapoints, ii) create new unobserved Connections among representations of states in the latent graph, and iii) Explore new regions of the latent space in a targeted manner. We validate the proposed approach on both simulated box stacking and real-world folding task showing the applicability for rigid and deformable object manipulation tasks, respectively.

This paper considers a scenario where a robot and a human operator share the same workspace, and the robot is able to both carry out autonomous tasks and physically interact with the human in order to achieve common goals. In this context, both intentional and accidental contacts between human and robot might occur due to the complexity of tasks and environment, to the uncertainty of human behavior, and to the typical lack of awareness of each other actions. Here, a two stage strategy based on Recurrent Neural Networks (RNNs) is designed to detect intentional and accidental contacts: the occurrence of a contact with the human is detected at the first stage, while the classification between intentional and accidental is performed at the second stage. An admittance control strategy or an evasive action is then performed by the robot, respectively. The approach also works in the case the robot simultaneously interacts with the human and the environment, where the interaction wrench of the latter is modeled via Gaussian Mixture Models (GMMs). Control Barrier Functions (CBFs) are included, at the control level, to guarantee the satisfaction of robot and task constraints while performing the proper interaction strategy. The approach has been validated on a real setup composed of a Kinova Jaco2 robot.

In this work, we address a task allocation problem for human multi-robot settings. Given a set of tasks to perform, we formulate a general Mixed-Integer Linear Programming (MILP) problem aiming at minimizing the overall execution time while optimizing the quality of the executed tasks as well as human and robotic workload. Different skills of the agents, both human and robotic, are taken into account and human operators are enabled to either directly execute tasks or play supervisory roles; moreover, multiple manipulators can tightly collaborate if required to carry out a task. Finally, as realistic in human contexts, human parameters are assumed to vary over time, e.g., due to increasing human level of fatigue. Therefore, online monitoring is required and re-allocation is performed if needed. Simulations in a realistic scenario with two manipulators and a human operator performing an assembly task validate the effectiveness of the approach.

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the effectiveness of the method on a simulated box stacking task as well as a T-shirt folding task performed with a real robot.

In this paper, a scenario of physical human-robot collaboration is considered, in which a robot is able to both carry out autonomous tasks and to physically interact with a human operator to achieve a common objective. However, since human and robot share the same workspace both accidental and intentional contacts between them might arise. Therefore, a solution based on Recurrent Neural Networks (RNNs) is proposed to detect and classify the nature of the contact with the human, even in the case the robot is interacting with the environment because of its own task. Then, reaction strategies are defined depending on the nature of contact: human avoidance with evasive action in the case of accidental interaction, and admittance control in the case of intentional interaction. In regard to the latter, Control Barrier Functions (CBFs) are considered to guarantee the satisfaction of robot constraints, while endowing the robot with a compatible compliant behavior. The approach is validated on real data acquired from the interaction with a Kinova Jaco2.

The paper presents a Distributed Fault Detection and Isolation strategy for a team of mobile manipulators performing a cooperative mission. The overall system relies on an observer-controller scheme where each robot estimates the global state of the team through a distributed observer; then, the global state estimate is used by each robot to compute the estimated local input so as to achieve a specific global task. The observer-controller scheme also allows to define a set of residual vectors that can be used by the robots to detect and isolate faults affecting any member of the team, even if not in direct communication, and without increasing the computational burden and the information exchange. The approach is validated via numerical simulations with a team of four mobile manipulators performing a transportation mission.

In this paper, a distributed scheme to allow a human operator to physically interact with a multi-manipulator system is devised. Manipulators are tightly connected to a rigid object and a human operator interacts with it to perform, for example, a cooperative transportation task. The strategy foresees two layers. The top layer is in charge of assigning a compliant behaviour to the object through an admittance model whose reference trajectory is dynamically adjusted to regulate the human-object interaction force. Moreover, since the parameters of the dynamic model of the human arm end-point are supposed to be time-varying and completely unknowns with unknown bounds, a robust adaptive control is envisaged in this layer. The output of this layer is a desired object trajectory which is tracked by the bottom layer. In detail, the latter resorts to a robust adaptive control strategy to both track the object trajectory and control the internal stresses exerted by the manipulators on the object which unavoidably arise due to dynamic and kinematic uncertainties and synchronization errors. Simulations involving a setup with three dual-arm Movo mobile robots corroborate the theoretical findings.

In this paper the modeling and planning problems of a system composed of multiple ground and aerial robots involved in a transportation task are considered. The ground robots rigidly grasp a load, while the aerial vehicles are attached to the object through non-rigid inextensible cables. The idea behind such a heterogeneous multi-robot system is to benefit of the advantages of both types of robots that might be the precision of ground robots, the increased payload of multiple aerial vehicles and their larger workspace. The overall model of the system is derived and its expression and redundancy are exploited by setting a general constrained optimal planning problem. The problem is herein solved by dynamic programming and simulation results validated the proposed scheme.

The paper presents a two stages distributed algorithm for cooperative manipulating an unknown object rigidly grasped by mobile manipulators, in the absence of both a central unit and any explicit information exchange among robots. In the first stage, robots cooperatively estimate the object kinematic and dynamic parameters by properly moving the object or applying specific contact wrenches. In the second stage, the estimated parameters are used in a distributed cooperative algorithm aimed at controlling the object pose while limiting both the squeezing wrenches exerted by the manipulators and the wrench exerted by the environment on the object. Numerical simulations demonstrate the feasibility of the approach.

In this paper, a general framework to control in a distributed way a system composed by multiple robots is proposed. Each robot is characterized by a discrete-time linear dynamics, and the whole system is controlled via a linear static feedback law with a feed-forward term. Usually, this form of the global control input requires a central unit or an all-to-all communication for computing the local control input of each robot. To counteract the lack of a central unit, each robot estimates, via a local observer, the overall state of the team, and such an estimate is used to compute its local control input as in the case a central unit was present. Two simulations case studies are provided in the framework of multi-robot optimal control and formation control.

This paper presents a solution to the problem of decentralized control, fault detection and isolation for teams of cooperative autonomous mobile vehicles. The strategy is carried out in the discrete time domain. A local observer is used by each agent to estimate the overall state of the team. This estimate is, then, used both to compute its local control input and isolate faulty teammates, even in absence of direct communication with them. For diagnosis purposes, a set of residual vectors, each of them sensible to a fault occurring on a single vehicle, is designed and an adaptive threshold is derived in order to avoid false alarms. The approach is validated via numerical simulations involving 4 vehicles moving in formation in a 3D environment.

The paper deals with the problem of decentralized fault detection, isolation and recovery for teams of networked robots. The proposed strategy is a combination of distributed and local approaches that allow the robots to deal with both recoverable and unrecoverable faults. A local adaptive fault observer is used to locally compensate recoverable faults, while a distributed fault detection and isolation strategy is used to allow each robot to detect unrecoverable faults on other teammates even if not directly connected; once the faulty robots have been isolated, they are removed from the team and the mission is rearranged. Results of numerical simulations and experiments involving a team of 5 mobile robots are provided to show the effectiveness of the approach.

In this paper, a decentralized control strategy for networked multi-robot systems that allows the tracking of the team centroid and the relative formation is presented. The proposed solution consists of a distributed observer-controller scheme where, based only on local information, each robot estimates the collective state and tracks the two assigned control variables. We provide a formal stability analysis of the observer-controller scheme and we relate convergence properties to the topology of the connectivity graph. Experiments are presented to validate the approach.

Coverage of a given area by means of coordinated autonomous robots is a mission required in several applications such as, for example, patrolling, monitoring or environmental sampling. From a mathematical perspective, this can often be modeled as the need to estimate a scalar field, eventually time varying as in the security applications. In this paper, the problem is addressed for the challenging underwater scenario, where localization and communication pose additional constraints. The solution exploits the appealing properties of the Voronoi partition of a convex set within a probabilistic framework. In addition, the algorithm is totally distributed and characterized by a strong engineering perspective allowing the handling of asynchronous communication or possible loss or adjunct of vehicles. Beyond the test in dozen of numerical case studies, the algorithm has been validated by a challenging underwater test in 3 dimension involving two Autonomous Underwater Vehicles (AUVs). The experiments were run in the La Spezia harbor, in Italy, in February 2012 as demo of the European project Co3AUVs.

There are various activities that support software maintenance. Program comprehension and detection of design anomalies and their symptoms, like code smells and anti patterns, are particularly relevant for improving the quality and facilitating evolution of a system. In this paper we describe an empirical study on the detection of code smells, aiming at identifying the most frequent smells in systems of different domains and hence the domains characterized by more smells. Moreover, we study possible correlations existing among smells and the values of a set of software quality metrics using Spearman's rank correlation and Principal Component Analysis.

Several code smells detection tools have been developed providing different results, because smells can be subjectively interpreted and hence detected in different ways. Usually the detection techniques are based on the computation of different kinds of metrics, and other aspects related to the domain of the system under analysis, its size and other design features are not taken into account. In this paper we propose an approach we are studying based on machine learning techniques. We outline some common problems faced for smells detection and we describe the different steps of our approach and the algorithms we use for the classification.

A coordination mechanism to achieve the sampling task of static or dynamic fields by means of a system composed by multiple mobile robots is addressed in this paper. The problem is the estimation of a scalar field. To this aim in a probabilistic framework a solution is proposed that takes into account several constraints. The attention is focused on the vehicles motion generation and the developed strategy is designed for multiple, autonomous and distributed robots. It makes use of the Voronoi tessellation's properties to automatically distribute the vehicles' motion and of the Null-Space-Behavioral control to handle eventually conflicting motion tasks (as reaching a given point while avoiding obstacles). The algorithm can be tailored based on the communication and computational capabilities of the robots. A discussion and possible counterexamples of the applications of existing approaches are provided in the paper. Numerical simulations illustrate the results.

This paper describes a decentralized coordination strategy for multi robot patrolling missions. To this effect, the theory of Gaussian Processes (usually used for estimation purposes) is suitably adapted to tackle the problem of harbour patrolling. The introduction of a time varying dependency in the probabilistic formulation (thus allowing for the sampled field to be dynamic, i.e., changing in time) makes the proposed solution suitable for the type of mission considered. Moreover, the advantages of Voronoi tessellations are exploited to automatically distribute the vehicles over the environment. The resulting algorithm takes into account several constraints and can be tailored based on the communication and computational capabilities of the robots, thus making it suitable for heterogeneous systems. Numerical simulations and experiments involving three autonomous marine surface vehicles in a harbour scenario at the Parque Expo site in Lisbon are discussed.

In this paper a decentralized controller-observer scheme for centroid tracking with a multi-robot system is presented. The key idea is to develop, for each robot, an observer of the collective system's state; each local observer is updated by only using information of the state of the robot and of its neighbors. The local observers' estimations are then used by the individual robots to cooperatively track an assigned time-varying reference for the weighted centroid. Convergence of the scheme is proven for both fixed and switching communication topologies, as well as for directed and undirected communication graphs. Numerical simulations relative to different case studies are illustrated to validate the approach.

Differential-drive mobile robots are usually equipped with video-cameras for navigation purposes. In order to ensure proper operational capabilities of such systems, several calibration steps are required to estimate the following quantities: the video-camera intrinsic and extrinsic parameters, the relative pose between the camera and the vehicle frame and, finally, the odometric parameters of the vehicle. In this paper the simultaneous estimation of the above mentioned quantities is achieved by a systematic and effective calibration procedure that does not require any iterative step. The calibration procedure needs only on-board measurements given by the wheels encoders, the camera and a number of properly taken camera snapshots of a set of known landmarks. Numerical simulations and experimental results with a mobile robot Khepera III equipped with a low-cost camera confirm the effectiveness of the proposed technique.

This paper proposes a multiple robot control algorithm to approach the problem of patrolling an open or closed line. The algorithm is fully decentralized, i.e., no communication occurs between robots or with a central station. Robots behave according only to their sensing and computing capabilities to ensure high scalability and robustness towards robots' fault. The patrolling algorithm is designed in the framework of behavioral control and it is based on the concept of Action: an higher level of abstraction with respect to the behaviors. Each Action is obtained by combining more elementary behaviors in the Null-Space-Behavioral framework. A Finite-State-Automata is designed as supervisor in charge of selecting the appropriate action. The approach has been validated in simulation as well as experimentally with a patrol of 3 Pioneer robots available at the Distributed Intelligence Laboratory of the University of Tennessee.