Human-multi-robot teaming in precision agriculture presents a promising approach to addressing labor shortages and managing the complexities of agricultural practices. An effective coordination of these teams, including task allocation and scheduling strategies while accounting for the inherent unpredictability of human behavior, is crucial for maximizing system productivity and ensuring user comfort. In this study, we introduce a Mixed-Integer Linear Programming (MILP) approach that aims to minimize workers’ waiting times, robots’ energy consumption during the different phases of the robots’ motions, and the overall makespan. To enhance the robustness of our framework and consider human preferences, a user interface is designed to capture real-time human feedback; then, an adaptive online updating strategy that dynamically adjusts plans responding to variations in human operators’ parameters is devised. To handle large-scale problems, we extend the solution approach by leveraging Constraint Programming (CP) combined with a batch decomposition strategy. The approach is validated through extensive simulations in a Unity-based realistic virtual reality environment and laboratory experiments using two TurtleBot2 robots and two human operators performing grape harvesting tasks.

Deploying a team of autonomous robots, operating collaboratively towards a common objective within dynamic environments, has the potential to improve the system efficiency across several fields. This paper proposes a distributed comprehensive framework enabling a networked multi-robot system to serve time-varying requests arising from different locations within the environment in a distributed and safe manner, i.e., by guaranteeing no collisions with possible obstacles and preserving connectivity among the robots. To this aim, a two-layer architecture is proposed where the top layer is in charge of distributively assigning new service requests to the robots by resorting to an auction-based algorithm, while the bottom layer is in charge of safely navigating the environment to serve the assigned requests by relying on Control Barrier Functions. However, the presence of connectivity constraints might affect the number of service requests that the multi-robot system can handle simultaneously and might lead to deadlock situations where robots cannot reach the designated locations due to loss of network connectivity. Hence, a distributed strategy based on consensus algorithms to detect and solve deadlocks in a distributed fashion is proposed. The completeness of the approach is proved. Simulation results in an agricultural setting and real-world laboratory experiments are provided to validate the effectiveness of the proposed approach.Note to Practitioners—This paper was inspired by the necessity to coordinate a team of robots to perform tasks within an unstructured agricultural field, including both the decision-making and navigation strategies, with no central control unit as envisioned by the European project CANOPIES. To this aim, a distributed approach is designed where robots only rely on local data and information from neighboring robots to assign and execute tasks effectively in a coordinated manner. In addition, as working under local communication constraints may prevent parallel execution of all tasks, potentially leading to deadlock situations, a distributed strategy is developed to enable each robot to detect and solve such situations. The proposed approach can be employed in several domains where the cooperation of multiple autonomous robots might be beneficial, ranging from logistics settings to search and rescue scenarios up to agricultural environments. Laboratory experiments with three robots demonstrate the effectiveness of the approach.

This paper introduces a control architecture that enables a robotic system to ensure the safety of human operators entering its workspace. The proposed method utilizes an appropriate metric to measure safety levels and adjusts the robot’s motion to maintain this metric above a minimum threshold. To guarantee safety, the robot scales down and deviates from its intended path. For redundant robots, internal motion is exploited to enhance safety levels further. The approach is incorporated into a Hierarchical Quadratic Programming control framework, allowing the robot to address other control objectives simultaneously, such as handling joint limits. Experimental results with a dual-arm mobile robot developed as part of the EU-funded CANOPIES project demonstrate the effectiveness of the proposed method.Note to Practitioners—This paper was motivated by the problem of ensuring human safety in unstructured environments shared with human operators. We propose a control architecture that allows complex dual-arm robotic systems to operate effectively in such scenarios. The devised architecture gives the robot the capability to slow down a trajectory to follow as well as to deviate from a nominal path to keep a human operator safe. We tested the devised approach in a precision farming setting; however, it can be adopted in any human-robot interaction scenario.

Weeds management is a repetitive and crucial task for agricultural settings. This paper considers a four-wheeled robot, called Vitirover, designed for grass-cutting and weed management tasks in vineyards. The robot steering mechanism employs differential rotation of rear wheels, mounted on a universal joint. First, the kinematic model of the robot is derived. Next, based on the kinematic model, a Model Predictive Control (MPC) formulation is designed to encourage the robot to follow a desired path while targeting weeds in the environment using dynamic weights. Simulation results in Gazebo simulator are provided to validate the overall system.

Planning in learned latent spaces helps to decrease the dimensionality of raw observations. In this work, we propose to leverage the ensemble paradigm to enhance the robustness of latent planning systems. We rely on our Latent Space Roadmap (LSR) framework, which builds a graph in a learned structured latent space to perform planning. Given multiple LSR framework instances, that differ either on their latent spaces or on the parameters for constructing the graph, we use the action information as well as the embedded nodes of the produced plans to define similarity measures. These are then utilized to select the most promising plans. We validate the performance of our Ensemble LSR (ENS-LSR) on simulated box stacking and grape harvesting tasks as well as on a real-world robotic T-shirt folding experiment.

Visual planning methods are promising to handle complex settings where extracting the system state is challenging. However, none of the existing works tackles the case of multiple heterogeneous agents which are characterized by different capabilities and/or embodiment. In this work, we propose a method to realize visual action planning in multi-agent settings by exploiting a roadmap built in a low-dimensional structured latent space and used for planning. To enable multi-agent settings, we infer possible parallel actions from a dataset composed of tuples associated with individual actions. Next, we evaluate feasibility and cost of them based on the capabilities of the multi-agent system and endow the roadmap with this information, building a capability latent space roadmap (C-LSR). Additionally, a capability suggestion strategy is designed to inform the human operator about possible missing capabilities when no paths are found. The approach is validated in a simulated burger cooking task and a real-world box packing task.

Haptic feedback is essential for humans to successfully perform complex and delicate manipulation tasks. A recent rise in tactile sensors has enabled robots to leverage the sense of touch and expand their capability drastically. However, many tasks still need human intervention/guidance. For this reason, we present a teleoperation framework designed to provide haptic feedback to human operators based on the data from camera-based tactile sensors mounted on the robot gripper. Partial autonomy is introduced to prevent slippage of grasped objects during task execution. Notably, we rely exclusively on low-cost off-the-shelf hardware to realize an affordable solution. We demonstrate the versatility of the framework on nine different objects ranging from rigid to soft and fragile ones, using three different operators on real hardware.

The federated learning (FL) paradigm aims to distribute the computational burden of the training process among several computation units, usually called agents or workers, while preserving private local training datasets. This is generally achieved by resorting to a server-worker architecture where agents iteratively update local models and communicate local parameters to a server that aggregates and returns them to the agents. However, the presence of adversarial agents, which may intentionally exchange malicious parameters or may have corrupted local datasets, can jeopardize the FL process. Therefore, we propose selective trimmed average (SETA), which is a resilient algorithm to cope with the undesirable effects of a number of misbehaving agents in the global model. SETA is based on properly filtering and combining the exchanged parameters. We mathematically prove that the proposed algorithm is resilient against data and local model poisoning attacks. Most resilient methods presented so far in the literature assume that a trusted server is in hand. In contrast, our algorithm works both in server-worker and shared memory architectures, where the latter excludes the necessity of a trusted server. The theoretical findings are corroborated through numerical results on MNIST dataset and on multiclass weather dataset (MWD).

The requirements of modern production systems together with more advanced robotic technologies have fostered the integration of teams comprising humans and autonomous robots. While this integration has the potential to provide various benefits, it also raises questions about how to effectively manage these teams, taking into account the different characteristics of the agents involved. This paper presents a framework for task allocation in a human multi-robot collaborative scenario. The proposed solution combines an optimal offline allocation with an online reallocation strategy which accounts for inaccuracies of the offline plan and/or unforeseen events, human subjective preferences and cost of task switching. Experiments with two manipulators cooperating with a human operator in a box filling task are presented.

The use of heterogeneous human-multi-robot teams enables the combination of complementary skills of these two different types of agents. To have an effective collaboration, it is necessary to define a strategy for allocating and scheduling tasks among them. In this work, we distinguish robots in working robots and service ones: working robots and human operators can perform similar tasks in the environment and both are assisted by service robots. We propose a Mixed-Integer Linear Programming approach that aims to minimize the waiting times of the working agents, the energy consumption of the service robots, and the makespan while ensuring that the velocity constraints of the robots are met and the task ordering is correct. Furthermore, we propose an online updating strategy that tackles changes in the parameters of working agents and adapts the plan accordingly based on a heuristic algorithm. To validate our framework, we analyze a precision agriculture harvesting application with two human operators, two working robots, and two service robots.

In this work, we propose a route planning strategy for heterogeneous mobile robots in Precision Agriculture (PA) settings. Given a set of agricultural tasks to be performed at specific locations, we formulate a multi-Steiner Traveling Salesman Problem (TSP) to define the optimal assignment of these tasks to the robots as well as the respective optimal paths to be followed. The optimality criterion aims to minimize the total time required to execute all the tasks, as well as the cumulative execution times of the robots. Costs for travelling from one location to another, for maneuvering and for executing the task as well as limited energy capacity of the robots are considered. In addition, we propose a sub-optimal formulation to mitigate the computational complexity by leveraging the fact that generally in PA settings only a few locations require agricultural tasks in a certain period of interest compared to all possible locations in the field. A formal analysis of the optimality gap between the optimal and the sub-optimal formulations is provided. The effectiveness of the approach is validated in a simulated orchard where three heterogeneous aerial vehicles perform inspection tasks.Note to Practitioners—This paper aims at providing an efficient solution to PA needs by deploying a team of robots able to perform agricultural tasks at given locations in large-scale orchards. In particular, a novel general optimization problem is proposed that, given a set of mobile and possibly heterogeneous robots and a set of agricultural tasks to carry out, defines the assignment of these tasks to the robots as well as the routes to follow, while minimizing the total and the cumulative execution times of the robots. Existing approaches for route optimization in PA generally involves complete coverage of the field by one or multiple robots and do not account for maneuvering costs with general layouts of the field. We consider costs for travelling from one location to another, for executing the task and for maneuvering without any restriction on the layout of the plants as well as we take into account the limited energy capacity of the robots. We also provide a sub-optimal formulation which reduces the computational burden by relaxing the optimization of the maneuvering costs at the locations where agricultural tasks are carried out and formally derive the optimality gap. The proposed approach is flexible and can be easily adapted to any PA setting involving multiple mobile robots that are required to accomplish given tasks in an area of interest. We validate its effectiveness in a realistic simulated setup composed of three heterogeneous aerial vehicles performing inspection tasks. In future research, we aim to design algorithms to solve the proposed optimization problems in an efficient manner as well as to validate the formulations on real-world robotic platforms.

In this article, we present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces, focusing on manipulation of deformable objects. We propose a latent space roadmap (LSR) for task planning, which is a graph-based structure globally capturing the system dynamics in a low-dimensional latent space. Our framework consists of the following three parts. First, a mapping module (MM) that maps observations is given in the form of images into a structured latent space extracting the respective states as well as generates observations from the latent states. Second, the LSR, which builds and connects clusters containing similar states in order to find the latent plans between start and goal states, extracted by MM. Third, the action proposal module that complements the latent plan found by the LSR with the corresponding actions. We present a thorough investigation of our framework on simulated box stacking and rope/box manipulation tasks, and a folding task executed on a real robot.

Learning state representations enables robotic planning directly from raw observations such as images. Several methods learn state representations by utilizing losses based on the reconstruction of the raw observations from a lower-dimensional latent space. The similarity between observations in the space of images is often assumed and used as a proxy for estimating similarity between the underlying states of the system. However, observations commonly contain task-irrelevant factors of variation which are nonetheless important for reconstruction, such as varying lighting and different camera viewpoints. In this work, we define relevant evaluation metrics and perform a thorough study of different loss functions for state representation learning. We show that models exploiting task priors, such as Siamese networks with a simple contrastive loss, outperform reconstruction-based representations in visual task planning in case of task-irrelevant factors of variations.

Visual action planning particularly excels in applications where the state of the system cannot be computed explicitly, such as manipulation of deformable objects, as it enables planning directly from raw images. Even though the field has been significantly accelerated by deep learning techniques, a crucial requirement for their success is the availability of a large amount of data. In this work, we propose the Augment-Connect-Explore (ACE) paradigm to enable visual action planning in cases of data scarcity. We build upon the Latent Space Roadmap (LSR) framework which performs planning with a graph built in a low dimensional latent space. In particular, ACE is used to i) Augment the available training dataset by autonomously creating new pairs of datapoints, ii) create new unobserved Connections among representations of states in the latent graph, and iii) Explore new regions of the latent space in a targeted manner. We validate the proposed approach on both simulated box stacking and real-world folding task showing the applicability for rigid and deformable object manipulation tasks, respectively.

This paper considers a scenario where a robot and a human operator share the same workspace, and the robot is able to both carry out autonomous tasks and physically interact with the human in order to achieve common goals. In this context, both intentional and accidental contacts between human and robot might occur due to the complexity of tasks and environment, to the uncertainty of human behavior, and to the typical lack of awareness of each other actions. Here, a two stage strategy based on Recurrent Neural Networks (RNNs) is designed to detect intentional and accidental contacts: the occurrence of a contact with the human is detected at the first stage, while the classification between intentional and accidental is performed at the second stage. An admittance control strategy or an evasive action is then performed by the robot, respectively. The approach also works in the case the robot simultaneously interacts with the human and the environment, where the interaction wrench of the latter is modeled via Gaussian Mixture Models (GMMs). Control Barrier Functions (CBFs) are included, at the control level, to guarantee the satisfaction of robot and task constraints while performing the proper interaction strategy. The approach has been validated on a real setup composed of a Kinova Jaco2 robot.

In this work, we address a task allocation problem for human multi-robot settings. Given a set of tasks to perform, we formulate a general Mixed-Integer Linear Programming (MILP) problem aiming at minimizing the overall execution time while optimizing the quality of the executed tasks as well as human and robotic workload. Different skills of the agents, both human and robotic, are taken into account and human operators are enabled to either directly execute tasks or play supervisory roles; moreover, multiple manipulators can tightly collaborate if required to carry out a task. Finally, as realistic in human contexts, human parameters are assumed to vary over time, e.g., due to increasing human level of fatigue. Therefore, online monitoring is required and re-allocation is performed if needed. Simulations in a realistic scenario with two manipulators and a human operator performing an assembly task validate the effectiveness of the approach.

We present a framework for visual action planning of complex manipulation tasks with high-dimensional state spaces such as manipulation of deformable objects. Planning is performed in a low-dimensional latent state space that embeds images. We define and implement a Latent Space Roadmap (LSR) which is a graph-based structure that globally captures the latent system dynamics. Our framework consists of two main components: a Visual Foresight Module (VFM) that generates a visual plan as a sequence of images, and an Action Proposal Network (APN) that predicts the actions between them. We show the effectiveness of the method on a simulated box stacking task as well as a T-shirt folding task performed with a real robot.

In this paper, a scenario of physical human-robot collaboration is considered, in which a robot is able to both carry out autonomous tasks and to physically interact with a human operator to achieve a common objective. However, since human and robot share the same workspace both accidental and intentional contacts between them might arise. Therefore, a solution based on Recurrent Neural Networks (RNNs) is proposed to detect and classify the nature of the contact with the human, even in the case the robot is interacting with the environment because of its own task. Then, reaction strategies are defined depending on the nature of contact: human avoidance with evasive action in the case of accidental interaction, and admittance control in the case of intentional interaction. In regard to the latter, Control Barrier Functions (CBFs) are considered to guarantee the satisfaction of robot constraints, while endowing the robot with a compatible compliant behavior. The approach is validated on real data acquired from the interaction with a Kinova Jaco2.

The paper presents a Distributed Fault Detection and Isolation strategy for a team of mobile manipulators performing a cooperative mission. The overall system relies on an observer-controller scheme where each robot estimates the global state of the team through a distributed observer; then, the global state estimate is used by each robot to compute the estimated local input so as to achieve a specific global task. The observer-controller scheme also allows to define a set of residual vectors that can be used by the robots to detect and isolate faults affecting any member of the team, even if not in direct communication, and without increasing the computational burden and the information exchange. The approach is validated via numerical simulations with a team of four mobile manipulators performing a transportation mission.

In this paper, a distributed scheme to allow a human operator to physically interact with a multi-manipulator system is devised. Manipulators are tightly connected to a rigid object and a human operator interacts with it to perform, for example, a cooperative transportation task. The strategy foresees two layers. The top layer is in charge of assigning a compliant behaviour to the object through an admittance model whose reference trajectory is dynamically adjusted to regulate the human-object interaction force. Moreover, since the parameters of the dynamic model of the human arm end-point are supposed to be time-varying and completely unknowns with unknown bounds, a robust adaptive control is envisaged in this layer. The output of this layer is a desired object trajectory which is tracked by the bottom layer. In detail, the latter resorts to a robust adaptive control strategy to both track the object trajectory and control the internal stresses exerted by the manipulators on the object which unavoidably arise due to dynamic and kinematic uncertainties and synchronization errors. Simulations involving a setup with three dual-arm Movo mobile robots corroborate the theoretical findings.

In this paper the modeling and planning problems of a system composed of multiple ground and aerial robots involved in a transportation task are considered. The ground robots rigidly grasp a load, while the aerial vehicles are attached to the object through non-rigid inextensible cables. The idea behind such a heterogeneous multi-robot system is to benefit of the advantages of both types of robots that might be the precision of ground robots, the increased payload of multiple aerial vehicles and their larger workspace. The overall model of the system is derived and its expression and redundancy are exploited by setting a general constrained optimal planning problem. The problem is herein solved by dynamic programming and simulation results validated the proposed scheme.