As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.

We address the problem of grasping unknown objects identified from top-down images with a parallel gripper. When no object 3D model is available, the state-of-the-art grasp generators identify the best candidate locations for planar grasps using the RGBD image. However, while they generate the Cartesian location and orientation of the gripper, the height of the grasp center is often determined by heuristics based on the highest point in the depth map, which leads to unsuccessful grasps when the objects are not thick, or have transparencies or curved shapes. In this paper, we propose to learn a regressor that predicts the best grasp height based from the image. We train this regressor with a dataset that is automatically acquired thanks to the DIGIT optical tactile sensors, which can evaluate grasp success and stability. Using our predictor, the grasping success is improved by 6% for all objects, by 16% on average on difficult objects, and by 40% for objects that are notably very difficult to grasp (e.g., transparent, curved, thin).

Teleoperation of humanoid robots enables the integration of the cognitive skills and domain expertise of humans with the physical capabilities of humanoid robots. The operational versatility of humanoid robots makes them the ideal platform for a wide range of applications when teleoperating in a remote environment. However, the complexity of humanoid robots imposes challenges for teleoperation, particularly in unstructured dynamic environments with limited communication. Many advancements have been achieved in the last decades in this area, but a comprehensive overview is still missing. This survey article gives an extensive overview of humanoid robot teleoperation, presenting the general architecture of a teleoperation system and analyzing the different components. We also discuss different aspects of the topic, including technological and methodological advances, as well as potential applications.

Grasping made impressive progress during the last few years thanks to deep learning. However, there are many objects for which it is not possible to choose a grasp by only looking at an RGB-D image, might it be for physical reasons (e.g., a hammer with uneven mass distribution) or task constraints (e.g., food that should not be spoiled). In such situations, the preferences of experts need to be taken into account. In this paper, we introduce a data-efficient grasping pipeline (Latent Space GP Selector - LGPS) that learns grasp prefer-ences with only a few labels per object (typically 1 to 4) and generalizes to new views of this object. Our pipeline is based on learning a latent space of grasps with a dataset generated with any state-of-the-art grasp generator (e.g., Dex-Net). This latent space is then used as a low-dimensional input for a Gaussian process classifier that selects the preferred grasp among those proposed by the generator. The results show that our method outperforms both GR-ConvNet and GG-CNN (two state-of-the-art methods that are also based on labeled grasps) on the Cornell dataset, especially when only a few labels are used: only 80 labels are enough to correctly choose 80% of the grasps (885 scenes, 244 objects). Results are similar on our dataset (91 scenes, 28 objects).

The use of drones in human-populated areas is increasing day by day. Such robots flying in close proximity to humans and potentially interacting with them, as in object handover or delivery, need to carefully plan their navigation considering the presence of humans. We propose a humanaware 3D reactive planner based on stochastic optimization for drone navigation. Besides considering the kinematics constraints of the drone, we propose two criteria to produce socially acceptable trajectories. The first, called discomfort, considers the unease caused to the humans spatially close to fast-moving drones. The second, called visibility, promotes the drone's visibility for humans. We demonstrate the planner's performance and adaptability in various simulated experiments.

Designing controllers for complex robots such as humanoids is not an easy task. Often, researchers hand-tune controllers, but this is a time-consuming approach that yields a single controller which cannot generalize well to varied tasks. This work presents a method which uses the NSGA-II multi-objective optimization algorithm with various training trajectories to output a diverse Pareto set of well-functioning controller weights and gains. The best of these are shown to also work well on the real Talos robot. The learned Pareto front is then used in a Bayesian optimization (BO) algorithm both as a search space and as a source of prior information in the initial mean estimate. This combined learning approach, leveraging the two optimization methods together, finds a suitable parameter set for a new trajectory within 20 trials and outperforms both BO in the continuous parameter search space and random search along the precomputed Pareto front. The few trials required for this formulation of BO suggest that it could feasibly be applied on the physical robot using a Pareto front generated in simulation.

Overheadwork is classifiedas one of the major risk factors for the onset of shoulder work-related musculoskeletal disorders and muscle fatigue. Upper-limb exoskeletons can be used to assist workers during the execution of industrial overhead tasks to prevent such disorders. Twelve novice participants have been equipped with inertial and force/torque sensors to simultaneously estimate the whole-body kinematics and the joint torques (i.e., internal articular stress) by means of a probabilistic estimator, while performing an overhead task with a pointing tool. An evaluation has been performed to analyze the effect at the whole-body level by considering the conditions of wearing and not-wearing PAEXO, a passive exoskeleton for upper-limb support during overhead work. Results point out that PAEXO provides a reduction of the whole-body joint effort across the experimental task blocks (from 66% to 86%). Moreover, the analysis along with five different body areas shows that 1) the exoskeleton provides support at the human shoulders by reducing the joint effort at the targeted limbs, and 2) that part of the internal wrenches is intuitively transferred from the upper body to the thighs and legs, which is shown with an increment of the torques at the legs joints. The promising outcomes show that the probabilistic estimation algorithm can be used as a validation metric to quantitatively assess PAEXO performances, paving thus the way for the next challenging milestone, such as the optimization of the human joint torques via adaptive exoskeleton control.

Human-humanoid collaborative tasks require that the robot take into account the goals of the task, interaction forces with the human, and its own balance. We present a formulation for a real-time humanoid controller which allows the robot to keep itself balanced, while also assisting the human in achieving their shared objectives. We achieve this with a multi-robot quadratic program controller, which solves for human dynamics reconstruction and optimal robot controls in a single optimization problem. Our experiments on a simulated robot platform demonstrate the ability to generate interaction motions and forces that are similar to what a human collaborator would produce.

One of the key problems in planning and control of redundant robots is the fast generation of controls when multiple tasks and constraints need to be satisfied. In the literature, this problem is classically solved by multi-task prioritized approaches, where the priority of each task is determined by a weight function, describing the task strict/soft priority. In this paper, we propose to leverage machine learning techniques to learn the temporal profiles of the task priorities, represented as parametrized weight functions: we automatically determine their parameters through a stochastic optimization procedure. We show the effectiveness of the proposed method on a simulated 7 DOF Kuka LWR and both a simulated and a real Kinova Jaco arm. We compare the performance of our approach to a state-of-the-art method based on soft task prioritization, where the task weights are typically hand-tuned.

In whole-body control, joint torques and external forces need to be estimated accurately. In principle, this can be done through pervasive joint-torque sensing and accurate system identification. However, these sensors are expensive and may not be integrated in all links. Moreover, the exact position of the contact must be known for a precise estimation. If contacts occur on the whole body, tactile sensors can estimate the contact location, but this requires a kinematic spatial calibration, which is prone to errors. Accumulating errors may have dramatic effects on the system identification. As an alternative to classical model-based approaches we propose a data-driven mixture-of-experts learning approach using Gaussian processes. This model predicts joint torques directly from raw data of tactile and force/torque sensors. We compare our approach to an analytic model-based approach on real world data recorded from the humanoid iCub. We show that the learned model accurately predicts the joint torques resulting from contact forces, is robust to changes in the environment and outperforms existing dynamic models that use of force/ torque sensor data.

Classically robot force control passes through joint torques measurement or estimation. Within this context, classical torque sensing technologies rely on current sensing on motor windings and on torsion sensing on motor shaft. An alternative approach was recently proposed in [1] and combines whole-body distributed 6-axis force/torque (F/T) sensors, gyroscopes, accelerometers and tactile sensors (i.e. artificial skin). A further advantage of this method is that it simultaneously estimates (internal) joint torques and (external) contact forces with no need of joint redesign. As a drawback, the method relies on a model of the robot dynamics, as it consists on reordering the classical recursive Newton-Euler algorithm (RNEA). In this paper we consider the problem of the parametric identification of the robot dynamic model from embedded F/T sensors. We extend recent results on parametric identification [2] by considering an arbitrary reordering of the classical RNEA. The theoretical framework is validated on the iCub humanoid, which is equipped with both 6-axis F/T sensors and joint torque sensors. We estimated the system inertial parameters using only one F/T sensor. We used the obtained parameters to estimate the joint torques (as proposed in [1]) and compared the results with direct joint torque measurements, used in this context only as a ground truth.

In the last years, several regression algorithms have been proposed to learn accurate mechanical models of robots. Comparisons are proposed at the conceptual level or through the use of recorded databases, but they deliver limited conclusions with respect to the real performance of these algorithms in their true context of use, i.e. online learning on the real robot interacting with its environment, within a feedback control loop. In this paper, we provide an empirical study of three state-of-the-art regression methods through online learning on the iCub robot holding a tool. We show that they can effectively learn a visuo-motor kinematic model for a simple visual servoing task in a very limited time (few minutes), without making any a priori hypothesis on the geometry of the robot and its tool. Furthermore, we can draw from the results some stronger conclusions about the comparison of the algorithms than previous studies based on databases.

Muscle co-contraction can be modeled as an active modulation of the passive musculo-skeletal compliance. Within this context, recent findings in human motor control have shown that active compliance modulation is fundamental when planning movements in presence of unpredictability and uncertainties. Along this line of research, this paper investigates the link between active impedance control and unpredictability, with special focus on robotic applications. Different types of actuators are considered and confronted to extreme situations such as moving in an unstable force field and controlling a system with significant delays in the feedback loop. We use tools from stochastic optimal control to illustrate the possibility of optimally planning the intrinsic system stiffness when performing movements in such situations. In the extreme case of total feedback absence, different actuators model are considered and their performance in dealing with unpredictability compared. Finally, an application of the proposed theories on planning reaching movements with the iCub humanoid platform is proposed.

Online optimal planning of robotic arm movement is addressed. Optimality is inspired by computational models, where a “cost function” is used to describe limb motions according to different criteria. A method is proposed to implement optimal planning in Cartesian space, minimizing some cost function, by means of numerical approximation to a generalized nonlinear model predictive control problem. The Extended RItz Method is applied as a functional approximation technique. Differently from other approaches, the proposed technique can be applied on platforms with strict control temporal constraints and limited processing capability, since the computational burden is completely concentrated in an off-line phase. The trajectory generation on-line is therefore computationally efficient. Task to joint space conversion is implemented on-line by a closed loop inverse kinematics algorithm, taking into account the robot's physical limits. Experimental results, where a 4DOF arm moves according to a particular nonlinear cost, show the effectiveness of the proposed approach, and suggest interesting future developments.

The Extended RItz Method (ERIM) can be used to face optimal decision and control problems when finding the global solution is hard, because the problem is ill-conditioned or we can only compute the solution via numerical approximations. It consists in constraining the control functions to take on a fixed structure with a certain number of free parameters to be optimized. We will show the use of such method for the solution of a communication problem in a mixed (analog/digital) transmission environment. A noisy channel is used to convey information from a limited-energy analog device to a sink; in the presence of a binary link, how can we reduce the energy spent for transmission without renouncing reconstruction capability and real-time encoding?