International audience

International audience

Task-aware robotic grasping is a challenging problem that requires the integration of semantic understanding and geometric reasoning. This paper proposes a novel framework that leverages Large Language Models (LLMs) and Quality Diversity (QD) algorithms to enable zero-shot task-conditioned grasp synthesis. The framework segments objects into meaningful subparts and labels each subpart semantically, creating structured representations that can be used to prompt an LLM. By coupling semantic and geometric representations of an object’s structure, the LLM’s knowledge about tasks and which parts to grasp can be applied in the physical world. The QD-generated grasp archive provides a diverse set of grasps, allowing us to select the most suitable grasp based on the task. We evaluated the proposed method on a subset of the YCB dataset with a Franka Emika robot. A consolidated ground truth for task-specific grasp regions is established through a survey. Our work achieves a weighted intersection over union (IoU) of 73.6% in predicting task-conditioned grasp regions in 65 task-object combinations. An end-to-end validation study on a smaller subset further confirms the effectiveness of our approach, with 88% of responses favoring the task-aware grasp over the control group. A binomial test shows that participants significantly prefer the task-aware grasp.

Fine dexterous manipulation requires reactive control based on rich sensing of manipulator-object interactions. Tactile sensing arrays provide rich contact information across the manipulator’s surface. However their implementation faces two main challenges: accurate force estimation across complex surfaces like robotic hands, and integration of these estimates into reactive control loops. We present a data-efficient calibration method that enables rapid, full-array force estimation across varying geometries, providing online feedback that accounts for non-linearities and deformation effects. Our force estimation model serves as feedback in an online closed-loop control system for interaction force tracking. The accuracy of our estimates is independently validated against measurements from a calibrated force-torque sensor. Using the Allegro Hand equipped with Xela uSkin sensors, we demonstrate precise force application through an admittance control loop running at 100Hz, achieving up to 0.12±0.08 [N] error margin—results that show promising potential for dexterous manipulation.

Recent advances in large language models (LLMs) have led to significant progress in robotics, enabling embodied agents to understand and execute open-ended tasks. However, existing LLM-based approaches face limitations in grounding their outputs within the physical environment and aligning with the capabilities of the robot. While fine-tuning is an attractive approach to addressing these issues, the required data can be expensive to collect, especially when using very large language models. Smaller language models, while more computationally efficient, are less robust in task planning and execution, leading to a difficult trade-off between performance and tractability. In this paper, we present a novel, modular architecture designed to enhance the robustness of locally-executable LLMs in the context of robotics by addressing these grounding and alignment issues. We formalize the task planning problem within a goal-conditioned POMDP framework, identify key failure modes in LLM-driven planning, and propose targeted design principles to mitigate these issues. Our architecture introduces an "expected outcomes" module to prevent mischaracterization of subgoals and a feedback mechanism to enable real-time error recovery. Experimental results, both in simulation and on physical robots, demonstrate that our approach leads to significant improvements in success rates for pick-and-place and manipulation tasks, surpassing baselines using larger models. Through hardware experiments, we also demonstrate how our architecture can be run efficiently and locally. This work highlights the potential of smaller, locally-executable LLMs in robotics and provides a scalable, efficient solution for robust task execution and data collection.1

Personalization in social robotics is critical for fostering effective human-robot interactions, yet systems often face the cold start problem, where initial user preferences or characteristics are unavailable. This paper proposes a novel framework called USER-LLM R1 for a user-aware conversational agent that addresses this challenge through dynamic user profiling and model initiation. Our approach integrates chain-of-thought (CoT) reasoning models to iteratively infer user preferences and vision-language models (VLMs) to initialize user profiles from multimodal inputs, enabling personalized interactions from the first encounter. Leveraging a Retrieval-Augmented Generation (RAG) architecture, the system dynamically refines user representations within an inherent CoT process, ensuring contextually relevant and adaptive responses. Evaluations on the ElderlyTech-Vqa Bench demonstrate significant improvements in ROUGE-1 (+23.2%) ROUGE-2 (+0.6%) and ROUGE-L (+8%) F1 scores over state-of-the-art baselines, with ablation studies underscoring the impact of reasoning model size on performance. Human evaluations further validate the framework’s efficacy, particularly for elderly users, where tailored responses enhance engagement and trust. Ethical considerations, including privacy preservation and bias mitigation, are rigorously discussed and addressed to ensure responsible deployment.

Recent advances in AI have led to significant results in robotic learning, including natural language-conditioned planning and efficient optimization of controllers using generative models. However, the interaction data remains the bottleneck for generalization. Getting data for grasping is a critical challenge, as this skill is required to complete many manipulation tasks. Quality-Diversity (QD) algorithms optimize a set of solutions to get diverse, high-performing solutions to a given problem. This paper investigates how QD can be combined with priors to speed up the generation of diverse grasps poses in simulation compared to standard 6-DoF grasp sampling schemes. Experiments conducted on 4 grippers with 2-to-5 fingers on standard objects show that QD outperforms commonly used methods by a large margin. Further experiments show that QD optimization automatically finds some efficient priors that are usually hard coded. The deployment of generated grasps on a 2-finger gripper and an Allegro hand shows that the diversity produced maintains sim-to-real transferability. We believe these results to be a significant step toward the generation of large datasets that can lead to robust and generalizing robotic grasping policies.

Dexterous in-hand manipulation is a unique and valuable human skill requiring sophisticated sensorimotor interaction with the environment while respecting stability constraints. Satisfying these constraints with generated motions is essential for a robotic platform to achieve reliable in-hand manipulation skills. Explicitly modelling these constraints can be challenging, but they can be implicitly modelled and learned through experience or human demonstrations. We propose a learning and control approach based on dictionaries of motion primitives generated from human demonstrations. To achieve this, we defined an optimization process that combines motion primitives to generate robot fingertip trajectories for moving an object from an initial to a desired final pose. Based on our experiments, our approach allows a robotic hand to handle objects like humans, adhering to stability constraints without requiring explicit formalization. In other words, the proposed motion primitive dictionaries learn and implicitly embed the constraints crucial to the in-hand manipulation task.

Many human-robot collaboration scenarios can be seen as a redundant leader-follower setup where the human (i.e., the leader) can potentially perform the task without the assistance of the robot (i.e., the follower). Thus, the goal of the collaboration, beside stable execution of the task, is to reduce the human cost; e.g., ergonomic, or cognitive cost. Such system redundancies (where the same task be achieved in different manner) can also be exploited as a communication channel for the human to convey his/her intention to the robot; since it is essential for the overall performance (both execution and assistance) that the follower recognizes the intended task in an online fashion. Having an estimation for the intended task, the robot can assist the human by reducing the human cost over the task null-space; i.e., the null-space which arises from the overall system redundancies with respect to the intended task. With the prospective of supernumerary and prosthetic robots, in this work, we primarily focus on serial manipulation in which the proximal/distal part of the kinematic chain is controlled by the leader/follower respectively. By exploiting kinematic redundancies for intention-recognition and cost-minimization, our proposed control strategy (for the follower) ensures assistance under stable execution of the task. Our results (simulations and preliminary experimentation) show the efficacy of our method in providing a seamless robotic assistance (i.e., improving human posture) toward human intended tasks (i.e., reaching motions) for wearable robotics.

In many tasks such as finishing operations, achieving accurate force tracking is essential. However, uncertainties in the robot dynamics and the environment limit the force tracking accuracy. Learning a compensation model for these uncertainties to reduce the force error is an effective approach to overcome this limitation. However, this approach requires an adaptive and robust framework for motion and force generation. In this paper, we use the time-invariant Dynamical System (DS) framework for force adaptation in contact tasks. We propose to improve force tracking accuracy through online adaptation of a state-dependent force correction model encoded with Radial Basis Functions (RBFs). We evaluate our method with a KUKA LWR IV+ robotic arm. We show its efficiency to reduce the force error to a negligible amount with different target forces and robot velocities. Furthermore, we study the effect of the hyper-parameters and provide a guideline for their selection. We showcase a collaborative cleaning task with a human by integrating our method to previous works to achieve force, motion, and task adaptation at the same time. Thereby, we highlight the benefits of using adaptive force control in real-world environments where we need reactive and adaptive behaviours in response to interactions with the environment.

Many manipulation tasks require coordinated motions for arm and fingers. Complexity increases when the task requires to control for the force at contact against a non-flat surface; This becomes even more challenging when this contact is done on a human. All these challenges are regrouped when one, for instance, massages a human limb. When massaging, the robotic arm is required to continuously adapt its orientation and distance to the limb while the robot fingers exert desired patterns of forces and motion on the skin surface. To address these challenges, we adopt a Dynamical System (DS) approach that offers a unified motion-force control approach and enables to easily coordinate multiple degrees of freedom. As each human limb may slightly differ, we learn a model of the surface using support vector regression (SVR) which enable us to obtain a distance-to-surface mapping. The gradient of this mapping, along with the DS, generates the desired motions for the interaction with the surface. A DS-based impedance control for the robotic fingers allows to control separately for force along the normal direction of the surface while moving in the tangential plane. We validate our approach using the KUKA IIWA robotic arm and Allegro robotic hand for massaging a mannequin arm covered with a skin-like material. We show that our approach allows for 1) reactive motion planning to reach for an unknown surface, 2) following desired motion patterns on the surface, and 3) exerting desired interaction forces profiles. Our results show the effectiveness of our approach; especially the robustness toward uncertainties for shape and the given location of the surface.

Social robotic assistants have been widely studied and deployed as telepresence tools or caregivers. Evaluating their design and impact on the people interacting with them is of prime importance. In this research, we evaluate the usability and impact of ARMAR-6, an industrial robotic assistant for maintenance tasks. For this evaluation, we have used a modified System Usability Scale (SUS) to assess the general usability of the robotic system and the Godspeed questionnaire series for the subjective perception of the coworker. We have also recorded the subjects' gaze fixation patterns and analyzed how they differ when working with the robot compared to a human partner.

In this work, we present an adaptive motion planning approach for impedance-controlled robots to modify their tasks based on human physical interactions. We use a class of parameterized time-independent dynamical systems for motion generation where the modulation of such parameters allows for motion flexibility. To adapt to human interactions, we update the parameters of our dynamical system in order to reduce the tracking error (i.e., between the desired trajectory generated by the dynamical system and the real trajectory influenced by the human interaction). We provide analytical analysis and several simulations of our method. Finally, we investigate our approach through real world experiments with a 7-DOF KUKA LWR 4+ robot performing tasks such as polishing and pick-and-place.

In this paper, we present the dynamical equations of a nonlinear adaptive natural oscillator (NANO) in order to exploit the natural dynamics in robotic systems. The presented oscillator tries to minimize an energy-based cost function by adapting the shape and frequency of the reference trajectory. Stability, convergence, and optimality of this oscillator are guaranteed analytically. Moreover, the performance of this oscillator is investigated by applying it to three different types of robotic models; i.e., the pendulum, the adaptive-toy, and the hopper-leg.

Catching a fast flying object is particularly challenging as it consists of two tasks: extremely precise estimation of the object's motion and control of the robot's motion. Any small imprecision may lead the fingers to close too abruptly and let the object fly away from the hand before closing. We present a strategy to overcome for sensorimotor imprecision by introducing softness in the catching approach. Soft catching consists of having the robot moves with the object for a short period of time, so as to leave more time for the fingers to close on the object. We use a dynamic system-based control law to generate the appropriate reach and follow motion, which is expressed as a linear parameter varying (LPV) system. We propose a method to approximate the parameters of LPV systems using Gaussian mixture models, based on a set of kinematically feasible demonstrations generated by an offline optimal control framework. We show theoretically that the resulting DS will intercept the object at the intercept point, at the right time with the desired velocity direction. Stability and convergence of the approach are assessed through Lyapunov stability theory. The proposed method is validated systematically to catch three objects that generate elastic contacts and demonstrate important improvement over a hard catching approach.

We present a data-driven method for designing parallel compliance. Designing such compliance helps the system to improve energy efficiency, mainly by reducing negative work. The core idea is to design a controller first and then find springs working in parallel with each actuator such that force-displacement graph is lined up around displacement axis. By doing so, we simply shape the natural dynamics for performing the task efficiently. Maximum torque reduction for actuators is a byproduct of this design method. The method can be used in different cyclic robotic application, especially in legged locomotion systems. In this paper, we design a spinal compliance for a bounding quadruped robot in Webots. The results show that the power consumption and the maximum torque are reduced significantly.

Many daily tasks involve spatio-temporal coordination between two agents. Study of such coordinated actions in human-human and human-robot interaction has received increased attention of late. In this work, we use the mirror paradigm to study coupling of hand motion in a leader-follower game. The main aim of this study is to model the motion of the follower, given a particular motion of the leader. We propose a mathematical model consistent with the internal model hypothesis and the delays in the sensorimotor system. A qualitative comparison of data collected in four human dyads shows that it is possible to successfully model the motion of the follower.

We studied the effect of the control of an active spine versus a fixed spine, on a quadruped robot running in bound gait. Active spine supported actuation led to faster locomotion, with less foot sliding on the ground, and a higher stability to go straight forward. However, we did no observe an improvement of cost of transport of the spine-actuated, faster robot system compared to the rigid spine.