Large Language Models (LLMs) have demonstrated the ability to perform semantic reasoning, planning and write code for robotics tasks. However, most methods rely on pre-existing primitives (i.e. pick, open drawer) or similar examples of robot code alone, which heavily limits their scalability to new scenarios. We present PromptBook, a collection of different prompting paradigms to generate code for successfully executing new manipulation skills. We demonstrate example-based, instruction-based and chain-of-thought to write robot code; as well as a method to build the prompt leveraging LLMs and human feedback. We show PromptBook enables LLMs to write code for new low-level manipulation skills in a zero-shot manner: from picking diverse objects, opening/closing drawers, to whisking, and waving hello. We evaluate the new skills on a mobile manipulator with 83% success rate at picking, 50-71% at opening drawers and 100% at closing them. Notably, the LLM is able to infer gripper orientation for grasping a drawer handle (z-axis aligned) vs. a top-down grasp (x-axis aligned).

We propose a framework to enable multipurpose assistive mobile robots to autonomously wipe tables to clean spills and crumbs. This problem is challenging, as it requires planning wiping actions while reasoning over uncertain latent dynamics of crumbs and spills captured via high-dimensional visual observations. Simultaneously, we must guarantee constraints satisfaction to enable safe deployment in unstructured cluttered environments. To tackle this problem, we first propose a stochastic differential equation to model crumbs and spill dynamics and absorption with a robot wiper. Using this model, we train a vision-based policy for planning wiping actions in simulation using reinforcement learning (RL). To enable zero-shot sim-to-real deployment, we dovetail the RL policy with a whole-body trajectory optimization framework to compute base and arm joint trajectories that execute the desired wiping motions while guaranteeing constraints satisfaction. We extensively validate our approach in simulation and on hardware. Video of experiments: https://youtu.be/inORKP4F3EI

Autonomous grasping of unknown objects by a robot is a highly challenging skill that is receiving increasing attention in the last years. This problem becomes still more challenging (and less explored) in underwater environments, with highly unstructured scenarios, limited availability of sensors and, in general, adverse conditions that affect in different degree the robot perception and control systems. This paper describes an approach for semi-autonomous grasping and recovery on underwater unknown objects from floating vehicles. A laser stripe emitter is attached to a robot forearm that performs a scan of a target of interest. This scan is captured by a camera that also estimates the motion of the floating vehicle while doing the scan. The scanned points are triangulated and transformed according to the motion estimation, thus recovering partial 3D structure of the scene with respect to a fixed frame. A user then indicates the part where to grab the object, and the final grasp is automatically planned on that area. The approach herein presented is tested and validated in water tank conditions.

This paper presents UWSim: a new software tool for visualization and simulation of underwater robotic missions. The software visualizes an underwater virtual scenario that can be configured using standard modeling software. Controllable underwater vehicles, surface vessels and robotic manipulators, as well as simulated sensors, can be added to the scene and accessed externally through network interfaces. This allows to easily integrate the simulation and visualization tool with existing control architectures, thus allowing hardware-in-the-loop simulations (HIL). UWSim has been successfully used for simulating the logics of underwater intervention missions and for reproducing real missions from the captured logs. The software is offered as open source, thus filling a gap in the underwater robotics community, where commercial simulators oriented to ROV pilot training predominate.

Robust grasping of objects in uncertainty conditions can be achieved with the visual monitoring of the interaction between the robot hand and the object. In this paper we propose a new approach for the visual tracking of a robot hand suitable to observe the interaction between robot and object. It consists on the continuous vision-based recovery of the articular pose of the robot hand. It is based on the principles of virtual visual servoing, which allows to deal with articulated bodies and occlusions. Its suitability is shown by tracking a parallel jaw gripper under different conditions such as self-occlusion, articulated motion and important changes in the point-of-view. The potential applications range from estimating the robot hand articulated pose under poor hand-eye calibration and joint feedback, until detecting deficient contact configurations, incipient slips, etc.

This paper presents an approach for integrating vision, tactile and force sensors in a robotic manipulation framework. Having an initial estimation of the object pose in the environment, a position-based visual servoing loop controls the hand for task execution, based on the input received from a model-based articular pose estimator following the Virtual Visual Servoing approach. The visual control is combined with another control signal obtained from tactile feedback, through a set of selection matrices that can be modified at runtime in order to select the best modality for a given cartesian degree of freedom. The result of the preliminary integration is modified by an impedance force controller, in charge of performing the task motion along the task direction, at the same time that forces are regulated on the rest of directions. The design of the controller allows to perform the task even if a sensor is not available or provides inaccurate data. Several experiments are performed, first by considering only force feedback, and then adding vision and, finally, tactile information. Errors in the estimation of the object initial position are manually introduced in the experiments, and results show how the vision-tactile-force combination is able to deal with them, performing much better than the vision-force and force-alone approaches.

In this paper we present a robot that is able to perform daily manipulation tasks in a home environment, such as opening doors and drawers. Taking as input a simplified object model and the task to perform, the robot automatically finds a grasp suitable for the task and performs it. For this, we identify a set of hand preshapes and classify them according to the grasp wrench space they generate. Given a task, the robot selects the most suitable hand preshape and automatically plans a set of actions in order to reach the object and to perform the task, taking continuously into account the task forces. The concept of hand preshape is extended for the inclusion of a task frame, which is a concept from task planning, thus filling the gap between the grasp and the task.

In this paper, we present a novel approach for sensor-guided robotic execution of everyday tasks, which is amenable to be integrated in current mobile manipulators and humanoid robots. We consider a robot which is observing simultaneously his hand and the object to manipulate, by using an external camera (i.e. robot head). Task-oriented grasping algorithms are used in order to plan a suitable grasp on the object according to the task to perform. A new vision/force coupling approach [1] is used in order to, first, guide the robot hand towards the grasp position and, second, perform the task taking into account external forces. Experimental results on a real robot are presented which validate our approach.

This work deals with the problem of dependable physical interaction through manipulation in partially-known everyday human environments. We present a modular software architecture that allows the definition and compliant execution of manipulation tasks under the Task Frame Formalism. Several software modules implemented within this architecture, enable higher levels of adaptability and robustness, as well as the incremental incorporation of more complex skills in a modular fashion. The whole system is validated making a real robot arm with a three-finger hand perform two different manipulation tasks: opening a doorhandle and taking a book out of a bookshelf.

This paper deals with the problem of dependable physical interaction through manipulation in partially-known everyday human environments. We present a modular software architecture that allows the definition and compliant execution of manipulation tasks under the task frame formalism. We show the details of several software modules implemented within this architecture, that enable higher levels of adaptability and robustness, as well as the incremental incorporation of more complex skills in a modular fashion. The whole system is validated making a real robot arm with a three-finger hand perform a complex manipulation task: taking a book out of a bookshelf. Results show how the presented framework is suitable for easily defining and performing a great variety of manipulation tasks

This paper describes design and implementation aspects of an autonomous robot that is intended to perform service tasks in a library. We first address problems concerning the arrangement of its components into a compact mobile manipulator suitable for our needs. Then, software issues are also discussed, related to the techniques that are necessary in order to navigate and to reliably manipulate books in a library. In this context, two methods are presented, concerning visual tracking and force/vision control. The first one is aimed at locating a book in a bookshelf by means of vision. For this, a tracking algorithm is implemented, which takes into account the spatio-temporal constraints of the problem. Book tags are located in the image and processed using open source optical character recognition (OCR) software, until the desired book is found. The second of the methods addresses the problem of extracting the book from the shelf. A hybrid force/vision control algorithm has been implemented, which uses vision for guiding the gripper to the correct book, and force for detecting contacts.

This paper presents work in progress towards a complete system working to assist users in a library. With this aim, the system must be capable to looking for a specific book in a shelf, asked by any user, and whether it is found, deliver it as soon as possible to the user. To get its objectives the system integrates automatic object recognition, visually guided grasping, and force feedback, among other advanced capabilities. Implementation details about the main modules developed presently are shown. Finally, after success in preliminary results obtained in our campus, we are encouraged to fallow working in this way to obtain the complete prototype.