Visual Reinforcement Learning (RL) methods often require extensive amounts of data. As opposed to model-free RL, model-based RL (MBRL) offers a potential solution with efficient data utilization through planning. Additionally, RL lacks generalization capabilities for real-world tasks. Prior work has shown that incorporating pre-trained visual representations (PVRs) enhances sample efficiency and generalization. While PVRs have been extensively studied in the context of model-free RL, their potential in MBRL remains largely unexplored. In this paper, we benchmark a set of PVRs on challenging control tasks in a model-based RL setting. We investigate the data efficiency, generalization capabilities, and the impact of different properties of PVRs on the performance of model-based agents. Our results, perhaps surprisingly, reveal that for MBRL current PVRs are not more sample efficient than learning representations from scratch, and that they do not generalize better to out-of-distribution (OOD) settings. To explain this, we analyze the quality of the trained dynamics model. Furthermore, we show that data diversity and network architecture are the most important contributors to OOD generalization performance.

Assessing grasp quality and, subsequently, predicting grasp success is useful for avoiding failures in many autonomous robotic applications. In addition, interest in nonprehensile grasping and manipulation has been growing as it offers the potential for a large increase in dexterity. However, while force-closure grasping has been the subject of intense study for many years, few existing works have considered quality metrics for non-prehensile grasps. Furthermore, no studies exist to validate them in practice. In this work we use a real-world data set of non-prehensile balancing grasps and use it to experimentally validate a wrench-based quality metric by means of its grasp success prediction capability. The overall accuracy of up to 84 % is encouraging and in line with existing results for force-closure grasps.

Robotic technology has transformed manufacturing industry ever since the first industrial robot was put in use in the beginning of the 60s. The challenge of developing flexible solutions where production lines can be quickly re-planned, adapted and structured for new or slightly changed products is still an important open problem. Industrial robots today are still largely preprogrammed for their tasks, not able to detect errors in their own performance or to robustly interact with a complex environment and a human worker. The challenges are even more serious when it comes to various types of service robots. Full robot autonomy, including natural interaction, learning from and with human, safe and flexible performance for challenging tasks in unstructured environments will remain out of reach for the foreseeable future. In the envisioned future factory setups, home and office environments, humans and robots will share the same workspace and perform different object manipulation tasks in a collaborative manner. We discuss some of the major challenges of developing such systems and provide examples of the current state of the art.

This paper is concerned with automating fleets of autonomous robots. This involves solving a multitude of problems, including goal assignment, motion planning, and coordination, while maximizing some performance criterion. While methods for solving these sub-problems have been studied, they address only a facet of the overall problem, and make strong assumptions on the use-case, on the environment, or on the robots in the fleet. In this paper, we formulate the overall fleet management problem in terms of Optimal Control. We describe a scheme for solving this problem in the particular case of fleets of non-holonomic robots navigating in an environment with obstacles. The method is based on a two-phase approach, whereby the first phase solves for fleet-wide boolean decision variables via Mixed Integer Quadratic Programming, and the second phase solves for real-valued variables to obtain an optimized set of trajectories for the fleet. Examples showcasing the features of the method are illustrated, and the method is validated experimentally.

This article presents an approach for assisted teleoperation of a robot arm, formulated within a real-time stack-of-tasks (SoT)whole-body motion control framework. The approach leverages the hierarchical nature of the SoT framework to integrate operator commands with assistive tasks, such as joint limit and obstacle avoidance or automatic gripper alignment. Thereby some aspects of the teleoperation problem are delegated to the controller and carried out autonomously. The key contributions of this work are two-fold: the first is a method for unobtrusive integration of autonomy in a telemanip-ulation system; and the second is a user study evaluation of the proposed system in the context of teleoperated pick-and-place tasks. The proposed approach of assistive control was found to result in higher grasp success rates and shorter trajectories than achieved through manual control, without incurring additional cognitive load to the operator.

Wrench-based quality metrics play an important role in many applications such as grasp planning or grasp success prediction. In this work, we study the following discrepancy which is frequently overlooked in practice: the quality metrics are commonly computed under the assumption of sum-magnitude bounded contact forces, but the corresponding grasps are executed by a fully actuated device where the contact forces are limited independently. By means of experiments carried out in simulation and on real hardware, we show that in this setting the values of these metrics are severely underestimated. This can lead to erroneous conclusions regarding the actual capabilities of the grasps under consideration. Our findings highlight the importance of matching the physical properties of the task and the grasping device with the chosen quality metrics.

Predicting grasp success is useful for avoiding failures in many robotic applications. Based on reasoning in wrench space, we address the question of how well analytic grasp success prediction works if tactile feedback is incorporated. Tactile information can alleviate contact placement uncertainties and facilitates contact modeling. We introduce a wrench-based classifier and evaluate it on a large set of real grasps. The key finding of this work is that exploiting tactile information allows wrench-based reasoning to perform on a level with existing methods based on learning or simulation. Different from these methods, the suggested approach has no need for training data, requires little modeling effort and is computationally efficient. Furthermore, our method affords task generalization by considering the capabilities of the grasping device and expected disturbance forces/moments in a physically meaningful way.

Grasping systems that build upon meticulously planned hand postures rely on precise knowledge of object geometry, mass and frictional properties — assumptions which are often violated in practice. In this work, we propose an alternative solution to the problem of grasp acquisition in simple autonomous pick and place scenarios, by utilizing the concept of grasp envelopes: sets of constraints on gripper postures. We propose a fast method for extracting grasp envelopes for objects that fit within a known shape category, placed in an unknown environment. Our approach is based on grasp envelope primitives, which encode knowledge of human grasping strategies. We use environment models, reconstructed from noisy sensor observations, to refine the grasp envelope primitives and extract bounded envelopes of collision-free gripper postures. Also, we evaluate the envelope extraction procedure both in a stand alone fashion, as well as an integrated component of an autonomous picking system.

In this work we tackle the problem of planning grasps for an underactuated gripper which enable it to retrieve target objects from a cluttered environment. Furthermore, we investigate how additional manipulation capabilities of the gripping device, provided by active surfaces on the inside of the fingers, can lead to performance improvement in the grasp execution process. To this end, we employ a simple strategy, in which the target object is `pulled-in' towards the palm during grasping which results in firm enveloping grasps. We show the effectiveness of the suggested methods by means of experiments conducted in a real-world scenario.

The synthesis of multi-fingered grasps on nontrivial objects requires a realistic representation of the contact between the fingers of a robotic hand and an object. In this work, we use a patch contact model to approximate the contact between a rigid object and a deformable anthropomorphic finger. This contact model is utilized in the computation of Independent Contact Regions (ICRs) that have been proposed as a way to compensate for shortcomings in the finger positioning accuracy of robotic grasping devices. We extend the ICR algorithm to account for the patch contact model and show the benefits of this solution.

The synthesis and evaluation of multi-fingered grasps on complex objects is a challenging problem that has received much attention in the robotics community. Although several promising approaches have been developed, applications to real-world systems are limited to simple objects or gripper configurations. The paradigm of Independent Contact Regions (ICRs) has been proposed as a way to increase the tolerance to grasp positioning errors. This concept is well established, though only on precise geometric object models. This work is concerned with the application of the ICR paradigm to models reconstructed from real-world range data. We propose a method for increasing the robustness of grasp synthesis on uncertain geometric models. The sensitivity of the ICR algorithm to noisy data is evaluated and a filtering approach is proposed to improve the quality of the final result.

The concept of independent contact regions on a target object's surface, in order to compensate for shortcomings in the positioning accuracy of robotic grasping devices, is well known. However, the numbers and distributions of contact points forming such regions is not unique and depends on the underlying computational method. In this work we present a computation scheme allowing to prioritize contact points for inclusion in the independent regions. This enables a user to affect their shape in order to meet the demands of the targeted application. The introduced method utilizes frictionless contact constraints and is able to efficiently approximate the space of disturbances resistible by all grasps comprising contacts within the independent regions.

Since the introduction of independent contact regions in order to compensate for shortcomings in the positioning accuracy of robotic hands, alternative methods for their generation have been proposed. Due to the fact that (in general) such regions are not unique, the computation methods used usually reflect the envisioned application and/or underlying assumptions made. This paper introduces a parallelizable algorithm for the efficient computation of independent contact regions, under the assumption that a user input in the form of initial guess for the grasping points is readily available. The proposed approach works on discretized 3D-objects with any number of contacts and can be used with any of the following models: frictionless point contact, point contact with friction and soft finger contact. An example of the computation of independent contact regions comprising a non-trivial task wrench space is given.