Increasing the robustness of grasping actions and the recovery from failure is key to improving a robot’s autonomy. Endowing robots with the ability to robustly grasp and manipulate unknown difficult objects such as stones is required for sample collection in unknown environments. In this paper, we present a complete system for robust grasping of stones, which integrates stone segmentation based on depth information, the generation of grasp hypotheses and pushing actions as well as their execution. In particular, our system has been designed to solve these tasks on robots with limited computing resources. We evaluate the performance in real robot experiments in the context of stone sample collection. The results show that such a challenging task is achievable under computing resource constraints.

We present a soft humanoid hand with in-finger integrated cameras and in-hand real-time image processing system for fast reactive grasping. Specifically, we describe an FPGA-based, in-hand integrated, embedded system for processing visual data captured by the five in-finger cameras while avoiding high bandwidth raw data streaming via the robots real-time data bus. The hardware acceleration allows fast detection and localization of objects based on finger-camera images and provides input for a grasping controller. To this end, we implement a resource-aware encoder-decoder Convolutional Neural Network (CNN) for pixel-wise object segmentation and run inference on the in-hand embedded system at 3.58 GOPS. We evaluate the system, consisting of the soft hand with in-finger vision and the in-hand FPGA-accelerated CNN in several experiments on the humanoid robot ARMAR-6. Specifically, we evaluate the overall system response time, the ability to perform precision grasps and test reactivity and reliability that are required for handover actions. We obtain an overall system response time of 154 ms for catching a falling object and obtain a success rate of 90 % reliability for the power drill handover tasks. Further, we successfully demonstrate ability of dexterous grasping and manipulation of a pencil from a cup.

Engineering humanoid robot hands with the ability to dexterously grasp objects of different sizes, shapes, mate-rial properties and weights requires sophisticated tactile sensing and intelligent controllers able to interpret sensory information and adapt contact forces with the object to achieve a stable and safe grasp. In this paper, we present a new soft humanoid hand equipped with a multimodal sensor system in each finger and a human-inspired grasp-phases controller that is able to detect the different phases of a grasping and manipulation task, adapt interaction forces with the manipulated object and balance the force distribution in both precision and power grasps based on tactile feedback. To evaluate the controller, we conducted experiments with the hand on the humanoid robot ARMAR-6 and 31 different soft and rigid everyday objects and food items with weights ranging from 4.8 g of a paper cup to 1133.8 g of a bottle, different shapes and material properties. The results show that grasping force can be reduced by 65% compared to a naive grasping approach using maximum force for grasping and manipulating both fragile objects without destruction as well as heavy objects.

In real world applications, robotic solutions remain impractical due to the challenges that arise in unknown and unstructured environments. To perform complex manipulation tasks in complex and cluttered situations, robots need to be able to identify the interaction possibilities with the scene, i.e. the affordances of the objects encountered. In unstructured environments with noisy perception, insufficient scene understanding and limited prior knowledge, this is a challenging task. In this work, we present an approach for grasping unknown objects in cluttered scenes with a humanoid robot in the context of a nuclear decommissioning task. Our approach combines the convenience and reliability of autonomous robot control with the precision and adaptability of teleoperation in a semi-autonomous selection of grasp affordances. Additionally, this allows exploiting the expert knowledge of an experienced human worker. To evaluate our approach, we conducted 75 real world experiments with more than 660 grasp executions on the humanoid robot ARMAR-6. The results demonstrate that high-level decisions made by the human operator, supported by autonomous robot control, contribute significantly to successful task execution.

We address the question of how resource-aware concepts can be utilized in motion planning algorithms. Resource-awareness facilitate better resource allocation on global system level, e.g. when a humanoid robot needs to distribute and schedule a wide variety of concurrent algorithms. We present a motion planning approach that employs self-monitoring concepts in order to identify the difficulty of the planning problem. Resources are requested dynamically and adapted based on problem difficulty and current planning progress. We show how dynamic adaptation of resource allocation on algorithmic level can reduce the system workload as compared to static resource allocation while meeting Quality of Service (QoS) measures such as average workload or efficiency. We evaluate our approach both in several synthetic setups with varying difficulty and with the humanoid robot ARMAR-4.