Even as autonomous capabilities improve, many robot manipulation tasks require human(s)-in-the-loop to resolve high-level problems in uncertain environments or ambiguous situations. Prior work in highly autonomous applications tends to use interfaces with few human interface modalities, poten-tially missing out on the benefits that multimodal interfaces have demonstrated in lower-level operation. In this work, we demonstrate a system with a multimodal interface for a controlling a robot at a high level of autonomy. This example highlights how multiple modalities could enable redundant and robust interactions, increased situational awareness, and compact representations of complex commands, such as how to grasp an object.

Apple picking is a challenging manipulation task, but it is difficult to test solutions due to the limited window of time that apples are in season. Previous methods have built simulations of apple trees, but simulations rarely capture soft contact and deformation well, both of which are common in fruit picking. In this paper we present and validate a physical proxy that replicates the mechanics of a real world apple pick. This proxy, in conjunction with a novel hand with multiple sensors, enables large-scale capture of sensor data for data collection and testing. To validate our approach, we train a Long Short-Term Memory network to classify a pick as successful or failed based on sensor feedback from the robot hand. We show that a network trained on the proxy performs as well, or even better, than a network trained solely on real apple trees, with accuracies up to 90%. We determine which sensors are most important for pick classification and also demonstrate that our proxy preserves the most important sensor feature data for pick classification. For our hand, the most informative sensor was the finger's servomotor effort.

Studies have shown that picking techniques play an important role in determining fruit quality at harvest (e.g. bruising, stem retention, etc). When picking fruit such as apples and pears, professional pickers use active perception, incorporating both visual and tactile input about fruit orientation, stem location, and the fruit’s immediate surroundings. This combination of tactile, visual, and force feedback is what enables human workers to execute dynamic movements that quickly and efficiently remove fruit from the tree without damage. However, much of the prior work on robotic fruit picking has formulated the harvesting problem as a position-control problem, using visual feedback for closed-loop end-effector placement while disregarding feedback on physical contact. As a first step towards more intelligent fruit picking — combining proprioception, localized sensing, and observed forces — we have developed a custom end-effector with multiple in-hand sensors, including tactile sensors on the fingertips. This paper presents the mechatronic design of the device as well as results from multiple outdoor picking trials with a Honeycrisp apple tree. Preliminary results show that, with multi-modal sensing, fruit slip, fruit separation from the tree, and fruit release from the hand can be detected.