This paper presents a method of augmenting streaming point cloud data with pretouch proximity sensor information for the purposes of teleoperated grasping of transparent targets. When using commercial RGB-Depth (RGB-D) cameras, material properties can significantly affect depth measurements. In particular, transparent objects are difficult to perceive with RGB images and commercially available depth sensors. Geometric information of such objects needs to be gathered with additional sensors, and in many scenarios, it is of interest to gather this information without physical contact. In this work, a non-contact pretouch sensor fixed to the robot end effector is used to sense and explore physical geometries previously unobserved. Thus, the point cloud representation of an unknown, transparent grasp target, can be enhanced through telerobotic exploration in real-time. Furthermore, real-time haptic rendering algorithms and haptic virtual fixtures used in combination with the augmented streaming point clouds assist the teleoperator in collision avoidance during exploration. Theoretical analyses are performed to design virtual fixtures suitable for pretouch sensing, and experiments show the effectiveness of this method to gather geometry data without collision and eventually to successfully grasp a transparent object.

Robotic grasping has been hindered by the inability of robots to perceive unstructured environments. Because these environments can be complex or dynamic, it is important to obtain additional and precise sensing information just before grasping. This paper expands upon the pretouch modality by introducing a transmissive optical sensor. It can unambiguously indicate the presence or lack of objects in close proximity. A wide variety of items that other sensors fail to sense, such as extremely soft or shiny objects, can be detected by the proposed sensor. The sensor is also fully integrated into the fingertips of the PR2 robotic platform and manufactured with inexpensive, commercially-available components. Several experiments are conducted to verify its utility in both environment perception and robotic grasping. It is shown that the perception information supplied by the sensor facilitates effective robotic grasping.

This paper presents a unified framework to enable automatic exploration with a pretouch sensor to reduce object shape uncertainty before grasping. The robot starts with only the incomplete object shape data it acquires from a Kinect depth sensor-it has no model of the object. Combining the Kinect pointcloud with prior probability distributions for occlusion and transparency, it makes inferences about portions of the object that the Kinect was not able to observe. Operating on the inferred shape of the object, an iterative grasp replanning and exploration algorithm decides when further exploration is required, and where to explore in the scene using the pretouch sensor. The information gathered by the exploration action is added directly to the robot's environment representation and is considered automatically in the next grasp planning iteration.

This paper introduces seashell effect pretouch sensing, and demonstrates application of this new sensing modality to robot grasp control, and also to robot grasp planning.