We present a method for performing object recognition using multiple images acquired from a tactile sensor. The method relies on using the tactile sensor as an imaging device, and builds an object representation based on mosaics of tactile measurements. We then describe an algorithm that is able to recognize an object using a small number of tactile sensor readings. Our approach makes extensive use of sequential state estimation techniques from the mobile robotics literature, whereby we view the object recognition problem as one of estimating a consistent location within a set of object maps. We examine and test approaches based on both traditional particle filtering and histogram filtering. We demonstrate both the mapping and recognition / localization techniques on a set of raised letter shapes using real tactile sensor data.

This paper explores the connection between sensor-based perception and exploration in the context of haptic object identification. The proposed approach combines 1) object recognition from tactile appearance with 2) purposeful haptic exploration of unknown objects to extract appearance information. The recognition component brings to bear computer-vision techniques by viewing tactile-sensor readings as images. We present a bag-of-features framework that uses several tactile-image descriptors, some that are adapted from the vision domain and others that are novel, to estimate a probability distribution over object identity as an unknown object is explored. Haptic exploration is treated as a search problem in a continuous space to take advantage of sampling-based motion planning to explore the unknown object and construct its tactile appearance. Simulation experiments of a robot arm equipped with a haptic sensor at the end-effector provide promising validation, thereby indicating high accuracy in identifying complex shapes from tactile information gathered during exploration. The proposed approach is also validated by using readings from actual tactile sensors to recognize real objects.

We describe a general methodology for tracking 3-dimensional objects in monocular and stereo video that makes use of GPU-accelerated filtering and rendering in combination with machine learning techniques. The method operates on targets consisting of kinematic chains with known geometry. The tracked target is divided into one or more areas of consistent appearance. The appearance of each area is represented by a classifier trained to assign a class-conditional probability to image feature vectors. A search is then performed on the configuration space of the target to find the maximum likelihood configuration. In the search, candidate hypotheses are evaluated by rendering a 3D model of the target object and measuring its consistency with the class probability map. The method is demonstrated for tool tracking on videos from two surgical domains, as well as in a human hand-tracking task.

Certain surgical procedures require a high degree of precise manual control within a very restricted area. Retinal surgeries are part of this group of procedures. During vitreoretinal surgery, the surgeon must visualize, using a microscope, an area spanning a few hundreds of microns in diameter and manually correct the potential pathology using direct contact, free hand techniques. In addition, the surgeon must find an effective compromise between magnification, depth perception, field of view, and clarity of view. Pre-operative images are used to locate interventional targets, and also to assess and plan the surgical procedure. This paper proposes a method of fusing information contained in pre-operative imagery, such as fundus and OCT images, with intra-operative video to increase accuracy in finding the target areas. We describe methods for maintaining, in real-time, registration with anatomical features and target areas using image processing. This registration allows us to produce information enhanced displays that ensure that the retinal surgeon is always in visual contact with his/her area of interest.

Human machine collaborative systems (HMCS) have been developed to enhance sensation and suppress extraneous motions or forces during surgical tasks requiring precise motion. However, to date such systems have enforced constraints on the position or path of a tool, but have not considered the dynamics of motion. Also, the focus has been on the effect of guidance of motion during a task, rather than on the learning of motion skills through repetition. We present a pseudo-admittance framework for HMCS design to guide the user's velocity in such tasks. Two different fixture design approaches are analyzed, implemented and compared. Three tests are then conducted, showing the fixtures' promise for both guiding and learning motions with dynamics

In this paper, we describe a robot that interacts with humans in a crowded conference environment. The robot detects faces, determines the shirt color of onlooking conference attendants, and reacts with a combination of speech, musical, and movement responses. It continuously updates an internal emotional state, modeled realistically after human psychology research. Using empirically-determined mapping functions, the robot's state in the emotion space is translated to a particular set of sound and movement responses. We successfully demonstrate this system at the AAAI '05 Open Interaction Event, showing the potential for emotional modeling to improve human-robot interaction.