Tele-operation can enable an operator to control a robot remotely in inaccessible and hazardous environments. However, controlling a robot remotely via a conventional monitor and control panel can be difficult and slow. To achieve faster and more dexterous operation of the robot, enhanced 3D perception and some form of tactile or neural feedback is needed to achieve some degree of embodiment within the robot's physical structure and world. To achieve this objective we have devised an immersive tele-operation system comprised of a stereo vision headset and an electro-tactile feedback system that is worn by the operator, connected to stereo cameras and various sensors mounded on the robot. This arrangement enables the remote operator to see in 3D what the robot sees and experience what the robot feels via electro-tactile feedback in response to hand gesture based control actions. We provide experimental results showing how our tele-operation system can enable the operator to achieve better control of a mobile robot via a sense of being partially "embodied" within the robot.

This paper presents a novel human-computer interface that enables the computer display to be perceived without any use of the eyes. Our system works by tracking the user's head position and orientation to obtain their 'gaze' point on a virtual screen, and by indicating to the user what object is present at the gaze location via haptic feedback to the fingers and synthetic speech or Braille text. This is achieved by using the haptic vibration frequency delivered to the fingers to indicate the type of screen object at the gaze position, and the vibration amplitude to indicate the screen object's window-layer, when the object is contained in overlapping windows. Also, objects that are gazed at momentarily have their name output to the user via a Braille display or synthetic speech. Our experiments have shown that by browsing over the screen and receiving haptic and voice (or Braille) feedback in this manner, the user is able to acquire a mental two-dimensional representation of the virtual screen and its content without any use of the eyes. This form of blind screen perception can then be used to locate screen objects and controls and manipulate them with the mouse or via gaze control. Our experimental results are provided in which we demonstrate how this form of blind screen perception can effectively be used to exercise point-and-click and drag-and-drop control of desktop objects and open windows by using the mouse, or the user's head pose, without any use of the eyes.

The development of effective user interfaces, appropriate sensors, and information processing techniques for enabling the blind to achieve additional perception of the environment is a relentless challenge confronting HCI and sensor researchers. To address this challenge we have developed a novel 3D vision system that can enable the 3D structure of the immediate environment to be perceived via head mounted stereo video cameras and electro-tactile data gloves without requiring any use of the eyes. The electro-neural vision system (ENVS) works by extracting a depth map from the camera images by measuring the disparity between the stereo images. This range data is then delivered to the fingers via electro-neural stimulation to indicate to the user the range of objects being viewed by the cameras. To interpret this information, the user only has to imagine that the hands are held in the direction viewed by the cameras, with fingers extended, and the amount of stimulation felt by each finger indicates the range of objects in the direction pointed at by each finger. This intuitive means of perceiving the 3D structure of the environment in real time effectively enables the user to navigate the environment without use of the eyes or other blind aids. Experimental results are provided demonstrating the potential that this form of 3D environment perception has at enabling the user to achieve localisation and obstacle avoidance skills without using the eyes.

We describe an unsupervised robot learning method which is based on the robot learning a mapping between sensors and trajectory velocities. This enables the robot to acquire object avoidance, wall following and goal seeking behaviors simultaneously without incurring the credit assignment problem. To improve the robot's perception and behaviors we provide the robot with 7 fuzzy associative matrices (FAMs) so that sensors can be mapped to each trajectory independently. We provide results demonstrating how a mobile robot equipped with 16 sonar sensors is able to achieve improved perception and behaviors by using 7 FAMs to map sensors to trajectories.