This paper investigates the modification of an existing game console emulator utilizing scene recognition to perform audio-visual enhancement of vintage video games. Such games featured low-resolution graphics with limited colors and low-quality electronic music. Information contained about the virtual hardware state information is exploited to provide high-resolution, full-color graphics and a recorded soundtrack, complete with atmospheric effects, in real-time using the original game logic. The system developed as part of this research uses a modified version of the FCEUX emulator for the Nintendo Entertainment System. The system is capable of loading a commercially-released game, marking scenes for later recognition, mapping scenes to audio-visual events, and recognizing those scenes to enable those events. The result is an immersive game experience that far exceeds what was possible in the original game hardware.

To address the alarming decrease in students in Rhode Island computer science programs and the under-representation of women and minorities, we have devised a program to introduce students to research in computer graphics, art and new media. This program integrates good mentoring practice and pedagogy, including problem-based learning. Special attention is paid to creating a cohort of students who come together every week to learn about the research process, and ethical and societal issues related to it. Each student takes a small project from the proposal stage, through design and implementation, to publication and presentation. We report on the first two years of the program.

We present a building recognition module for an automated tour guide system which, instead of using user input or artificial landmarks, relies solely on real-time video recorded from a wearable camcorder to recognize faces of buildings and localize the user. Strong vertical edges are first detected on each frame to suggest the existence of building objects. Additional information such as color feature is collected to eliminate edges from trees, cars, and pedestrians. Vertical edges, in addition to horizontal edges, are further processed to locate window frames and thereafter faces of a building. An image section identified as a building face is then rectified for perspective distortion and recognized by a neural network. We train the neural network using whole and partial building face images before the recognition takes place.

We present a landmark-based model for agents to uniquely learn a partial map of an unknown environment. Our definition of landmark recognition and agent abilities allow different agents to construct different maps. We define landmarks in terms of feature sets that tie the various agent abilities into the landmark recognition process. The ultimate goal of having agents learn partial and unique maps is to promote different behavioral outcomes from higher level planning. Our implementation focuses on providing unexpected and believable Non-Player Characters (NPC) in modern games.

We present a 3D tracking system for detecting and tracking multiple targets. An extended Kalman Filter (EKF) is used to maintain each target's 3D state and provide location predictions to the pattern matchers whose task it is to follow the targets in images. An adaptive background modeling algorithm is used together with the tracking process to detect moving objects in complex environments. We propose a warping-based pattern matching approach to deal with object deformation during tracking. We present examples of results of our tracker for outdoor scenes.

This article presents an innovative solution to the interposition problem in augmented reality in natural, unstructured environments when the real data to augment consists of video streams provided by a stereoscopic pair of cameras. The realistic and physically-consistent embedding of a 3D virtual object into a real scene is accomplished by first considering this object as a set of independent 3D points. Rather than relying on the 3D reconstruction of the scene, our method exploits results of local stereo matching to decide whether each virtual point should be visible or not for either camera. The second phase of our method uses the spatiotemporal piecewise-constant nature of the visibility property of virtual points to build a regularized visibility map for the virtual object as a whole

An innovative solution to the interposition problem in augmented reality is presented. The realistic and physically-consistent embedding of a 3D virtual object (3DVO) into a real scene is accomplished by considering it as a set of points or vertices. Our method deals with stereoscopic views of an unstructured and unknown real scene and avoids explicit 3D reconstruction. Instead, we rely on local stereo matching to decide whether each vertex should be visible or not. After a geometrical analysis of the problem, we detail our solution and present preliminary results of interposition for 3DVOs and real scenes.

We propose a real-time visual hand tracking and posture estimation system to study prehension gestures. This system is based on an original 26 degree-of-freedom model of the hand, for which forward and inverse kinematics formulations have been developed. The hand, wearing a dark glove marked with colored cues, is tracked in real time in the image. The position of the color markers in the image is used to reconstruct the pose of the hand and the postures of the fingers. Occlusion handling is achieved by prediction of finger positions and is validated by testing 3D geometric visibility conditions. Results on real and synthesized image sequences with finger occlusions are presented.

This paper proposes a real-time visual hand tracking and posture estimation system to guide a robotic arm in gripping gestures. We present an original 26 degree-of-freedom model of the hand, for which forward and inverse kinematics formulations have been developed. The hand, wearing a dark glove marked with colored cues, is tracked in real time in the image while completing a grasping task. The position of the color markers in the image is used to reconstruct the hand's pose. Occlusions are handled by position prediction and validation. A robot gripper can then be guided by the different configurations of the hand with respect to the object to be grasped, provided the relationship between the hand and the gripper is known. Successful results for the teleoperated control of a robot manipulator in simulation are presented.

The article discusses the interposition problem in augmented reality-that is, the realistic and physically-consistent embedding of a 3D virtual object (3DVO) into a real scene. Our method deals with stereoscopic views of an unstructured and unknown real scene and avoids explicit 3D reconstruction. Instead, we consider a 3DVO as a set of vertices and rely on local stereo matching to decide whether each vertex should be visible or not. We focus on the choice of an optimal matching criterion in the particular context of interposition in augmented reality. After a brief presentation of our innovative solution, we review classical matching spaces and criteria, and select the most appropriate for our task. We compare and interpret their performance against parameters such as noise level and matching window size and present preliminary results of interposition for real scenes.

The paper presents a solution to the dynamic registration problem in augmented reality for telerobotics applications. The objective is to align the virtual camera used to render a 3D graphical replica of a robot manipulator with the real camera, providing images from the work site. We proceed in two steps: first, a static registration process requiring the intervention of the operator; next, a dynamic registration process that continuously and automatically updates the registration data. Our dynamic algorithm relies on feature tracking and an extended Kalman filter for state estimation. We present experimental results for the dynamic registration algorithm and discuss possible extensions and improvements of the method.

This project is a part of a general study of exploratory navigation by a vision-guided mobile robot maneuvering in a large, unknown, dynamic environment such as an underground mine complex. As its way of finding its way around is based on intersections, our problem is to learn and to recognize images representing intersections represented by a gray-scale 360/spl deg/ panoramic view. We propose a multilayer neural network to make associations between these representations (as input) and indexes corresponding to encountered intersections (as output).

This paper presents a study of the role of the observer in the hand/eye coordination problem. We show, or rather confirm analytically the intuitive concept that all poses of the observer are not equally good in terms of the visual servoing to perform, and that a good pose can be chosen that facilitates both the local control problem and the learning of the visuo-motor coupling. We represent the relation between the visual input and the movements of the actuators by a surface, whose shape depends on the geometry of the manipulator and on the position of the observer (camera) relative to it. If the observer is active, it can therefore, simply by changing its viewpoint, modify the shape of the surface, and sometimes even its topology. In the context of this representation, a good activity of the observer is one that achieves "desirable" effects on the shape of the surface, thereby improving the performance of control algorithms and the facilitating the learning of the visuo-motor coupling itself.

This paper presents an overview of an ongoing project about the study of vision-based interactions between mobile robots. We first introduce the concept of dynamic map, which is a unified representation of the dynamic capabilities of a robot along with its interactions with its environment. An overview of the experimental setup is given, presenting the robot architecture, including the vision-based simulated odometric system. We address the problems of the learning and execution of pursuit strategies using a neural network-based trajectory planner.

An active approach to the integration of shape from x modules-here shape from shading and shape from texture-is proposed. The question of what constitutes a good motion for the active observer is addressed. Generally, the role of the visual system is to provide depth information to an autonomous robot; a trajectory module will then interpret it to determine a motion for the robot, which in turn will affect the visual information received. It is suggested that the motion can also be chosen so as to improve the performance of the visual system.>

The authors consider the multiple applications of vision in the solution of dynamic robot and positioning problems such as catching, hitting, and intercepting a moving object. They propose a framework for qualitative reasoning about such problems, and then present a robust vision-based control strategy that uses a Kalman filter to track image features. In the formulation, a given positioning problem is transformed into one of constrained trajectory planning on a topological surface defined by the relationship between positions of the robot and observable image features. By considering only the qualitative properties of this surface, the need for precise calibration of the various components of the system is eliminated.>

It is shown that a binocular observer can recover the depth and three-dimensional motion of a rigid planar patch without using any correspondences between the left and right image frames (static) or between the successive dynamic frames (dynamic). Uniqueness and robustness issues are studied with respect to this problem and experimental results are given from the application of the theory to real images.>