Food recognition is difficult because food items are de-formable objects that exhibit significant variations in appearance. We believe the key to recognizing food is to exploit the spatial relationships between different ingredients (such as meat and bread in a sandwich). We propose a new representation for food items that calculates pairwise statistics between local features computed over a soft pixel-level segmentation of the image into eight ingredient types. We accumulate these statistics in a multi-dimensional histogram, which is then used as a feature vector for a discriminative classifier. Our experiments show that the proposed representation is significantly more accurate at identifying food than existing methods.

We consider the problem of zero-shot learning, where the goal is to learn a classifier $f: X \rightarrow Y$ that must predict novel values of $Y$ that were omitted from the training set. To achieve this, we define the notion of a semantic output code classifier (SOC) which utilizes a knowledge base of semantic properties of $Y$ to extrapolate to novel classes. We provide a formalism for this type of classifier and study its theoretical properties in a PAC framework, showing conditions under which the classifier can accurately predict novel classes. As a case study, we build a SOC classifier for a neural decoding task and show that it can often predict words that people are thinking about from functional magnetic resonance images (fMRI) of their neural activity, even without training examples for those words.

Registration of medical images enables quantitative study of anatomical differences between populations, as well as detection of abnormal variations indicative of pathologies. However inherent anatomical variabilities between individuals and possible pathologies make registration difficult. This paper presents a bootstrap strategy for characterizing non-pathological variations in human brain anatomy, as well its application to achieve accurate 3-D deformable registration. Inherent anatomical variations are initially extracted by deformably registering training data with an expert-segmented 3-D image, a digital brain atlas. Statistical properties of the density and geometric variations in brain anatomy are extracted and encoded into the atlas to build a statistical atlas. These statistics are then used as prior knowledge to guide the deformation process. A bootstrap loop is formed by registering the statistical atlas to larger training sets as more data becomes available, so as to ensure more robust knowledge extraction, and to achieve more precise registration. Compared to an algorithm with no knowledge guidance, registration using the statistical atlas reduces the overall error by 34%.

We study an application of image registration in the medical domain. Based on a 3-D hierarchical deformable registration algorithm, we have developed a prototype system which automatically aligns a standard atlas to a subject's data to create a customized atlas. Combined with domain knowledge, the registration algorithm can also detect asymmetries and abnormal variations in the subject's data that indicate the existence and location of pathologies. We have conducted tests on 106 MRI scans of normal brains, 3 MRI and 1 CT scan of brains with pathologies, with results qualitatively comparable to manual segmentation.

This paper discusses how we integrated our 3D car tracking approach with the lane following module RALPH on the Navlab autonomous vehicles, obtaining a hybrid vision system that tracks both the road and cars better than those two systems in isolation. The tracking system brings precise and crisp measurements of the car in the image, and performs image stabilization. However, because it does not know, about the yaw or lateral offset of the ego-vehicle, its curvature estimate can be misguided. RALPH takes a more global image processing approach and can provide this missing information, as well as a good estimate of curvature, so that the combined curvature estimate is superior to both taken in isolation. The additional information provided by RALPH also improves tracking performance, and allows us to estimate properties of the tracked car that were previously unobservable, in particular its in-lane displacement. Better car tracking, and a better idea of where the road is, gives us a substantial foundation on which to base other capabilities needed to realize fully autonomous vehicles.

AURORA is a vision-based system designed to warn a vehicle driver of possible impending roadway departure accidents. It employs a downward looking color video camera with a wide angle lens, a digitizer, and a portable Sun Sparc workstation. Using a binormalized adjustable template correlation algorithm, it reliably detects lane markers on structured roads at 60 Hz. A time-to-lane-crossing (TLC) measurement is calculated for each image based on the estimation of vehicle's lateral position and velocity. This measurement is used to trigger an alarm when the TLC falls below a preset threshold. Promising results have been achieved under a variety of weather and lighting conditions, on many road types.

The use of artificial neural networks in the domain of autonomous driving has produced promising results. ALVINN has shown that a neural system can drive a vehicle reliably and safely on many different types of roads, ranging from paved paths to interstate highways. The next step in the evolution of autonomous driving systems is to intelligently handle road junctions. In this paper the authors present an addition to the basic ALVINN driving system which makes autonomous detection of roads and traversal of simple intersections possible. The addition is based on geometrically modelling the world, accurately imaging interesting parts of the scene using this model, and monitoring ALVINN's response to the created image.

In many vision based tasks, the ability to focus attention on the important portions of a scene is crucial for good performance on the tasks. In this paper we present a simple method of achieving spatial selective attention through the use of a saliency map. The saliency map indicates which regions of the input retina are important for performing the task. The saliency map is cre(cid:173) ated through predictive auto-encoding. The performance of this method is demonstrated on two simple tasks which have multiple very strong distract(cid:173) ing features in the input retina. Architectural extensions and application directions for this model are presented.

Each year people spend a huge amount of time typing. The text people type typically contains a tremendous amount of redundancy due to predictable word usage patterns and the text's structure. This paper describes a neural network system call AutoTypist that monitors a person's typing and predicts what will be entered next. AutoTypist displays the most likely subsequent word to the typist, who can accept it with a single keystroke, instead of typing it in its entirety. The multi-layer perceptron at the heart of Auto'JYpist adapts its predictions of likely subsequent text to the user's word usage pattern, and to the characteristics of the text currently being typed. Increases in typing speed of 2-3% when typing English prose and 10-20% when typing C code have been demonstrated using the system, suggesting a potential time savings of more than 20 hours per user per year. In addition to increasing typing speed, AutoTypist reduces the number of keystrokes a user must type by a similar amount (2-3% for English, 10- 20% for computer programs). This keystroke savings has the potential to significantly reduce the frequency and severity of repeated stress injuries caused by typing, which are the most common injury suffered in today's office environment.

In 1991, Karl Sims presented work on artificial evolution in which he used genetic algorithms to evolve complex structures for use in computer-generated images and animations. The evolution of the computer-generated images progressed from simple, randomly generated shapes to interesting images which the users created interactively. The evolution advanced under the constant guidance and supervision of the user. This paper describes attempts to automate the process of image evolution through the use of artificial neural networks. The central objective of this study is to learn the user's preferences, and to apply this knowledge to evolve aesthetically pleasing images which are similar to those evolved through interactive sessions with the user. This paper presents a detailed performance analysis of both the successes and shortcomings encountered in the use of five artificial neural network architectures. Further possibilities for improving the performance of a fully automated system are also discussed.

The ALVINN (Autonomous Land Vehicle In a Neural Network) project addresses the problem of training artificial neural networks in real time to perform difficult perception tasks. ALVINN is a backpropagation network designed to drive the CMU Navlab, a modified Chevy van. This paper describes the training techniques that allow ALVINN to learn in under 5 minutes to autonomously control the Navlab by watching the reactions of a human driver. Using these techniques, ALVINN has been trained to drive in a variety of circumstances including single-lane paved and unpaved roads, and multilane lined and unlined roads, at speeds of up to 20 miles per hour.