To further advance driver monitoring and assistance systems, it is important to understand how drivers allocate their attention, in other words, where do they tend to look and why. Traditionally, factors affecting human visual attention have been divided into bottom-up (involuntary attraction to salient regions) and top-down (driven by the demands of the task being performed). Although both play a role in directing drivers’ gaze, most of the existing models for drivers’ gaze prediction apply techniques developed for bottom-up saliency and do not consider influences of the drivers’ actions explicitly. Likewise, common driving attention benchmarks lack relevant annotations for drivers’ actions and the context in which they are performed. Therefore, to enable analysis and modeling of these factors for drivers’ gaze prediction, we propose the following: 1) we correct the data processing pipeline used in DR(eye)VE to reduce noise in the recorded gaze data; 2) we then add per-frame labels for driving task and context; 3) we benchmark a number of baseline and SOTA models for saliency and driver gaze prediction and use new annotations to analyze how their performance changes in scenarios involving different tasks; and, lastly, 4) we develop a novel model that modulates drivers’ gaze prediction with explicit action and context information. While reducing noise in the DR(eye)VE gaze data improves results of all models, we show that using task information in our proposed model boosts performance even further compared to bottom-up models on the cleaned up data, both overall (by 24% KLD and 89% NSS) and on scenarios that involve performing safety-critical maneuvers and crossing intersections (by up to 10–30% KLD). Extended annotations and code are available at https://github.com/ykotseruba/SCOUT.

Driving is a visuomotor task, i.e., there is a connection between what drivers see and what they do. While some models of drivers’ gaze account for top-down effects of drivers’ actions, the majority learn only bottom-up correlations between human gaze and driving footage. The crux of the problem is lack of public data with annotations that could be used to train top-down models and evaluate how well models of any kind capture effects of task on attention. As a result, top-down models are trained and evaluated on private data and public benchmarks measure only the overall fit to human data. In this paper, we focus on data limitations by examining four large-scale public datasets, DR(eye)VE, BDD-A, MAAD, and LBW, used to train and evaluate algorithms for drivers’ gaze prediction. We define a set of driving tasks (lateral and longitudinal maneuvers) and context elements (intersections and right-of-way) known to affect drivers’ attention, augment the datasets with annotations based on the said definitions, and analyze the characteristics of data recording and processing pipelines w.r.t. capturing what the drivers see and do. In sum, the contributions of this work are: 1) quantifying biases of the public datasets, 2) examining performance of the SOTA bottom-up models on subsets of the data involving non-trivial drivers’ actions, 3) linking shortcomings of the bottom-up models to data limitations, and 4) recommendations for future data collection and processing. The new annotations and code for reproducing the results are available at https://github.com/ykotseruba/SCOUT.

Accurate prediction of drivers’ gaze is an important component of vision-based driver monitoring and assistive systems. Of particular interest are safety-critical episodes, such as performing maneuvers or crossing intersections. In such scenarios, drivers’ gaze distribution changes significantly and becomes difficult to predict, especially if the task and context information is represented implicitly, as is common in many state-of-the-art models. However, explicit modeling of top-down factors affecting drivers’ attention often requires additional information and annotations that may not be readily available.In this paper, we address the challenge of effective modeling of task and context with common sources of data for use in practical systems. To this end, we introduce SCOUT+, a task-and context-aware model for drivers’ gaze prediction, which leverages route and map information inferred from commonly available GPS data. We evaluate our model on two datasets, DR(eye)VE and BDD-A, and demonstrate that using maps improves results compared to bottom-up models and reaches performance comparable to the top-down model SCOUT which relies on privileged ground truth information. Code is available at https://github.com/ykotseruba/SCOUT.

Driving safety has been a concern since the first cars appeared on the streets. Driver inattention has been singled out as a major cause of accidents early on. This is hardly surprising, as drivers routinely perform other tasks in addition to controlling the vehicle. Decades of research into what causes lapses or misdirection of drivers’ attention resulted in improvements in road safety through better design of infrastructure, driver training programs, in- vehicle interfaces, and, more recently, the development of driving assistance systems (ADAS) and driving automation. This review focuses on the methods for modeling and detecting spatio-temporal aspects of drivers’ attention, i. e. where and when they look, for the two latter categories of applications. We start with a brief theoretical background on human visual attention, methods for recording and measuring attention in the driving context, types of driver inattention, and factors causing it. We then discuss machine learning approaches for 1) modeling gaze for assistive and self-driving applications and 2) detecting gaze for driver monitoring. Following the overview of state-of-the-art models, we provide an extensive list of publicly available datasets that feature recordings of drivers’ gaze and other attention-related annotations. We conclude with a general overview of the remaining challenges, such as data availability and quality, evaluation methods, and the limited scope of attention modeling, and outline steps toward rectifying some of these issues. Categorized and annotated lists of the reviewed models and datasets are available athttps://github.com/ykotseruba/attention_and_driving

Pedestrian action prediction has been a topic of active research in recent years resulting in many new algorithmic solutions. However, measuring the overall progress towards solving this problem is difficult due to the lack of publicly available benchmarks and common training and evaluation procedures. To this end, we introduce a benchmark based on two public datasets for pedestrian behavior understanding. Using the proposed evaluation procedures, we rank a number of baseline and state-of-the-art models and analyze their performance with respect to various properties of the data. Based on these findings we propose a new model for pedestrian crossing action prediction that uses attention mechanisms to effectively combine implicit and explicit features and demonstrate new state-of-the-art results. The code for models and evaluation is available at https://github.com/ykotseruba/PedestrianActionBenchmark.

Driving in urban traffic requires making quick and safe decisions while interacting with multiple pedestrians and other road users. Early anticipation of others' intentions is especially important for predicting their future behavior. In this work, we explore the human ability to estimate intentions of pedestrians in typical urban traffic conditions. Towards this goal, we analyze the results of our large-scale experiment that involved over 700 subjects to establish a human reference point for the task of pedestrian intention estimation. We determine what visual features correlate with human decisions and the relative difficulty of scenarios and validate our conclusions using a linear logistic model. Furthermore, we propose two models to demonstrate the benefits of using intention for pedestrian trajectory and future crossing action prediction. Our experiments show that an improvement of up to 5 % can be achieved on both tasks.

Given an existing trained neural network, it is often desirable to learn new capabilities without hindering performance of those already learned. Existing approaches either learn sub-optimal solutions, require joint training, or incur a substantial increment in the number of parameters for each added domain, typically as many as the original network. We propose a method called Deep Adaptation Modules (DAM) that constrains newly learned filters to be linear combinations of existing ones. DAMs precisely preserve performance on the original domain, require a fraction (typically 13 percent, dependent on network architecture) of the number of parameters compared to standard fine-tuning procedures and converge in less cycles of training to a comparable or better level of performance. When coupled with standard network quantization techniques, we further reduce the parameter cost to around 3 percent of the original with negligible or no loss in accuracy. The learned architecture can be controlled to switch between various learned representations, enabling a single network to solve a task from multiple different domains. We conduct extensive experiments showing the effectiveness of our method on a range of image classification tasks and explore different aspects of its behavior.

One of the major challenges that autonomous cars are facing today is driving in urban environments. To make it a reality, autonomous vehicles require the ability to communicate with other road users and understand their intentions. Such interactions are essential between vehicles and pedestrians, the most vulnerable road users. Understanding pedestrian behavior, however, is not intuitive and depends on various factors, such as demographics of the pedestrians, traffic dynamics, environmental conditions, and so on. In this paper, we identify these factors by surveying pedestrian behavior studies, both the classical works on pedestrian-driver interaction and the modern ones that involve autonomous vehicles. To this end, we will discuss various methods of studying pedestrian behavior and analyze how the factors identified in the literature are interrelated. We will also review the practical applications aimed at solving the interaction problem, including design approaches for autonomous vehicles that communicate with pedestrians and visual perception and reasoning algorithms tailored to understanding pedestrian intention. Based on our findings, we will discuss the open problems and propose future research directions.

Pedestrian behavior anticipation is a key challenge in the design of assistive and autonomous driving systems suitable for urban environments. An intelligent system should be able to understand the intentions or underlying motives of pedestrians and to predict their forthcoming actions. To date, only a few public datasets were proposed for the purpose of studying pedestrian behavior prediction in the context of intelligent driving. To this end, we propose a novel large-scale dataset designed for pedestrian intention estimation (PIE). We conducted a large-scale human experiment to establish human reference data for pedestrian intention in traffic scenes. We propose models for estimating pedestrian crossing intention and predicting their future trajectory. Our intention estimation model achieves 79% accuracy and our trajectory prediction algorithm outperforms state-of-the-art by 26% on the proposed dataset. We further show that combining pedestrian intention with observed motion improves trajectory prediction. The dataset and models are available at http://data.nvision2.eecs.yorku.ca/PIE_dataset/.

Visual attention is a field with a considerable history, with eye movement control and prediction forming an important subfield. Fixation modeling in the past decades has been largely dominated computationally by a number of highly influential bottom-up saliency models, such as the Itti-Koch-Niebur model. The accuracy of such models has dramatically increased recently due to deep learning. However, on static images the emphasis of these models has largely been based on non-ordered prediction of fixations through a saliency map. Very few implemented models can generate temporally ordered human-like sequences of saccades beyond an initial fixation point. Towards addressing these shortcomings we present STAR-FC, a novel multi-saccade generator based on the integration of central high-level and object-based saliency and peripheral lower-level feature-based saliency. We have evaluated our model using the CAT2000 database, successfully predicting human patterns of fixation with equivalent accuracy and quality compared to what can be achieved by using one human sequence to predict another.

The contribution of this paper is twofold. The first is a novel dataset for studying behaviors of traffic participants while crossing. Our dataset contains more than 650 samples of pedestrian behaviors in various street configurations and weather conditions. These examples were selected from approx. 240 hours of driving in the city, suburban and rural roads. The second contribution is an analysis of our data from the point of view of joint attention. We identify what types of non-verbal communication cues road users use at the point of crossing, their responses, and under what circumstances the crossing event takes place. It was found that in more than 90% of the cases pedestrians gaze at the approaching cars prior to crossing in non-signalized crosswalks. The crossing action, however, depends on additional factors such as time to collision, explicit driver's reaction or structure of the crosswalk.

A recent trend in saliency algorithm development is large-scale benchmarking and algorithm ranking with ground truth provided by datasets of human fixations. In order to accommodate the strong bias humans have toward central fixations, it is common to replace traditional ROC metrics with a shuffled ROC metric which uses randomly sampled fixations from other images in the database as the negative set. However, the shuffled ROC introduces a number of problematic elements, including a fundamental assumption that it is possible to separate visual salience and image spatial arrangement. We argue that it is more informative to directly measure the effect of spatial bias on algorithm performance rather than try to correct for it. To capture and quantify these known sources of bias, we propose a novel metric for measuring saliency algorithm performance: the spatially binned ROC (spROC). This metric provides direct in-sight into the spatial biases of a saliency algorithm without sacrificing the intuitive raw performance evaluation of traditional ROC measurements. By quantitatively measuring the bias in saliency algorithms, researchers will be better equipped to select and optimize the most appropriate algorithm for a given task. We use a baseline measure of inherent algorithm bias to show that Adaptive Whitening Saliency (AWS) [14], Attention by Information Maximization (AIM) [8], and Dynamic Visual Attention (DVA) [20] provide the least spatially biased results, suiting them for tasks in which there is no information about the underlying spatial bias of the stimuli, whereas algorithms such as Graph Based Visual Saliency (GBVS) [18] and Context-Aware Saliency (CAS) [15] have a significant inherent central bias.

In graph-based semi-supervised learning approaches, the classification rate is highly dependent on the size of the availabel labeled data, as well as the accuracy of the similarity measures. Here, we propose a semi-supervised multi-class/multi-label classification scheme, dynamic label propagation (DLP), which performs transductive learning through propagation in a dynamic process. Existing semi-supervised classification methods often have difficulty in dealing with multi-class/multi-label problems due to the lack in consideration of label correlation, our algorithm instead emphasizes dynamic metric fusion with label information. Significant improvement over the state-of-the-art methods is observed on benchmark datasets for both multi-class and multi-label tasks.

Most current algorithm evaluation protocols use large image databases, but give little consideration to imaging characteristics used to create the data sets. This paper evaluates the effects of camera shutter speed and voltage gain under simultaneous changes in illumination and demonstrates significant differences in the sensitivities of popular vision algorithms under variable illumination, shutter speed, and gain. These results show that offline data sets used to evaluate vision algorithms typically suffer from a significant sensor specific bias which can make many of the experimental methodologies used to evaluate vision algorithms unable to provide results that generalize in less controlled environments. We show that for typical indoor scenes, the different saturation levels of the color filters are easily reached, leading to the occurrence of localized saturation which is not exclusively based on the scene radiance but on the spectral density of individual colors present in the scene. Even under constant illumination, foreshortening effects due to surface orientation can affect feature detection and saliency. Finally, we demonstrate that active and purposive control of the shutter speed and gain can lead to significantly more reliable feature detection under varying illumination and nonconstant viewpoints.

Computational models of visual processes with biological inspiration - and even biological realism - are currently of great interest in the computer vision community. This paper provides a biologically plausible model of 2D shape which incorporates intermediate layers of visual representation that have not previously been fully explored. We propose that endstopping and curvature cells are of great importance for shape selectivity and show how their combination can lead to shape selective neurons. This shape representation model provides a highly accurate fit with neural data from and provides comparable results with real-world images to current computer vision systems. The conclusion is that such intermediate representations may no longer require a learning approach as a bridge between early representations based on Gabor or Difference of Gaussian filters (that are not learned since they are well-understood) and later representations closer to object representations that still can benefit from a learning methodology.

We study the problem of actively searching for an object in a three-dimensional (3-D) environment under the constraint of a maximum search time using a visually guided humanoid robot with 26 degrees of freedom. The inherent intractability of the problem is discussed, and a greedy strategy for selecting the best next viewpoint is employed. We describe a target probability updating scheme approximating the optimal solution to the problem, providing an efficient solution to the selection of the best next viewpoint. We employ a hierarchical recognition architecture, inspired by human vision, that uses contextual cues for attending to the view-tuned units at the proper intrinsic scales and for active control of the robotic platform sensor's coordinate frame, which also gives us control of the extrinsic image scale and achieves the proper sequence of pathognomonic views of the scene. The recognition model makes no particular assumptions on shape properties like texture and is trained by showing the object by hand to the robot. Our results demonstrate the feasibility of using state-of-the-art vision-based systems for efficient and reliable object localization in an indoor 3-D environment.

This paper presents a novel framework for semantic place labeling by formulating the problem in terms of energy minimization. A method based on graph cuts is used to minimize energy for a function of data cost and smoothness cost. While the data term aims at assigning visual observations to a set of pre-specified place categories, using appearance-based hierarchical classifiers, the smoothness term incorporates contextual evidence from neighbors to ensure that the labels vary smoothly almost everywhere while preserving discontinuities at the borders between adjacent places in the environment. Our proposed method achieved a performance of 91.85%, labeling 2,146 images from the challenging COLD database with place semantics. Correct labeling of 14.5% of images was the result of incorporating contextual information.

In this paper a novel technique for face recognition is proposed. Using the statistical local feature analysis (LFA) technique, a set of feature points is extracted from each face image, at locations with highest deviations from the statistical expected face. Each feature point is described by a set of Gabor wavelet responses at different frequencies and orientations. A triangle-inequality-based pruning algorithm is developed for fast matching, which automatically chooses a set of key features from the database of model features and uses the pre-computed distances of the keys to the database, along with the triangle inequality, in order to speedily compute lower bounds on the distances from a query feature to the database, and eliminate the unnecessary direct comparisons. Our proposed technique achieves perfect results on the ORL face set and an accuracy rate of 99.1% on the FERET face set, which shows the superiority of the proposed technique over all considered state-of-the-art face recognition methods.

We present some theoretical results related to the problem of actively searching for a target in a 3D environment, under the constraint of a maximum search time. We define the object localization problem as the maximization over the search region of the Lebesgue integral of the scene structure probabilities. We study variants of the problem as they relate to actively selecting a finite set of optimal viewpoints of the scene for detecting and localizing an object. We do a complexity-level analysis and show that the problem variants are NP-Complete or NP-Hard. We study the tradeoffs of localizing vs. detecting a target object, using single-view and multiple-view recognition, under imperfect dead-reckoning and an imperfect recognition algorithm. These results motivate a set of properties that efficient and reliable active object localization algorithms should satisfy.

This paper presents a novel appearance-based technique for qualitative spatial localization. A vocabulary of visual words is built automatically, representing local features that repeatedly occur in the set of training images. An information maximization technique is then applied to build a hierarchical classifier for each environment by learning informative visual words. Child nodes in this hierarchy encode information redundant with information coded by their parents. In localization, hierarchical classifiers are used in a top-down manner, where top-level visual words are examined first, and for each top-level visual word which does not respond as expected, its lower-level visual words are examined. This allows inference to recover from missing features encoded by higher-level visual words. Several experiments on a challenging localization database demonstrate the advantages of our hierarchical framework and show a significant improvement over the traditional bag-of-features approaches.

We present a biologically plausible computational model for solving the visual feature binding problem, based on recent results regarding the time course and processing sequence in the primate visual system. The feature binding problem appears due to the distributed nature of visual processing in the primate brain, and the gradual loss of spatial information along the processing hierarchy. This paper puts forward the proposal that by using multiple passes of the visual processing hierarchy, both bottom-up and top-down, and using task information to tune the processing prior to each pass, we can explain the different recognition behaviors that primate vision exhibits. To accomplish this, four different kinds of binding processes are introduced and are tied directly to specific recognition tasks and their time course. The model relies on the reentrant connections so ubiquitous in the primate brain to recover spatial information, and thus allow features represented in different parts of the brain to be integrated in a unitary conscious percept. We show how different tasks and stimuli have different binding requirements, and present a unified framework within the Selective Tuning model of visual attention.

A representation for observing local image content is proposed for the purpose of considering the distinguishing characteristics of visual content that tends to draw a human observers gaze. Within this representation, the spectral profile distinguishing fixated from non-fixated locations is considered. Finally, the possibility of designing saliency operators based on the proposed local magnitude spectrum representation is explored, revealing a promising domain for predicting human gaze patterns.

Motion detection and estimation is a first step in the much larger framework of attending to visual motion based on Selective Tuning Model of Visual Attention (Tsotsos, et al., 2002). In order to be able to detect and estimate complex motion in a hierarchical system it is necessary to use robust and efficient methods which encapsulate as much information as possible about the motion together with a measure of reliability of that information. One such method is the orientation tensor formalism which incorporates a confidence measure that propagates into subsequent processing steps. The tensor method is implemented in a neural network simulator which allows distributed processing and visualization of results. As output we obtain information about the moving objects from the scene

Selective Tuning (ST) presents a framework for modeling attention and in this work we show how it performs in covert visual search tasks by comparing its performance to human performance. Two implementations of ST have been developed. The Object Recognition Model recognizes and attends to simple objects formed by the conjunction of various features and the Motion Model recognizes and attends to motion patterns. The validity of the Object Recognition Model was first tested by successfully duplicating the results of Nagy and Sanchez. A second experiment was aimed at an evaluation of the model's performance against the observed continuum of search slopes for feature-conjunction searches of varying difficulty. The Motion Model was tested against two experiments dealing with searches in the visual motion domain. A simple odd-man-out search for counter-clockwise rotating octagons among identical clockwise rotating octagons produced linear increase in search time with the increase of set size. The second experiment was similar to one described by Thorton and Gilden. The results from both implementations agreed with the psychophysical data from the simulated experiments. We conclude that ST provides a valid explanatory mechanism for human covert visual search performance, an explanation going far beyond the conventional saliency map based explanations.

Ensemble methods such as bootstrap, bagging or boosting have had a considerable impact on recent developments in machine learning, pattern recognition and computer vision. Theoretical and practical results alike have established that, in terms of accuracy, ensembles of weak classifiers generally outperform monolithic solutions. However, this comes at the cost of an extensive training process. The work presented in this paper results from projects on advanced human machine interaction. In scenarios like ours, online learning is a major requirement, and lengthy training is prohibitive. We therefore propose a different approach to ensemble learning. Instead of a set of weak classifiers, we combine strong, separable, multilinear discriminant functions. These are especially suited for computer vision: they train very quickly and allow for rapid classification of image content. Training different classifiers for different contexts or on semantically organized data provides ensembles of experts. We collapse a set of experts into a single multilinear function and thus achieve the same runtime for arbitrarily many classifiers as for a single one. Moreover, carrying out the classification in the frequency domain results in faster framerates. Experiments with image sequences recorded in typical home environments show that our ensemble training schemes yield high accuracy on unconstrained and cluttered data.

This paper presents a wavelet-based framework for enhancing the coherent structures attributable to the target organ in cardiac magnetic resonance (MR) images. Previous approaches focus on the Rician nature of noise in magnitude MR images. Image noise is but only one of the confounding factors that obscure the anatomical structures of the target organ. This paper models the image noise in a magnitude MR image in terms of two noise classes which occur over different ranges of signal intensity. An adaptive enhancement scheme is developed to achieve simultaneous attenuation of the effects of these factors and improvement in image contrast

This paper presents a fast method to compute homeomorphisms between 2D lattices and shapes found in binary images. Unlike many other methods, this mapping is not restricted to simply connected shapes but applies to arbitrary topologies. Moreover, it provides an avenue to the embedding of shapes in vectorspaces over R and C and thus enables robust shape recognition.

A model of bottom-up overt attention is proposed based on the principle of maximizing information sampled from a scene. The proposed opera(cid:173) tion is based on Shannon's self-information measure and is achieved in a neural circuit, which is demonstrated as having close ties with the cir(cid:173) cuitry existent in the primate visual cortex. It is further shown that the proposed saliency measure may be extended to address issues that cur(cid:173) rently elude explanation in the domain of saliency based models. Results on natural images are compared with experimental eye tracking data re(cid:173) vealing the efficacy of the model in predicting the deployment of overt attention as compared with existing efforts.

We present a robust and portable visual-based skin and face detection system developed for use in a multiple speaker teleconferencing system, employing both audio and video cues. An omni-directional video sensor is used to provide a view of the entire visual hemisphere, thereby allowing for multiple dynamic views of all the participants. Regions of skin are detected using simple statistical methods, along with histogram color models for both skin and non-skin color classes. Regions of skin belonging to the same person are grouped together, and using simple spatial properties, the position of each person's face is inferred. Preliminary results suggest the system is capable of detecting human faces present in an omni-directional image despite the poor resolution inherent with such an omni-directional sensor.

The problem of detecting areas of motion in video sequences and estimating parameters such as speed, direction and dynamics is addressed in many applications of image processing such as video surveillance, object tracking, image stream compression or autonomous navigation systems. Real world computer vision highly depends on reliable, robust systems for recognition of motion cues to make accurate high-level decisions about its surroundings. In this paper, we present a simple, yet high performance low-level filter for motion detection and estimation in digitized video signals. The algorithm is based on constant characteristics of a common, 2-frame interlaced video signal, yet its applicability to generically acquired image sequences is shown as well. In general, our approach presents a computationally low-cost solution to motion estimation application and compares very well to existing approaches due to its robustness towards environmental changes. A simple application of motion parameter estimation based on a pedestrian surveillance application is illustrated.

Object search is the task of searching for a given 3D object in a given 3D environment by a controllable camera. Sensor planning for object search refers to the task of how to select the sensing parameters of the camera so as to bring the target into the field of view of the camera and to make the image of the target to be easily recognized by the available recognition algorithms. In this paper, we study the task of sensor planning for object search from the theoretical point of view. We formulate the task and point out many of its important properties. We then analyze this task from the complexity level and prove that this task is NP‐Complete.

In this paper we introduce the concept of knowledge granularity and study the relationship between different knowledge representation schemes and the scaling problem. By scale to a task, we mean that an agent's planning system and knowledge representation scheme are able to generate the range of behaviors required by the task in a timely fashion. Action selection is critical to an agent performing a task in a dynamic, unpredictable environment. Knowledge representation is central to the agent's action selection process. It is important to study how an agent should adapt its methods of representation such that its performance can scale to different task requirements. Here we study the following issues. One is the knowledge granularity problem: to what detail should an agent represent a certain kind of knowledge if a single granularity of representation is to be used. Another is the representation scheme problem: to scale to a given task, should an agent represent its knowledge using a single granularity or a set of hierarchical granularities.

We study how agents can facilitate and mediate interaction, communication, and cooperation among people. We propose the concepts of a smart distance and an awareness network in a distributed collaborative environment. We illustrate the architecture of an agent-mediated collaborative system - the agent-buddy system that can create a sense of group presence and, at the same time, preserve the privacy of each user. Virtual springs systems are used to model the awareness degrees among team members. Each agent makes decisions by considering multiple factors. The goal of the multi-agent team is to minimize the global awareness frustrations with respect to different kinds of tasks. Empirical studies were conducted to analyze the influence of individual behavior on global performance for various kinds of tasks.

A new technique for fast pattern recognition using normalized grey-scale correlation (NGC) is described. While NGC has traditionally been slow due to computational intensity issues, the introduction of both a pyramid structure and a local estimate of the correlation surface gradient allows for recognition in 10-50 ms using modest microcomputer hardware. The algorithm is designed to analyze the target off-line prior to starting the search. Issues surrounding determining an appropriate depth for the pyramid representation and performing sub-pixel localization of the target instance are discussed. The speed and robustness of the method makes it attractive for industrial applications.

We quantify the observation by Kender and Freudenstein (1987) that degenerate views occupy a significant fraction of the viewing sphere surrounding an object. For a perspective camera geometry, we introduce a computational model that can be used to estimate the probability that a view degeneracy will occur in a random view of a polyhedral object. For a typical recognition system parameterization, view degeneracies typically occur with probabilities of 20 percent and, depending on the parameterization, as high as 50 percent. We discuss the impact of view degeneracy on the problem of object recognition and, for a particular recognition framework, relate the cost of object disambiguation to the probability of view degeneracy. To reduce this cost, we incorporate our model of view degeneracy in an active focal length control paradigm that balances the probability of view degeneracy with the camera field of view. In order to validate both our view degeneracy model as well as our active focal length control model, a set of experiments are reported using a real recognition system operating on real images.

We quantify the observation by Kender and Freudenstein (1987) that degenerate views occupy a significant fraction of the viewing sphere surrounding an object. This demonstrates that systems for recognition must explicitly account for the possibility of view degeneracy. We show that view degeneracy cannot be detected from a single camera viewpoint. As a result, systems designed to recognize objects from a single arbitrary viewpoint must be able to function in spite of possible undetected degeneracies, or else operate with imaging parameters that cause acceptably low probabilities of degeneracy. To address this need, we give a prescription for active control of focal length that allows a principled tradeoff between the camera field of view and probability of view degeneracy.>

Perspective alignment is a novel method of solving backprojection, the well-known problem of computing three dimensional (3D) position and orientation (pose) of a model from two-dimensional (2D) image features. This paper demonstrates that previous backprojection methods can violate the visibility constraint by computing solution poses in which the model occludes features which should be visible. By definition, these visibility errors are associated with incorrect pose solutions. Yet they occur frequently when previous backprojection methods are used in underconstrained situations. We empirically analyze the frequency and consequences of visibility errors in previous backprojection methods. We then show how perspective alignment satisfies the visibility constraint during the pose solution process to eliminate these errors. The algorithm has been implemented and used in a real-time model-based object tracking system. We describe the algorithm and results of tracking real objects in real-time. The algorithm also has implications for reducing the combinatorics of image-model feature pairing in model-based recognition.

When a particular point is fixated by an active stereo system different portions of the world are brought into interocular alignment. This region is known as the horoptor. Through an examination of the horoptor under different viewing conditions it is demonstrated that for certain binocular tasks it is desirable to manipulate the horoptor by rotating (torquing) the cameras about their optical axes. This manipulation can be passive for operations such as stereo based obstacle detection for mobile robots, or active for active binocular heads. Techniques for both situations are presented.>

When a particular 3D point is fixated by a robotic stereo system different portions of the world are brought into interocular alignment. This region is known as the horoptor. Purposeful modifications to the binocular geometry can be used to bring different regions of three-space closer to the horoptor: camera pan and tilt define the rough structure of the horoptor, while camera torsion can be used to change its local shape. Theoretical and empirical results suggest that for binocular vision tasks: 1) it is important to understand the region of three space that contains the horoptor curve; and 2) it is possible to control this shape in an active way so as to simplify certain binocular tasks.

This paper proposes a model that investigates a new avenue for attention control based on dynamic scenes. We have derived a computational model to detect abrupt changes and have examined how the most prominent change can be determined. With such a model, we explore the possibility of an attentional mechanism, in part guided by abrupt changes, for gaze control. The computational model is derived from the difference of Gaussian (DOG) model and it examines the change in the response of the DOG operator over time to determine if changes have occurred. On and off-DOG operators are used to detect "on" and "off" events respectively. The response of these operators is examined over various temporal window sizes so that changes at different rates can be found. The most salient "on" and "off" events are determined from the corresponding winner-take-all (WTA) network. The model has been tested with image sequences which have changes caused by brightness or motion and the results are satisfactory.

This paper describes research on the ARK (Autonomous Mobile Robot in a Known Environment) project. The technical objective of the project is to build a robot that can navigate and carry out survey/inspection tasks in a complex but known industrial environment. Rather than altering the robots environment by adding easily identifiable beacons the robot relies on naturally occurring objects to use as visual landmarks for navigation. The robot is equipped with various sensors that are used to detect unmapped obstacles, landmarks and objects. This paper describes the robot's industrial environment, it's control architecture, and some results in processing the robot's range and vision sensor data for navigation.>

Probabilistic algorithms are presented for efficient storage and retrieval of sets of feature vectors, given a known error process operating on the query set, that perturbs the query set away from the corresponding stored set. The algorithms operate by mapping each set to a corresponding generalized indicator vector and then performing a pruned search of a tree containing stored indicator vectors. The pruning is based on the probability of the query, given the stored items below the current position in the tree. Analysis and trial results show that this approach requires less total computation than existing methods based on parallel architectures. The indicator vector retrieval method can also cope efficiently with query vectors of much higher dimensionality than existing serial algorithms for nearest-neighbor searches.>

We present the design of a controllable stereo vision head. TRISH (The Toronto IRIS Stereo Head) is a binocular camera mount, consisting of two fixed focal length color cameras with automatic gain control forming a verging stereo pair. TRISH is capable of version (rotation of the eyes about the vertical axis so as to maintain a constant disparity), vergence (rotation of each eye about the vertical axis so as to change the disparity), pan (rotation of the entire head about the vertical axis), and tilt (rotation of each eye about the horizontal axis). One novel characteristic of the design is that each camera can rotate about its own optical axis (torsion). Torsional movement makes it possible to minimize the vertical component of the two-dimensional search which is associated with stereo processing in verging stereo systems.

The concept of active object recognition is introduced, and a proposal for its solution is described. The camera is mounted on the end of a robot arm on a mobile base. The system exploits the mobility of the camera by using low-level image data to drive the camera to a standard viewpoint with respect to an unknown object. From such a viewpoint, the object recognition task is reduced to a two-dimensional pattern recognition problem. The system uses an efficient tree-based, probabilistic indexing scheme to find the model object that is likely to have generated the observed data, and for line tracking uses a modification of the token-based tracking scheme of J.L. Crowley et al. (1988). The system has been successfully tested on a set of origami objects. Given sufficiently accurate low-level data, recognition time is expected to grow only logarithmically with the number of objects stored.>

Mounting evidence suggests that attentional mechanisms may be required to successfully perform many vision tasks. The paper presents an attentional prototype for early visual processing. The model is composed of a processing hierarchy and an attention beam that traverses the hierarchy, passing through the regions of greatest interest and inhibiting the regions that are not relevant. The type of input to the prototype is not limited to visual stimuli. Aspects of attention such as localizing spatial regions of interest and ordering their importance are addressed; other aspects of attention such as the role of task guidance are encompassed by the model but are not detailed here. Simulations using high-resolution digitized images were conducted, with oriented edge information as the input to the model. The results confirm that this prototype is both robust and fast, and promises to be essential to any real-time vision system.>

An approach for describing objects for the purpose of recognition is developed. The authors deal with two key issues: building natural descriptions of curved objects, and making these descriptions compact and abstract. Typical examples of the types of curve one is able to describe as both qualitatively similar, yet discriminably different, are shown. Methods based on curvature extrema alone are likely to find three of the four of these shapes indistinguishable, while methods based on approaches such as shape templates may be oblivious to their similarity. The central ideas of the approach are outlined.>

A novel approach to computing egomotion and detecting points not moving rigidly with the scene when an observer moves with unrestricted motion is presented. The approach, using collinear image points, is based on an exact method for cancelling effects of the observer's rotation from optic flow. For each point only the component of velocity normal to the direction of the line joining the collinear points is needed. The algorithm is simple, appears robust, and is ideal for parallel implementation.>

The authors present a technique for both the smoothing and decomposition of planar curves. This technique, called curvature-tuned smoothing, provides for robust rotation- and translation-invariant smoothing of planar curves. As the smoothing is performed, parts are extracted robustly and at multiple scales. These parts can subsequently be used for object recognition. The parts extracted correspond to regions of roughly uniform curvature and constitute a rich description of the original data. This representation describes some regions at multiple scales, since multiple structures may occur.>

We address the problem of interpreting image velocity fields generated by a moving monocular observer viewing a stationary environment under perspective projection to obtain 3-D information about the relative motion of the observer (egomotion) and the relative depth of environmental surface points (environmental layout). The algorithm presented in this paper involves computing motion and structure from a spatio-temporal distribution of image velocities that are hypothesized to belong to the same 3-D planar surface. However, the main result of this paper is not just another motion and structure algorithm that exhibits some novel features but rather an extensive error analysis of the algorithm’s preformance for various types of noise in the image velocities. Waxman and Ullman [83] have devised an algorithm for computing motion and structure using image velocity and its 1st and 2d order spatial derivatives at one image point. We generalize this result to include derivative information in time as well. Further, we show the approximate equivalence of reconstruction algorithms that use only image velocities and those that use one image velocity and its 1st and/or 2”d spatio-temporal derivatives at one image point. The main question addressed in this paper is: “How accurate do the input image velocities have to be?’ or equivalently, “How accurate does the input image velocity and its I~ and 2& order derivatives have to be?“. The answer to this question involves worst case error analysis. We end the paper by drawing some conclusions about the feasibility of motion and structure calculations in general. I.1 Introduction In this paper, we present a algorithm for computing the motion and strncture parameters that describe egomotion and environmental layout from image velocity fields generated by a moving monocular observer viewing a stationary environment. Egomotion is defined as the motion of the observer relative to his environment and can be described by 6 parameters; 3 dvth-scaled translational parameters, Z and 3 rotation parameters, o. Environmental layout refers to the 3-D shape and location of objects in the environment. For monocular image sequences, en$ronmental layout is described by the normalized surface gradient, a, at each image point. To determine these motion and structure parameters we derive nonlinear equations relating image velocity at some image int ?(?*,t ‘) to the underlying motion and structure parameters at (P,c). The computaP tion of egomotion and environmental layout from image velocity is sometimes called the reconstruction problem; we reconstruct the observer’s motion, and the layout of his environment, from (timevarying) image velocity. A lot of research has been devoted to

The so‐called “first generation” expert systems were rule‐based and offered a successful framework for building applications systems for certain kinds of tasks. Spatial, temporal, and causal reasoning, knowledge abstractions, and structuring are among topics of research for “second generation” expert systems. It is proposed that one of the keys for such research is knowledge organization. Knowledge organization determines control structure design, explanation and evaluation capabilities for the resultant knowledge base, and has strong influence on system performance. We are exploring a framework for expert system design that focuses on knowledge organization, for a specific class of input data, namely, continuous, time‐varying data (image sequences or other signal forms). Such data are rich in temporal relationships as well as temporal changes of spatial relations, and are thus a very appropriate testbed for studies involving spatio‐temporal reasoning. In particular, the representation formalism specifies the semantics of the organization of knowledge classes along the relationships of generalization/specialization, decomposition/aggregation, temporal precedence, instantiation, and expectation‐activated similarity. Á hypothesize‐and‐test control structure is driven by the class organizational principles, and includes several interacting dimensions of search (data‐driven, model‐driven, goal‐driven temporal, and failure‐driven search). The hypothesis ranking scheme is based on temporal cooperative computation, with hypothesis “fields of influence” being defined by the hypothesis’ organizational relationships. This control structure has proven to be robust enough to handle a variety of interpretation tasks for continuous temporal data. A particular incarnation, the ALVEN system, for left ventricular performance assessment from X‐ray image sequences, will be summarized in this paper.