Computer Science (CS 1) offerings in most universities tend to be notoriously difficult. Over the past 60 years about a third of the students either fail or drop out of the course. Past research has focused on improving teaching methods through small changes without changing the overall course structure.

Accurately estimating the position of static objects, such as traffic lights, from the moving camera of a self-driving car is a challenging problem. In this work, we present a system that improves the localization of static objects by jointly-optimizing the components of the system via learning. Our system is comprised of networks that perform: 1) 5DoF object pose estimation from a single image, 2) association of objects between pairs of frames, and 3) multi-object tracking to produce the final geo-localization of the static objects within the scene. We evaluate our approach using a publicly-available data set, focusing on traffic lights due to data availability. For each component, we compare against contemporary alternatives and show significantly-improved performance. We also show that the end-to-end system performance is further improved via joint-training of the constituent models. Code is available at: https://github.com/MedChaabane/Static_Objects_Geolocalization.

In this paper, we present a novel, end-to-end 6D object pose estimation method that operates on RGB inputs. Our approach is composed of 2 main components: the first component classifies the objects in the input image and proposes an initial 6D pose estimate through a multi-task, CNN-based encoder/multi-decoder module. The second component, a refinement module, includes a renderer and a multi-attentional pose refinement network, which iteratively refines the estimated poses by utilizing both appearance features and flow vectors. Our refiner takes advantage of the hybrid representation of the initial pose estimates to predict the relative errors with respect to the target poses. It is further augmented by a spatial multi-attention block that emphasizes objects' discriminative feature parts. Experiments on three benchmarks for 6D pose estimation show that our proposed pipeline outperforms state-of-the-art RGB-based methods with competitive runtime performance.

State of the art unimodal dialogue agents lack some core aspects of peer-to-peer communication—the nonverbal and visual cues that are a fundamental aspect of human interaction. To facilitate true peer-to-peer communication with a computer, we present Diana, a situated multimodal agent who exists in a mixed-reality environment with a human interlocutor, is situation- and context-aware, and responds to the human's language, gesture, and affect to complete collaborative tasks.

In 2018, Colorado State University redesigned their CS-0 course to become a general education requirement for the university within Arts and Humanities, and a guaranteed transfer course across the state for a similar category in other universities. The first CS course in the State to be accepted as a GT-Pathway course. This redesign had to be carefully done due to a need to introduce liberal arts style topics such as CS History, Philosophy and Ethics and Inclusive Design issues, while maintaining current coding and student success standards that were already expected for the CS-0 at the university. We termed this combination as Culture and Coding. In order to add more without reducing retention, the course was redesigned around the Psychology of Learning and spacing of topics in a "Spiral" Manner. Each topic was briefly introduced, and throughout the semester, students would dive deeper into the topics. This allowed for a 50% reduction of time focused on teaching coding topics, with students performing equivalent on exams compared to previous models of the course that focused 100% of the time on coding topics. Furthermore, students taught by the spiral teaching method outperformed students taught using traditional methods in the follow-on course. Our evaluation suggests that the spiral model of teaching computer science may allow for greater retention of topics, allowing classes to either cover additional concepts or go more in depth on current topics.

In this paper, we present an end-to-end future-prediction model that focuses on pedestrian safety. Specifically, our model uses previous video frames, recorded from the perspective of the vehicle, to predict if a pedestrian will cross in front of the vehicle. The long term goal of this work is to design a fully autonomous system that acts and reacts as a defensive human driver would - predicting future events and reacting to mitigate risk. We focus on pedestrian-vehicle interactions because of the high risk of harm to the pedestrian if their actions are miss-predicted. Our end-to-end model consists of two stages: the first stage is an encoder/decoder network that learns to predict future video frames. The second stage is a deep spatiotemporal network that utilizes the predicted frames of the first stage to predict the pedestrian's future action. Our system achieves state-of-the-art accuracy on the Joint Attention for Autonomous Driving (JAAD) dataset on both future frames prediction, with a pixel-wise prediction l1error of 1.12, and pedestrian behavior prediction with an average precision of 86.7.

Multi-channel architectures are becoming increasingly common, setting the state-of-the-art for performance in gesture recognition challenges. Unfortunately, we lack a clear explanation of why multi-channel architectures outperform single channel ones. This paper considers two hypotheses. The Bagging hypothesis says that multi-channel architectures succeed because they average the result of multiple unbiased weak estimators in the form of different channels. The Society of Experts (SoE) hypothesis suggests that multi-channel architectures succeed because the channels differentiate themselves, developing expertise with regard to different aspects of the data.(/p)(/p)To distinguish between these hypotheses, this paper reports on two experiments. The first measures the drop in individual channel performance when the input is degraded by removing high frequency, color, or motion information. The second looks at fusion weights relative to gesture properties. Both experiments support the SoE hypothesis, suggesting multi-channel architectures succeed because of channel specialization.

Gesture recognition is an important task with the potential to revolutionize human/computer interfaces (HCI). Gestures, however, are dynamic. While a few gestures may be static poses, most gestures are complex temporal sequences of motions. For most HCI applications, gestures must be recognized in real-time in streaming data. Therefore, most recognition systems analyze each frame as it comes in, fusing data across time to detect gestures. This paper presents results of the first systematic study of temporal fusion techniques for streaming gesture recognition. These results show that the choice of the best fusion strategy depends on whether the input is global (i.e. full-frame) or a spatially focused window, and on whether the input is unprocessed RGB or depth depth versus a flow field.This conclusion is then used to extend a state-of-the-art architecture for isolated gesture recognition, FOANet [24], to continuous gesture recognition. The result is a system that established a new state-of-the-art for recognition performance on the ChaLearn ConGD data set, with a mean Jaccard Index of 0.77 compared to the previous best result of 0.61. This paper also establishes a baseline of performance for the newer, continuous version of the NVIDIA dataset.

Gestures are a common form of human communication and important for human computer interfaces (HCI). Recent approaches to gesture recognition use deep learning methods, including multi-channel methods. We show that when spatial channels are focused on the hands, gesture recognition improves significantly, particularly when the channels are fused using a sparse network. Using this technique, we improve performance on the ChaLearn IsoGD dataset from a previous best of 67.71% to 82.07%, and on the NVIDIA dataset from 83.8% to 91.28%.

Video annotation is a vital part of research examining gestural and multimodal interaction as well as computer vision, machine learning, and interface design. However, annotation is a difficult, time-consuming task that requires high cognitive effort. Existing tools for labeling and annotation still require users to manually label most of the data, limiting the tools helpfulness. In this paper, we present the Easy Automatic Segmentation Event Labeler (EASEL), a tool supporting gesture analysis. EASEL streamlines the annotation process by introducing assisted annotation, using automatic gesture segmentation and recognition to automatically annotate gestures. To evaluate the efficacy of assisted annotation, we conducted a user study with 24 participants and found that assisted annotation decreased the time needed to annotate videos with no difference in accuracy compared with manual annotation. The results of our study demonstrate the benefit of adding computational intelligence to video and audio annotation tasks.

People, cars and other moving objects in videos generate time series data that can be labeled in many ways. For example, classifiers can label motion tracks according to the object type, the action being performed, or the trajectory of the motion. These labels can be generated for every frame as long as the object stays in view, so object tracks can be modeled as Markov processes with multiple noisy observation streams. A challenge in video recognition is to recover the true state of the track (i.e. its class, action and trajectory) using Markov models without (a) counter-factually assuming that the streams are independent or (b) creating a fully coupled Hidden Markov Model (FCHMM) with an infeasibly large state space. This paper introduces a new method for labeling sequences of hidden states. The method exploits external consistency constraints among streams without modeling complex joint distributions between them. For example, common sense semantics suggest that trees cannot walk. This is an example of an external constraint between an object label (“tree”) and an action label (“walk”). The key to exploiting external constraints is a new variation of the Viterbi algorithm which we call the Viterbi–Segre (VS) algorithm. VS restricts the solution spaces of factorized HMMs to marginal distributions that are compatible with joint distributions satisfying sets of external constraints. Experiments on synthetic data show that VS does a better job of estimating true states with the given observations than the traditional Viterbi algorithm applied to (a) factorized HMMs, (b) FCHMMs, or (c) partially-coupled HMMs that model pairwise dependencies. We then show that VS outperforms factorized and pairwise HMMs on real video data sets for which FCHMMs cannot feasibly be trained.

Visual classifiers are part of many applications including surveillance, autonomous navigation and scene understanding. The raw data used to train these classifiers is abundant and easy to collect but lacks labels. Labels are necessary for training supervised classifiers, but the labeling process requires significant human effort. Techniques like active learning and group-based labeling have emerged to help reduce the labeling workload. However, the possibility of collecting label noise affects either the efficiency of these systems or the performance of the trained classifiers. Further, many introduce latency by iteratively retraining classifiers or re-clustering data. We introduce a technique that searches for structural change in hierarchically clustered data to identify a set of clusters that span a spectrum of visual concept granularities. This allows us to efficiently label clusters with less label noise and produce high performing classifiers. The data is hierarchically clustered only once, eliminating latency during the labeling process. Using benchmark data we show that collecting labels with our approach is more efficient than existing labeling techniques, and achieves higher classification accuracy. Finally, we demonstrate the speed and efficiency of our system using real-world data collected for an autonomous navigation task.

Many computer vision algorithms employ subspace models to represent data. Many of these approaches benefit from the ability to create an average or prototype for a set of subspaces. The most popular method in these situations is the Karcher mean, also known as the Riemannian center of mass. The prevalence of the Karcher mean may lead some to assume that it provides the best average in all scenarios. However, other subspace averages that appear less frequently in the literature may be more appropriate for certain tasks. The extrinsic manifold mean, the L2-median, and the flag mean are alternative averages that can be substituted directly for the Karcher mean in many applications. This paper evaluates the characteristics and performance of these four averages on synthetic and real-world data. While the Karcher mean generalizes the Euclidean mean to the Grassman manifold, we show that the extrinsic manifold mean, the L2-median, and the flag mean behave more like medians and are therefore more robust to the presence of outliers among the subspaces being averaged. We also show that while the Karcher mean and L2-median are computed using iterative algorithms, the extrinsic manifold mean and flag mean can be found analytically and are thus orders of magnitude faster in practice. Finally, we show that the flag mean is a generalization of the extrinsic manifold mean that permits subspaces with different numbers of dimensions to be averaged. The result is a "cookbook" that maps algorithm constraints and data properties to the most appropriate subspace mean for a given application.

The Point-and-Shoot Face Recognition Challenge (PaSC) is a performance evaluation challenge including 1401 videos of 265 people acquired with handheld cameras and depicting people engaged in activities with non-frontal head pose. This report summarizes the results from a competition using this challenge problem. In the Video-to-video Experiment a person in a query video is recognized by comparing the query video to a set of target videos. Both target and query videos are drawn from the same pool of 1401 videos. In the Still-to-video Experiment the person in a query video is to be recognized by comparing the query video to a larger target set consisting of still images. Algorithm performance is characterized by verification rate at a false accept rate of 0.01 and associated receiver operating characteristic (ROC) curves. Participants were provided eye coordinates for video frames. Results were submitted by 4 institutions: (i) Advanced Digital Science Center, Singapore; (ii) CPqD, Brasil; (iii) Stevens Institute of Technology, USA; and (iv) University of Ljubljana, Slovenia. Most competitors demonstrated video face recognition performance superior to the baseline provided with PaSC. The results represent the best performance to date on the handheld video portion of the PaSC.

Methods for assessing the impact of factors and image-quality metrics for still face images are well-understood. The extension of these factors and quality measures to faces in video has not, however, been explored. We present a specific methodology for carrying out this extension from still to video. Using the Point-and-Shoot Challenge (PaSC) dataset, our study investigates the effect of nine factors on three face recognition algorithms, and identifies the most important factors for algorithm performance in video. We also evaluate four factor metrics for characterizing a single video as well as two comparative metrics for pairs of videos. For video-based face recognition, the analysis shows that distribution-based metrics are generally more effective in quantifying factor values than algorithm-dependent metrics. For predicting face recognition performance in video, we observe that the face detection confidence and face size factors are potentially useful quality measures. From our data, we also find that males are easier to identify than females, and Asians easier to identify than Caucasians. Finally, for this PaSC video dataset, face recognition algorithm performance is primarily driven by environment and sensor factors.

A new algorithm for learning binary codes is presented using randomized initial assignments of bit labels to classes followed by iterative refinement to minimize intraclass Hamming distance. This Randomized Intraclass-Distance Minimizing Binary Codes (RIDMBC) algorithm is introduced in the context of face recognition, an area of biometrics where binary codes have rarely been used (unlike iris recognition). A cross-database experiment is presented training RIDMBC on the Labeled Faces in the Wild (LFW) and testing it on the Point-and-Shoot Challenge (PaSC). The RIDMBC algorithm performs better than both PaSC baselines. RIDMBC is compared with the Predictable Discriminative Binary Codes (DBC) algorithm developed by Rastegari et al. The DBC algorithm has an upper bound on the number of bits in a binary code; RIDMBC does not. RIDMBC outperforms DBC when using the same bit code length as DBC's upper bound and RIDMBC further improves when more bits/features are added.

Handheld videos include unintentional motion (jitter) and often intentional motion (pan and/or zoom). Human viewers prefer to see jitter removed, creating a smoothly moving camera. For video analysis, in contrast, aligning to a fixed stable background is sometimes preferable. This paper presents an algorithm that removes both forms of motion using a novel and efficient way of tracking background points while ignoring moving foreground points. The approach is related to image mosaicing, but the result is a video rather than an enlarged still image. It is also related to multiple object tracking approaches, but simpler since moving objects need not be explicitly tracked. The algorithm presented takes as input a video and returns one or several stabilized videos. Videos are broken into parts when the algorithm detects the background changing and it becomes necessary to fix upon a new background. Our approach assumes the person holding the camera is standing in one place and that objects in motion do not dominate the image. Our algorithm performs better than several previously published approaches when compared on 1,401 handheld videos from the recently released Point-and-Shoot Face Recognition Challenge (PASC). The source code for this algorithm is being made available.

Labeling data to train visual concept classifiers requires significant human effort. Active learning addresses labeling overhead by selecting a meaningful subset of data, but often these approaches assume that the set of visual concepts is known in advance. Clustering approaches perform bottom-up discovery of concepts, and reduce labeling effort by moving from instance-based to group-based labeling. Unfortunately, clustering techniques assume a one-to-one mapping between clusters and visual concepts even though learned groups are often not coherent and fail to represent all concepts. We introduce Selective Guidance, a technique that hierarchically clusters data and selectively queries labels of coherent clusters representing different visual concepts. Unlike most active learning and clustering techniques, Selective Guidance does not require any a priori knowledge. Using benchmark data sets we show that Selective Guidance achieves classification accuracy better than active learning and clustering approaches with fewer labeling queries.

In face recognition, quality is typically thought of as a property of individual images, not image pairs. The implicit assumption is that high-quality images should be easy to match to each other, while low quality images should be hard to match. This paper presents a relational graph-based evaluation technique that uses match scores produced by face recognition algorithms to determine the “quality” of images. The resulting analysis demonstrates that only a small fraction of the images in a well-studied data set (FRVT 2006) are low-quality images. It is much more common to find relationships in which two images that are hard to match to each other can be easily matched with other images of the same person. In other words, these images are simultaneously both high and low quality. The existence of such contrary images represents a fundamental challenge for approaches to biometric quality that cast quality as an intrinsic property of a single image. Instead it indicates that quality should be associated with pairs of images. In exploring these contrary images, we find a surprising dependence on whether elements of an image pair are acquired at the same location, even in circumstances where one would be tempted to think of the locations as interchangeable. The results presented have important implications for anyone designing face recognition evaluations as well as those developing new algorithms.

Common human actions are instantly recognizable by people and increasingly machines need to understand this language if they are to engage smoothly with people. Here we introduce a new method for automated human action recognition. The proposed method represents videos as a tangent bundle on a Grassmann manifold. Videos are expressed as third order tensors and factorized to a set of tangent spaces. Tangent vectors are then computed between elements on a Grassmann manifold and exploited for action classification. In particular, logarithmic mapping is applied to map a point from the manifold to tangent vectors centered at a given element. The canonical metric is used to induce the intrinsic distance for a set of tangent spaces. Empirical results show that our method is effective on both uniform and non-uniform backgrounds for action classification. We achieve recognition rates of 91% on the Cambridge gesture dataset, 88% on the UCF sport dataset, and 97% on the KTH human action dataset. Additionally, our method does not require prior training.

The Good, the Bad, & the Ugly Face Challenge Problem was created to encourage the development of algorithms that are robust to recognition across changes that occur in still frontal faces. The Good, the Bad, & the Ugly consists of three partitions. The Good partition contains pairs of images that are considered easy to recognize. On the Good partition, the base verification rate (VR) is 0.98 at a false accept rate (FAR) of 0.001. The Bad partition contains pairs of images of average difficulty to recognize. For the Bad partition, the VR is 0.80 at a FAR of 0.001. The Ugly partition contains pairs of images considered difficult to recognize, with a VR of 0.15 at a FAR of 0.001. The base performance is from fusing the output of three of the top performers in the FRVT 2006. The design of the Good, the Bad, & the Ugly controls for pose variation, subject aging, and subject “recognizability.” Subject recognizability is controlled by having the same number of images of each subject in every partition. This implies that the differences in performance among the partitions are result of how a face is presented in each image.

Several studies have shown the existence of biometric zoos. The premise is that in biometric systems people fall into distinct categories, labeled with animal names, indicating recognition difficulty. Different combinations of excessive false accepts or rejects correspond to labels such as: Goat, Lamb, Wolf, etc. Previous work on biometric zoos has investigated the existence of zoos for the results of an algorithm on a data set. This work investigates biometric zoos generalization across algorithms and data sets. For example, if a subject is a Goat for algorithm A on data set X, is that subject also a Goat for algorithm B on data set Y? This paper introduces a theoretical framework for generalizing biometric zoos. Based on our framework, we develop an experimental methodology for determining if biometric zoos generalize across algorithms and data sets, and we conduct a series of experiments to investigate the existence of zoos on two algorithms in FRVT 2006.

Although not commonly used, correlation filters can track complex objects through rotations, occlusions and other distractions at over 20 times the rate of current state-of-the-art techniques. The oldest and simplest correlation filters use simple templates and generally fail when applied to tracking. More modern approaches such as ASEF and UMACE perform better, but their training needs are poorly suited to tracking. Visual tracking requires robust filters to be trained from a single frame and dynamically adapted as the appearance of the target object changes. This paper presents a new type of correlation filter, a Minimum Output Sum of Squared Error (MOSSE) filter, which produces stable correlation filters when initialized using a single frame. A tracker based upon MOSSE filters is robust to variations in lighting, scale, pose, and nonrigid deformations while operating at 669 frames per second. Occlusion is detected based upon the peak-to-sidelobe ratio, which enables the tracker to pause and resume where it left off when the object reappears.

Videos can be naturally represented as multidimensional arrays known as tensors. However, the geometry of the tensor space is often ignored. In this paper, we argue that the underlying geometry of the tensor space is an important property for action classification. We characterize a tensor as a point on a product manifold and perform classification on this space. First, we factorize a tensor relating to each order using a modified High Order Singular Value Decomposition (HOSVD). We recognize each factorized space as a Grassmann manifold. Consequently, a tensor is mapped to a point on a product manifold and the geodesic distance on a product manifold is computed for tensor classification. We assess the proposed method using two public video databases, namely Cambridge-Gesture gesture and KTH human action data sets. Experimental results reveal that the proposed method performs very well on these data sets. In addition, our method is generic in the sense that no prior training is needed.

We present an adaptive framework for condensation algorithms in the context of human-face tracking. We attack the face tracking problem by making factored sampling more efficient and appearance update more effective. An adaptive affine cascade factored sampling strategy is introduced to sample the parameter space such that coarse face locations are located first, followed by a fine factored sampling with a small number of particles. In addition, the local linearity of an appearance manifold is used in conjunction with a new criterion to select a tangent plane for updating an appearance in face tracking. Our proposed method seeks the best linear variety from the selected tangent plane to form a reference image. We demonstrate the effectiveness and efficiency of the proposed method on a number of challenging videos. These test video sequences show that our method is robust to illumination, appearance, and pose changes, as well as temporary occlusions. Quantitatively, our method achieves the average root-mean-square error at 4.98 on the well-known dudek video sequence while maintaining a proficient speed at 8.74 fps. Finally, while our algorithm is adaptive during execution, no training is required.

This paper introduces a class of correlation filters called average of synthetic exact filters (ASEF). For ASEF, the correlation output is completely specified for each training image. This is in marked contrast to prior methods such as synthetic discriminant functions (SDFs) which only specify a single output value per training image. Advantages of ASEF training include: insensitivity to over-fitting, greater flexibility with regard to training images, and more robust behavior in the presence of structured backgrounds. The theory and design of ASEF filters is presented using eye localization on the FERET database as an example task. ASEF is compared to other popular correlation filters including SDF, MACE, OTF, and UMACE, and with other eye localization methods including Gabor Jets and the OpenCV cascade classifier. ASEF is shown to outperform all these methods, locating the eye to within the radius of the iris approximately 98.5% of the time.

The theory of illumination subspaces is well developed and has been tested extensively on the Yale Face Database B (YDB) and CMU-PIE (PIE) data sets. This paper shows that if face recognition under varying illumination is cast as a problem of matching sets of images to sets of images, then the minimal principal angle between subspaces is sufficient to perfectly separate matching pairs of image sets from nonmatching pairs of image sets sampled from YDB and PIE. This is true even for subspaces estimated from as few as six images and when one of the subspaces is estimated from as few as three images if the second subspace is estimated from a larger set (10 or more). This suggests that variation under illumination may be thought of as useful discriminating information rather than unwanted noise.

This paper summarizes a study carried out on data from the Face Recognition Vendor Test 2006 (FRVT 2006). The finding of greatest practical importance is the discovery of a strong connection between a relatively simple measure of image quality and performance of state-of-the-art vendor algorithms in FRVT 2006. The image quality measure quantifies edge density and likely relates to focus. This effect is part of a larger four-way interaction observed between edge density, face size and whether images are acquired indoors our outdoors. This finding illustrates the broader potential for statistical modeling of empirical data to play an important role in finding and codifying biometric quality measures.

An image-set based face recognition algorithm is proposed that exploits the full geometrical interpretation of Canonical Correlation Analysis (CCA). CCA maximizes the correlation between two linear subspaces associated with image-sets, where an image-set is assumed to contain multiple images of a person's face. When these linear subspaces are viewed as points on a Grassmann manifold, then geodesic distance on the manifold becomes the natural way to compare image-sets. The proposed method is tested on the ORL data set where it achieves a rank one identification rate of 98.75%. The proposed method is also tested on a subset of the Face Recognition Grand Challenge Experiment 4 data. Specifically, 82 probe and 230 gallery subjects with 32 images per probe and gallery image-set. Our algorithm achieves a rank one identification rate of 87% and a verification rate of 81% at a false accept rate of 1/1;000. These results on FRGC are significantly better than the well-known image-set matching algorithm, Mutual Subspace Method (MSM), which does not use geodesic distance. Another important finding is that cohort normalization boosts verification performance by 50% when used in conjunction with image-set matching. These results suggest that excellent levels of face recognition performance are possible when using image-sets, geodesic distance and cohort normalization. Finally, the proposed approach is generic in the sense that no training is required.

In contrast with the diverse array of genetic algorithms, the Genetic Programming (GP) paradigm is usually applied in a relatively uniform manner. Heuristics have developed over time as to which replacement strategies and selection methods are best. The question addressed in this paper is relatively simple: since there are so many variants of evolutionary algorithm, how well do some of the other well known forms of evolutionary algorithm perform when used to evolve programs trees using s-expressions as the representation? Our results suggest a wide range of evolutionary algorithms are all equally good at evolving programs, including the simplest evolution strategies.

We present a Genetic Programming approach to evolve cooperative controllers for teams of UAVs. Our focus is a collaborative search mission in an uncertain and/or hostile environment. The controllers are decision trees constructed from a set of low-level functions. Evolved decision trees are robust to changes in initial mission parameters and approach the optimal bound for time-to-completion. We compare results between steady-state and generational approaches, and examine the effects of two common selection operators.

The Cameron project has developed a language called single assignment C (SA-C), and a compiler for mapping image-based applications written in SA-C to field programmable gate arrays (FPGAs). The paper tests this technology by implementing several applications in SA-C and compiling them to an Annapolis Microsystems (AMS) WildStar board with a Xilinx XV2000E FPGA. The performance of these applications on the FPGA is compared to the performance of the same applications written in assembly code or C for an 800 MHz Pentium III. (Although no comparison across processors is perfect, these chips were the first of their respective classes fabricated at 0.18 microns, and are therefore of comparable ages.) We find that applications written in SA-C and compiled to FPGAs are between 8 and 800 times faster than the equivalent program run on the Pentium III.

This paper describes the implementation of an automatic target recognition (ATR) Probing algorithm on a reconfigurable system, using the SA-C programming language and optimizing compiler. The reconfigurable system is 800 times faster than a comparable Pentium running a C implementation of the same probing task. The reasons for this are analyzed.

The FERET evaluation compared recognition rates for different semi-automated and automated face recognition algorithms. We extend FERET by considering when differences in recognition rates are statistically distinguishable subject to changes in test imagery. Nearest Neighbor classifiers using principal component and linear discriminant subspaces are compared using different choices of distance metric. Probability distributions for algorithm recognition rates and pairwise differences in recognition rates are determined using a permutation methodology. The principal component subspace with Mahalanobis distance is the best combination; using L2 is second best. Choice of distance measure for the linear discriminant subspace matters little, and performance is always worse than the principal components classifier using either Mahalanobis or L1 distance. We make the source code for the algorithms, scoring procedures and Monte Carlo study available in the hopes others will extend this comparison to newer algorithms.

This paper describes a course in image computation that is designed to follow and build up an established course in computer graphics. The course is centered on images: how they are generated, manipulated, matched and symbolically described. It builds on the student's knowledge of coordinate systems and the perspective projection pipeline. It covers image generation techniques not covered by the computer graphics course, most notably ray tracing. It introduces students to basic image processing concepts such as Fourier analysis and then to basic computer vision topics such as principal components analysis, edge detection and symbolic feature matching. The goal is to prepare students for advanced work in either computer vision or computer graphics.

As object recognition algorithms focus more closely on appearance, the problem of how best to measure similarity between images becomes critical. Within the context of an existing object recognition system, we present an empirical comparison of 27 different error functions on a set of 20 example problems. Each error function measures the similarity between an observed and a predicted image. Our goal is to select an error best suited to guide an heuristic search algorithm through a space of possible scene configurations.

There is a long tradition of using heuristic search within the context of object recognition. However, most of these techniques rely solely on object geometry and are based on matching model features to data features. We present a novel approach for using search coupled with an error function based on graphical rendering. An empirical approach is used to evaluate five different search algorithms on a set of 20 example problems.

Color is a powerful cue for determining the pose of 3D objects viewed by a single camera. We present a method based on hue histograms for locating multiple objects with respect to a fixed camera. The algorithm is tested on controlled blocks-world scenes and an indoor office scene. On these examples, a pose refinement algorithm performs better when guided by color than when guided by more traditional edge information.

Velocity Picking is the problem of picking velocity-time pairs based on a coherence metric between multiple seismic signals. Coherence as a function of velocity and time can be expressed as a 2-D color semblance velocity image. Currently, humans pick velocities by looking at the semblance velocity image; this process can take days or even weeks to complete for a seismic survey. The problem can be posed as a geometric feature matching problem. A feature extraction algorithm can recognize islands (peaks) of maximal semblance in the semblance velocity image: a heuristic combinatorial matching process can then be used to find a subset of peaks which maximizes the coherence metric. The peaks define a polyline through the image, and coherence is measured in terms of the summed velocity under the polyline and the smoothness of the polyline. Our best algorithm includes a constraint favoring solution near the median solution for the local area under consideration. Each image is first processed independently. Then, a second pass of optimization includes proximity to the median as an additional optimization criterion. Our results are similar to those produced by human experts.

Most model-based object recognition algorithms attempt to match stored model features to features extracted from imagery. The better the match, the more likely it is that the object is present in the scene. Problems arise when objects are occluded because matches will be incomplete. It is rare for an object recognition algorithm to employ knowledge to explain the absence of occluded features. The work presented here illustrates an approach to object recognition which propagates evidence of occlusion from range to optical sensors and thereby explains missing features.

Local search is a well established and highly effective method for solving complex combinatorial optimization problems. Here, local search is adapted to solve difficult geometric matching problems. Matching is posed as the problem of finding the optimal many-to-many correspondence mapping between a line segment model and image line segments. Image data is assumed to be fragmented, noisy, and cluttered. The algorithms presented have been used for robot navigation, photo interpretation, and scene understanding. This paper explores how local search performs as model complexity increases, image clutter increases, and additional model instances are added to the image data. Expected run-times to find optimal matches with 95 percent confidence are determined for 48 distinct problems involving six models. Nonlinear regression is used to estimate run-time growth as a function of problem size. Both polynomial and exponential growth models are fit to the run-time data. For problems with random clutter, the polynomial model fits better and growth is comparable to that for tree search. For problems involving symmetric models and multiple model instances, where tree search is exponential, the polynomial growth model is superior to the exponential growth model for one search algorithm and comparable for another.

This paper presents a three-dimensional (3-D) model-based ATR algorithm that operates simultaneously on imagery from three heterogeneous, approximately boresight aligned sensors. An iterative search matches models to range and optical imagery by repeatedly predicting detectable features, measuring support for these features in the imagery, and adjusting the transformations relating the target to the sensors in order to improve the match. The result is a locally optimal and globally consistent set of 3-D transformations that precisely relate the best matching target features to combined range, IR, and color images. Results show the multisensor algorithm recovers 3-D target pose more accurately than does a traditional single-sensor algorithm. Errors in registration between images are also corrected during matching. The intended application is imaging from semiautonomous military scout vehicles.

A least-squares method simultaneously solves for the model-to-sensor-suite pose and sensor-to-sensor registration. The development is for a sensor-suite containing separate range and optical sensors. To address outliers and, more generally, match finding, a statistical method (median filtering) and a search method (local search) are developed. Sensitivity to Gaussian noise and the choice of initial pose estimates is investigated on synthetic data. Both of the matching methods are demonstrated on real data.

The object recognition, system presented combines on-line feature prediction with an iterative multisensor matching algorithm. Matching begins with an initial object type and pose hypothesis. An iterative generate-and-test procedure then refines the pose as well as the sensor-to-sensor registration for separate range and electro optical sensors. During matching, object features predicted to be visible are updated to reflect changes in hypothesized object pose and sensor registration. The match found is locally optimal in terms of the complete space of possible matches and globally consistent in the sense of preserving the 3D constraints implied by sensor and object geometry. Results on real data are presented which demonstrate the algorithm correcting for up to 30/spl deg/ errors in initial orientation and 5 m errors in initial translation.

We consider the problem of matching perspective views of coplanar structures composed of line segments. Both model-to-image and image-to-image correspondence matching are given a consistent treatment. These matching scenarios generally require discovery of an eight parameter projective mapping. However, when the horizon line of the object plane can be found in the image, done here using vanishing point analysis, these problems reduce to a simpler six parameter affine matching problem. When the intrinsic lens parameters of the camera are known, the problem further reduces to four parameter affine similarity matching.

Full-perspective mappings between 3-D objects and 2-D images are more complicated than weak-perspective mappings, which consider only rotation, translation, and scaling. Therefore, in 3-D model-based robot navigation, it is important to understand how and when full-perspective must be taken into account. A probabilistic combinatorial optimization algorithm is used to search for an optimal match between 3-D landmarks and 2-D image features. Three variations are considered. A weak-perspective algorithm rotates, translates, and scales an initial 2-D projection of the 3-D landmark. A full perspective selects a most promising alternative, but then updates the pose and reprojects the landmark. Like the full-perspective algorithm, the hybrid algorithm reliably recovers the true pose of the robot, and like the weak-perspective algorithm, it runs 5 to 10 faster than the full-perspective algorithm.>

The history of the image understanding environment (IUE) project, a five-year program to develop a common software environment for the development of algorithms and application systems, is reviewed. An overview of some of the data structures that are currently evolving as a specification for the IUE is provided. The ultimate goal of the project is to provide the basic data structures and algorithms that are required to carry state-of-the-art research in image understanding.>

A local search optimization procedure, subset-convergent local search, is developed for identifying known 2D models in broken and skewed line data. Due to fragmentation in the data, the mapping between model and data line segments may be one-to-many. The proposed algorithm starts with a model-data mapping selected at random and refines it until it is locally optimal. Results are presented in which the algorithm frequency converges upon the globally optimal match. This best match can be found with greater confidence by independently running the algorithm several times. The match error being minimized has two basic components, a spatial fit error and an omission error. Spatial fit is a function of the integrated squared perpendicular distance between model lines and data line segments. This measure is minimized, subject to rotation, translation, and scaling, for every set of correspondences tested. The closed-form equation solving for this optimal transform is second order. Although the matching system itself is 2D, results are presented in which the matcher, in conjunction with a 3D robot navigation system, correctly updates estimates of robot position.>

The authors report on the system and methods used by UMass Mobile Robot Project. Model-based processing of the visual sensory data is the primary mechanism used for controlling movement of an autonomous land vehicle through the environment, measuring progress towards a given goal, and avoiding obstacles. Goal-oriented navigation takes place through a partially modeled, unchanging environment that contains no unmodeled obstacles; this simplified environment provides a foundation for research in more complicated domains. The navigation system integrates perception, planning, and execution of actions. Of particular importance is that the planning processes are reactive and reason about landmarks that should be perceived at various stages of task execution. Correspondence between image features and expected landmark locations are used at several abstraction levels to ensure proper plan execution. The system and some experiments that demonstrate the performance of its components is described.>