Identification of cracks is essential to assess the structural integrity of concrete infrastructure. However, robust crack segmentation remains a challenging task for computer vision systems due to the diverse appearance of concrete surfaces, variable lighting and weather conditions, and the overlapping of different defects. In particular recent data-driven methods struggle with the limited availability of data, the fine-grained and time-consuming nature of crack annotation, and face subsequent difficulty in generalizing to out-of-distribution samples. In this work, we move past these challenges in a two-fold way. We introduce a high-fidelity crack graphics simulator based on fractals and a corresponding fully-annotated crack dataset. We then complement the latter with a system that learns generalizable representations from simulation, by leveraging both a pointwise mutual information estimate along with adaptive instance normalization as inductive biases. Finally, we empirically highlight how different design choices are symbiotic in bridging the simulation to real gap, and ultimately demonstrate that our introduced system can effectively handle real-world crack segmentation.

Current deep learning methods are regarded as favorable if they empirically perform well on dedicated test sets. This mentality is seamlessly reflected in the resurfacing area of continual learning, where consecutively arriving data is investigated. The core challenge is framed as protecting previously acquired representations from being catastrophically forgotten. However, comparison of individual methods is nevertheless performed in isolation from the real world by monitoring accumulated benchmark test set performance. The closed world assumption remains predominant, i.e. models are evaluated on data that is guaranteed to originate from the same distribution as used for training. This poses a massive challenge as neural networks are well known to provide overconfident false predictions on unknown and corrupted instances. In this work we critically survey the literature and argue that notable lessons from open set recognition, identifying unknown examples outside of the observed set, and the adjacent field of active learning, querying data to maximize the expected performance gain, are frequently overlooked in the deep learning era. Hence, we propose a consolidated view to bridge continual learning, active learning and open set recognition in deep neural networks. Finally, the established synergies are supported empirically, showing joint improvement in alleviating catastrophic forgetting, querying data, selecting task orders, while exhibiting robust open world application.

Recognition of defects in concrete infrastructure, especially in bridges, is a costly and time consuming crucial first step in the assessment of the structural integrity. Large variation in appearance of the concrete material, changing illumination and weather conditions, a variety of possible surface markings as well as the possibility for different types of defects to overlap, make it a challenging real-world task. In this work we introduce the novel COncrete DEfect BRidge IMage dataset (CODEBRIM) for multi-target classification of five commonly appearing concrete defects. We investigate and compare two reinforcement learning based meta-learning approaches, MetaQNN and efficient neural architecture search, to find suitable convolutional neural network architectures for this challenging multi-class multi-target task. We show that learned architectures have fewer overall parameters in addition to yielding better multi-target accuracy in comparison to popular neural architectures from the literature evaluated in the context of our application.

Generalization performance of trained computer vision (CV) systems that use computer graphics (CG) generated data is not yet effective due to the concept of domain-shift between virtual and real data. Although simulated data augmented with a few real-world samples has been shown to mitigate domain shift and improve transferability of trained models, guiding or bootstrapping the virtual data generation with the distributions learnt from target real world domain is desired, especially in the fields where annotating even few real images is laborious (such as semantic labeling, optical flow, and intrinsic images etc.). In order to address this problem in an unsupervised manner, our work combines recent advances in CG, which aims at generating stochastic scene layouts using large collections of 3D object models, and generative adversarial training, which aims at training generative models by measuring discrepancy between generated and real data in terms of their separability in the space of a deep discriminatively-trained classifier. Our method uses iterative estimation of the posterior density of prior distributions for a generative graphical model. This is done within a rejection sampling framework. Initially, we assume uniform distributions as priors over parameters of a scene described by a generative graphical model. As iterations proceed the uniform prior distributions are updated sequentially to distributions that are closer to the unknown distributions of target data. We demonstrate the utility of adversarially tuned scene generation on two real world benchmark datasets (CityScapes and CamVid) for traffic scene semantic labeling with a deep convolutional net (DeepLab). We obtained performance improvements by 2.28 and 3.14 points on the IoU metric between the DeepLab models trained on simulated sets prepared from the scene generation models before and after tuning to CityScapes and CamVid respectively.

There is a growing interest to utilize Computer Graphics (CG) renderings to generate large scale annotated data in order to train machine learning systems, such as Deep convolutional neural networks, for Computer Vision (CV). However, there has been a long debate on the usefulness of CG generated data for tuning CV systems (even from the 1980's). Especially, the impact of modeling errors and computational rendering approximations, due to choices in the rendering pipeline, on trained CV systems generalization performance is still not clear. In this paper, we take a case study in traffic scenario to empirically analyze the performance degradation when CV systems trained with virtual data are transferred to real data. We: a) discuss a generative model coupled with 3D CAD shapes for scene instance synthesis and, b) explore system performance tradeoffs due to the choice of rendering engine (e.g. Lambertian shader (LS), ray-tracing (RT), and Monte-carlo path tracing (MCPT)) and their respective parameters. DeepLab, that performs semantic segmentation, is chosen as the CV system being evaluated. In our case study, involving traffic scenes, when the CV system is trained with CG data samples (that use MCPT or RT) and augmented with only 10% of real-world training data from CityScapes dataset, the performance levels achieved are comparable to that of training DeepLab with the complete CityScapes dataset. Use of samples from LS degraded the performance of DeepLab by 20%. Physics-based MCPT rendering improved the performance by 6% but at the cost of more than 3 times the rendering time.

Modeling representations of image patches that are quasi-invariant to spatial deformations is an important problem in computer vision. In this paper, we propose a novel concept, the texture trace, that allows sparse patch representations which are quasi-invariant to smooth deformations and robust against occlusions. We first propose a continuous domain model, the profile trace, which is a function only of the topological properties of an image and is by construction invariant to any homeomorphic transformation of the domain. We analyze its theoretical properties and then derive a discrete-domain approximation, the Discrete Texture Trace (DTT). DTTs are designed to be computationally practical and shown by a set of controlled experiments to be quasi-invariant to smooth spatial deformations as well as common image perturbations. We then show how DTTs can be naturally adapted to the incremental tracking problem, yielding highly precise results on par with the state of the art on challenging real data without using heavy machine learning tools. Indeed, we show that with even just using one image at the start of a sequence (i.e. no incremental updating), our method already outperforms four of six state of the art methods of the recent literature on challenging sequences.

We present a change detection method resistant to global and local illumination variations for use in visual surveillance scenarios. Approaches designed thus far for robustness to illumination change are generally based either on color normalization, texture (e.g. edges, rank order statistics, etc.), or illumination compensation. Normalization based methods sacrifice discriminability while texture based methods cannot operate on texture-less regions. Both types of method can produce large missing regions in the distance image which in turn pose problems for higher-level processing tasks that may be shape or region-based and require accurate foreground masks (e.g. person detection and tracking, crowd segmentation, etc.). Texture based methods have an additional problem in that they produce false alarms due to textures induced by local illumination effects (e.g. cast shadows). In this paper we propose a compensation based approach for change detection. Prior work on compensation has largely taken an empirical approach, and has not dealt with the important problem of rejecting outliers when they dominate the scene. In contrast, our generative approach and systematic handling of outliers enables us to achieve robustness to illumination change while eliminating the problems mentioned above. Furthermore, the computational complexity of our method is low enough for real-time performance. Results comparing images taken under strongly different illumination conditions, demonstrate the power and generality of the proposed method.

Vehicles are indispensable in modern life. The capability of monitoring them over a long range can play significant roles in many surveillance applications. However, due to high mobility of vehicles, tracking them is difficult and we need to utilize a large network of cameras and reason on discrete sets of observations made from non-overlapping cameras. In this paper, we introduce enabling techniques for such a surveillance need. Specifically, we build explicit 3D models and use them for vehicle signature extraction and matching. The algorithm uses a single active shape model (ASM) for all consumer vehicles. After detecting presence of a vehicle, \eg, by background subtraction, our algorithm then reconstructs a texture mapped 3D model. 3D car models enable us to monitor vehicles in many novel ways otherwise impossible. Two use cases are provided.

Robustness to illumination variations is a key requirement for the problem of change detection which in turn is a fundamental building block for many visual surveillance applications. The use of ordinal measures is a powerful way of filtering out illumination dependency in representing appearance, and several such measures have been proposed in the past for change detection. By design, these measures are invariant to unknown monotonic transformations that may be caused due to global illumination changes or automatic camera gain. However, previous work has left theoretical and practical gaps that limit their full potential from being realized. For instance, random noise has not been given a principled treatment. In this paper, we formulate the change detection problem in terms of order consistency and show that in the presence of noise with known statistical properties, significance tests for order consistency yield much better results than the state of the art. Since ordinal measures require a reordering of patches, they are usually expensive in practice (O(n*log n) at best). We improve upon this by connecting the problem to monotonic regression, and applying a fast algorithm from the corresponding literature. We also show that good trade offs between speed and accuracy can be made by quantization to achieve accurate and very fast matching algorithms in practice. We demonstrate superior performance on statistical simulations as well as real image sequences.

The capacity to robustly detect humans in video is a critical component of automated visual surveillance systems. This paper describes a bilattice based logical reasoning approach that exploits contextual information and knowledge about interactions between humans, and augments it with the output of different low level detectors for human detection. Detections from low level parts-based detectors are treated as logical facts and used to reason explicitly about the presence or absence of humans in the scene. Positive and negative information from different sources, as well as uncertainties from detections and logical rules, are integrated within the bilattice framework. This approach also generates proofs or justifications for each hypothesis it proposes. These justifications (or lack thereof) are further employed by the system to explain and validate, or reject potential hypotheses. This allows the system to explicitly reason about complex interactions between humans and handle occlusions. These proofs are also available to the end user as an explanation of why the system thinks a particular hypothesis is actually a human. We employ a boosted cascade of gradient histograms based detector to detect individual body parts. We have applied this framework to analyze the presence of humans in static images from different datasets.

This paper presents a fast, accurate, and novel method for the problem of estimating the number of humans and their positions from background differenced images obtained from a single camera where inter-human occlusion is significant. The problem is challenging firstly because the state space formed by the number, positions, and articulations of people is large. Secondly, in spite of many advances in background maintenance and change detection, background differencing remains a noisy and imprecise process, and its output is far from ideal: holes, fill-ins, irregular boundaries etc. pose additional challenges for our "mid- level" problem of segmenting it to localize humans. We propose a novel example-based algorithm which maps the global shape feature by Fourier descriptors to various configurations of humans directly. We use locally weighted averaging to interpolate for the best possible candidate configuration. The inherent ambiguity resulting from the lack of depth and layer information in the background difference images is mitigated by the use of dynamic programming, which finds the trajectory in state space that best explains the evolution of the projected shapes. The key components of our solution are simple and fast. We demonstrate the accuracy and speed of our approach on real image sequences.

Reliability of a model-based edge tracker critically depends on its ability to establish correct correspondences between points on the model edges and edge pixels in an image. This is a non-trivial problem especially in the presence of large inter-frame motions and in cluttered environments. We propose an online learning approach to solving this problem. An edge pixel is represented by a descriptor composed of a small segment of intensity patterns. From training examples the algorithm utilizes the randomized forest model to learn a posteriori distribution of correspondence given the descriptor. In a new frame, the edge pixels are classified using maximum a posteriori (MAP) estimation. The proposed method is very powerful and it enables us to apply the proposed tracker to many previously impossible scenarios with unprecedented robustness.

Single-camera face recognition has severe limitations when the subject is not cooperative, or there are pose changes and different illumination conditions. Face recognition using multiple synchronized cameras is proposed to overcome the limitations. We introduce a reliability measure trained from examples to evaluate the inherent quality of channel recognition. The recognition from the channel predicted to be the most reliable is selected as the final recognition results. In this paper, we enhance Adaboost to improve the component based face detector running in each channel as well as the channel reliability measure training. Effective features are designed to train the channel reliability measure using data from both face detection and recognition. The recognition rate is far better than that of either single channel, and consistently better than common classifier fusion rules

This paper proposes a new variational bound optimization framework for incorporating spatial prior information to the mean shift-based data-driven mode analysis, offering flexible control of the mean shift convergence. Two forms of Gaussian spatial priors are considered. Attractive prior pulls the convergence toward a desired location. Repulsive prior pushes away from such a location. Using a generic variational optimization formulation via construction of quadratic lower and upper bounds, we show that the priorconstrained mean shift step can be interpreted as an information fusion of the data and prior terms in the sense of the best linear unbiased estimator. This approach is used to propose a mode parsing algorithm using the inhibitionof-return principle. The proposed algorithm is used for a semi-automatic 3D segmentation of lung nodules in CT data for evaluating its effectiveness. Our experiments demonstrate that the proposed solution can successfully segment challenging wall-attached cases. 1

Determining the correspondence of image patches is one of the most important problems in Computer Vision. When the intensity space is variant due to several factors such as the camera gain or gamma correction, one needs methods that are robust to such transformations. While the most common assumption is that of a linear transformation, a more general assumption is that the change is monotonic. Therefore, methods have been developed previously that work on the rankings between different pixels as opposed to the intensities themselves. In this paper, we develop a new matching method that improves upon existing methods by using a combination of intensity and rank information. The method considers the difference in the intensities of the changed pixels in order to achieve greater robustness to Gaussian noise. Furthermore, only uncorrelated order changes are considered, which makes the method robust to changes in a single or a few pixels. These properties make the algorithm quite robust to different types of noise and other artifacts such as camera shake or image compression. Experiments illustrate the potential of the approach in several different applications such as change detection and feature matching.

We present a tunable representation for tracking that simultaneously encodes appearance and geometry in a manner that enables the use of mean-shift iterations for tracking. The classic formulation of the tracking problem using mean-shift iterations encodes spatial information very loosely (i.e. using radially symmetric kernels). A problem with such a formulation is that it becomes easy for the tracker to get confused with other objects having the same feature distribution but different spatial configurations of features. Subsequent approaches have addressed this issue but not to the degree of generality required for tracking specific classes of objects and motions (e.g. humans walking). In this paper, we formulate the tracking problem in a manner that encodes the spatial configuration of features along with their density and yet retains robustness to spatial deformations and feature density variations. The encoding of spatial configuration is done using a set of kernels whose parameters can be optimized for a given class of objects and motions, off-line. The formulation enables the use of meanshift iterations and runs in real-time. We demonstrate better tracking results on synthetic and real image sequences as compared to the original mean-shift tracker.

Computer vision has found applicability in a wide variety of industrial applications. These include surveillance and security, industrial inspection and automation, medical and automotive. This paper gives an overview of the work performed at Siemens Corporate Research in these areas. Some of the basic technologies that are needed for such applications are highlighted along with a brief overview of the particular constraints existing in such applications. Then, our view of the systems methodology required in order to build robust, scalable and reusable systems and modules is presented. Finally, we present our view of the future directions in this field.

Boosting algorithms have been widely applied in the machine vision systems. Two fundamental issues that have to be solved in these systems are how much training data and how many Boosting rounds are needed to achieve a desired performance. We view the Boosting algorithm as a nonlinear estimation scheme that estimates a strong classifier from a given training sample set (that is generated by sampling a true unknown distribution), the weak classifiers, and the number of Boosting rounds T. The performance characterization of this estimator involves the derivation of the classification error statistics of the trained strong classifier as a function of the training set and the collection of the weak classifiers. Although the convergence and the error bounds for the training error and generalization error of the algorithms have been studied for several years, the estimated bounds are still loose bounds that are only meaningful for large training sets. With no effective tools for determining the error bounds, users are now collecting training samples with as much data as they can afford, with no good way to know if they are sufficient. In this paper, we characterize the classification error statistics of the trained strong classifier as a function of the true distributions of classes, the collection of the weak classifiers, and the size of the training set. We show that the statistics can be numerically computed and the results are more accurate than previous bounds in the literature. Theoretical results are verified through the simulations. Face detection is used as a case study to illustrate the application of the theory on real data.

We describe a semi-automated approach for designing morphological operators to detect image structures. We automatically combine morphological primitives from a pool to generate an enumerable family of complex morphological operators spanning a receiver operating curve (ROC). Domain knowledge can be encoded by biasing the pool with primitives; the system subsequently automatically selects and combines operators based on the joint statistics of training data. The major advantages are that the designer can focus on constructing simple operators, yet is able to rapidly combine them to yield more powerful, system-specific solutions, whose operating point can easily be changed. We illustrate the approach using birth and death processes and associated operators. Examples of video text detection are presented.

Template matching is a fundamental operator in computer vision and is widely used in feature tracking, motion estimation, image alignment, and mosaicing. Under a certain parameterized warping model, the traditional template matching algorithm estimates the geometric warp parameters that minimize the SSD between the target and a warped template. The performance of the template matching can be characterized by deriving the distribution of warp parameter estimate as a function of the ideal template, the ideal warp parameters, and a given noise or perturbation model. In this paper, we assume a discretization of the warp parameter space and derive the theoretical expression for the probability mass function (PMF) of the parameter estimate. As the PMF is also a function of the template size, we can optimize the choice of the template or block size by determining the template/block size that gives the estimate with minimum entropy. Experimental results illustrate the correctness of the theory. An experiment involving feature point tracking in face video is shown to illustrate the robustness of the algorithm in a real-world problem.

In this paper, we propose an edge-driven bidirectional geometric flow for boundary extraction. To this end, we combine the geodesic active contour flow and the gradient vector flow external force for snakes. The resulting motion equation is considered within a level set formulation, can deal with topological changes and important shape deformations. An efficient numerical schema is used for the flow implementation that exhibits robust behavior and has fast convergence rate. Promising results on real and synthetic images demonstrate the potentials of the flow.

Background modeling and subtraction is a core component in motion analysis. The central idea behind such module is to create a probabilistic representation of the static scene that is compared with the current input to perform subtraction. Such approach is efficient when the scene to be modeled refers to a static structure with limited perturbation. In this paper, we address the problem of modeling dynamic scenes where the assumption of a static background is not valid. Waving trees, beaches, escalators, natural scenes with rain or snow are examples. Inspired by the work proposed by Doretto et al. (2003), we propose an on-line auto-regressive model to capture and predict the behavior of such scenes. Towards detection of events we introduce a new metric that is based on a state-driven comparison between the prediction and the actual frame. Promising results demonstrate the potentials of the proposed framework.

We derive a new class of photometric invariants that can be used for a variety of vision tasks including lighting invariant material segmentation, change detection and tracking, as well as material invariant shape recognition. The key idea is the formulation of a scene radiance model for the class of "separable" BRDFs, that can be decomposed into material related terms and object shape and lighting related terms. All the proposed invariants are simple rational functions of the appearance parameters (say, material or shape and lighting). The invariants in this class differ from one another in the number and type of image measurements they require. Most of the invariants in this class need changes in illumination or object position between image acquisitions. The invariants can handle large changes in lighting which pose problems for most existing vision algorithms. We demonstrate the power of these invariants using scenes with complex shapes, materials, textures, shadows and specularities.

A new approach toward target representation and localization, the central component in visual tracking of nonrigid objects, is proposed. The feature histogram-based target representations are regularized by spatial masking with an isotropic kernel. The masking induces spatially-smooth similarity functions suitable for gradient-based optimization, hence, the target localization problem can be formulated using the basin of attraction of the local maxima. We employ a metric derived from the Bhattacharyya coefficient as similarity measure, and use the mean shift procedure to perform the optimization. In the presented tracking examples, the new method successfully coped with camera motion, partial occlusions, clutter, and target scale variations. Integration with motion filters and data association techniques is also discussed. We describe only a few of the potential applications: exploitation of background information, Kalman tracking using motion models, and face tracking.

The paper presents a system for video object generation and selective encoding with applications in surveillance, mobile videophones, and the automotive industry. Object tracking and MPEG-4 compression are performed in real-time. The system belongs to a new generation of intelligent vision sensors called smart cameras, which execute autonomous vision tasks and report events and data to a remote base-station. A detection module signals the presence of an object of interest within the camera field of view, while the tracking part follows the target to generate temporal trajectories. The compression is MPEG-4 compliant and implements the simple profile of the standard, which is capable of encoding up to four video objects. At the same time, the compression is selective, maintaining a higher quality for foreground objects and a lower quality for background representation. This property contributes to bandwidth reduction while preserving the essential information of foreground objects. The system performance is demonstrated in experiments that involve objects representing faces and vehicles seen from both static and moving cameras.

In this paper, we propose a level set formulation to deal with the segmentation and registration of the left ventricle in Magnetic Resonance (MR) images. Our approach is based on the integration of visual information, anatomical constraints and a flexible shape-driven cardiac model. The visual information is expressed through an intensity-based grouping module. The anatomical constraint accounts for the relative positions of the structures of interest. Global shape consistency is introduced by seeking for the lowest potential of the distance between the solution and the prior model. Registration is obtained using the same criterion where the transformation that aligns the latest segmentation map to either the shape model or to the previous segmentation result (temporal domain) is to be recovered.

The traditional approach of handling motion tracking and structure from motion (SFM) independently in successive steps exhibits inherent limitations in terms of achievable precision and incorporation of prior geometric constraints about the scene. This paper proposes a projections onto convex sets (POCS) framework for iterative refinement of the measurement matrix in the well-known factorization method to incorporate multiple geometric constraints about the scene, thereby improving the accuracy of both 2D feature point tracking and 3D structure estimates. Regularities in the scene, such as points on line and plane and parallel lines and planes, that can be interactively identified and marked at each POCS iteration, enforce rank and parallelism constraints on appropriately defined local measurement matrices, one for each constraint. The POCS framework allows for the integration of the information in each of these local measurement matrices into a single measurement matrix that is "closest" to the initial observed measurement matrix in Frobenius norm, which is then factored in the usual manner. Experimental results demonstrate that the proposed interactive POCS framework consistently improves both 2D correspondences and 3D shape/motion estimates and similar results cannot be achieved by enforcing these constraints as either post or preprocessing.

As computer vision systems are increasingly developed and tested in the real-world, there is a significant need to formalize the process of system design and analysis so that engineers can rapidly design, test, and deploy vision systems for real-world applications. Our objective in this paper is to analyze the system design, analysis, and refinement cycle through a case study involving the systematic engineering of a dual-camera video surveillance system for people detection and zooming. We illustrate how an existing system designed and analyzed by following rigorous systematic engineering principles can be extended to relax the system operating conditions with minimal re-design and analysis efforts. The key conclusion is that by choosing appropriate modules and suitable statistical representations, we are able to re-use existing system design and performance analysis results.

There has been an increase in the use of video surveillance and monitoring in public areas to improve safety and security. Change detection and crowding/congestion density estimation are two sub-tasks in a subway monitoring system. We propose a method that decomposes this problem into two steps. The first step consists of a change detection algorithm that distinguishes the background from the foreground. This is done using a discontinuity preserving MRF-based approach where the information from different sources (background subtraction, intensity modeling) is combined with spatial constraints to provide a smooth motion detection map. Then, the obtained change detection map is combined with a geometry module that performs a soft auto-calibration to estimate a measure of congestion of the observed area (platform). Extensive experimental results in a metro station of a metropolitan city demonstrates the performance and the potential of our method.

Outdoor scene classification is challenging due to irregular geometry, uncontrolled illumination, and noisy reflectance distributions. This paper discusses a Bayesian approach to classifying a color image of an outdoor scene. A likelihood model factors in the physics of the image formation process, sensor noise distribution, and prior distributions over geometry, material types, and illuminant spectrum parameters. These prior distributions are learned through a training process that uses color observations of planar scene patches over time. An iterative linear algorithm estimates the maximum likelihood reflectance, spectrum, geometry, and object class labels for a new image. Experiments on images taken by outdoor surveillance cameras classify known material types and shadow regions correctly, and flag as outliers material types that were not seen previously.

Texture replacement in real images has many applications, such as interior design, digital movie making and computer graphics. The goal is to replace some specified texture patterns in an image while preserving lighting effects, shadows and occlusions. To achieve convincing replacement results we have to detect texture patterns and estimate the lighting map from a given image. Near regular planar texture patterns are considered in this paper. Given a sample texture patch, a standard tile is computed. Candidate texture regions are determined by mutual information between the standard tile and each image patch. Regions with high mutual information scores are used to estimate the admissible lighting distributions, which is represented by cached statistics. Spatial lighting change constraints are represented by a Markov random field model. Maximum a posteriori estimation of the texture segmentation and lighting map is solved in a stochastic annealing fashion, namely, the Markov chain Monte Carlo method. Visually satisfactory result is achieved using this statistical sampling model.

Accurate characterization of data distribution is of significant importance for vision problems. In many situations, multivariate visual data often spread into a nonlinear manifold in the high-dimensional space, which makes traditional linear modeling techniques ineffective. This paper proposes a generic nonlinear modeling scheme based on parametric data representations. We build a compact representation for the visual data using a set of parameterized basis (wavelet) functions, where the parameters are randomized to characterize the nonlinear structure of the data distribution. Meanwhile, a new progressive density approximation scheme is proposed to obtain an accurate estimate of the probability density, which imposes discrimination power on the model. Both synthetic and real image data are used to demonstrate the strength of our modeling scheme.

Rapid improvement in computing power, cheap sensing, and more flexible algorithms are facilitating increased development of real-time video surveillance and monitoring systems. The deployment of video understanding systems in certain critical applications in the real world can be done only if performance guarantees can be provided for these systems. This paper reviews past work on a systematic engineering methodology for vision systems performance characterization and illustrates how it can be adapted in practice to develop a real-time people detection and zooming system to meet given application requirements. A case study involving dual-camera real-time video surveillance is used to illustrate that by judiciously choosing the system modules and by performing a careful analysis of the influence of various tuning parameters on the system it is possible to perform proper statistical inference, to automatically set control parameters and to quantify performance limits.

A new method for real time tracking of non-rigid objects seen from a moving camera is proposed. The central computational module is based on the mean shift iterations and finds the most probable target position in the current frame. The dissimilarity between the target model (its color distribution) and the target candidates is expressed by a metric derived from the Bhattacharyya coefficient. The theoretical analysis of the approach shows that it relates to the Bayesian framework while providing a practical, fast and efficient solution. The capability of the tracker to handle in real time partial occlusions, significant clutter, and target scale variations, is demonstrated for several image sequences.

Background modeling is a common component in video surveillance systems and is used to quickly identify regions of interest. To increase the robustness of background subtraction techniques, researchers have developed techniques to update the background model and also developed probabilistic/statistical approaches for thresholding the difference. This paper presents an error analysis of this type of background modeling and pixel labeling, providing both theoretical analysis and experimental validation. Evaluation is centered around the tradeoff of probability of false alarm and probability of miss detection, and this paper shows how to efficiently compute these probabilities front simpler values that are more easily measured. It includes an analysis for both static and dynamic background modeling. The paper also examines the assumptions of Gaussian and mixture of Gaussian models for a pixel.

The engineering of computer vision systems that meet application specific computational and accuracy requirements is crucial to the deployment of real-life computer vision systems. This paper illustrates how past work on a systematic engineering methodology for vision systems performance characterization can be used to develop a real-time people detection and zooming system to meet given application requirements. We illustrate that by judiciously choosing the system modules and performing a careful analysis of the influence of various tuning parameters on the system it is possible to: perform proper statistical inference, automatically set control parameters and quantify limits of a dual-camera real-time video surveillance system. The goal of the system is to continuously provide a high resolution zoomed-in image of a person's head at any location of the monitored area. An omni-directional camera video is processed to detect people and to precisely control a high resolution foveal camera, which has pan, tilt and zoom capabilities. The pan and tilt parameters of the foveal camera and its uncertainties are shown to be functions of the underlying geometry, lighting conditions, background color/contrast, relative position of the person with respect to both cameras as well as sensor noise and calibration errors. The uncertainty in the estimates is used to adaptively estimate the zoom parameter that guarantees with a user specified probability, /spl alpha/, that the detected person's face is contained and zoomed within the image.

Image segmentation has traditionally been thought of us a low/mid-level vision process incorporating no high level constraints. However, in complex and uncontrolled environments, such bottom-up strategies have drawbacks that lead to large misclassification rates. Remedies to this situation include taking into account (1) contextual and application constraints, (2) user input and feedback to incrementally improve the performance of the system. We attempt to incorporate these in the context of pipeline segmentation in industrial images. This problem is of practical importance for the 3D reconstruction of factory environments. However it poses several fundamental challenges mainly due to shading. Highlights and textural variations, etc. Our system performs pipe segmentation by fusing methods from physics-based vision, edge and texture analysis, probabilistic learning and the use of the graph-cut formalism.

A new paradigm for the efficient color-based tracking of objects seen from a moving camera is presented. The proposed technique employs the mean shift analysis to derive the target candidate that is the most similar to a given target model, while the prediction of the next target location is computed with a Kalman filter. The dissimilarity between the target model and the target candidates is expressed by a metric based on the Bhattacharyya coefficient. The implementation of the new method achieves real-time performance, being appropriate for a large variety of objects with different color patterns. The resulting tracking, tested on various sequences, is robust to partial occlusion, significant clutter, target scale variations, rotations in depth, and changes in camera position.

Computer vision algorithms are composed of different sub-algorithms often applied in sequence. Previous work on performance characterization illustrated how random perturbation models can be setup at various stages of an algorithm sequence for the input and output data. In this paper we address the issue of how one could utilize these random perturbation models in order to automate the selection of free parameters used in an algorithm sequence. We consider an operation sequence that involves edge finding, linking, corner finding and matching. Appropriate prior distributions for the parameters that describe the graytone/geometric characteristics of the image features are specified and validated by using an annotation process. The annotation process involves the manual specification (outlining) of the geometry and spatial extent of the image features. Statistics are gathered for parameters describing features of interest and non-interest (clutter features). The appropriate prior distributions are used to derive the theoretical expressions for feature detector performance over a given image population. These performance measures are then optimized to determine the tuning parameters for the feature detector(s).>

A corner is modelled as the intersection of two lines. A corner point is that point on an input digital arc whose a posteriori probability of being a corner is the maximum among all the points on the arc. The performance of the corner detector is characterized by its false alarm rate, misdetection rate, and the corner location error all as a function of the noise variance, the included corner angle, and the arc length. Theoretical expressions for the quantities compare well with experimental results.>

This paper describes a corner detection method that obtains maximum a posteriori estimates for the corner location in a given sequence of points. The authors model an ideal corner as the intersection of two ideal line segments. The perturbations on the sample points in a given line segment are assumed to be i.i.d Gaussian random variables of zero mean and variance /spl sigma//sup 2/. Further, the perturbations on the points are assumed to be orthogonal to the ideal line. The paper discusses the theory of the corner detector and an algorithm that extends the basic theory to handle multilinear segment arcs. Experiments were conducted according to a specific protocol and performance curves showing the location error versus the noise variance, the included corner angle, and the arc length, are provided. Performance characterization of the corner detector is also performed by plotting the false alarm rate and the misdetect rate versus the context window length and included corner angle. It is shown that the experimental results match the theoretical error propagation.

The history of the image understanding environment (IUE) project, a five-year program to develop a common software environment for the development of algorithms and application systems, is reviewed. An overview of some of the data structures that are currently evolving as a specification for the IUE is provided. The ultimate goal of the project is to provide the basic data structures and algorithms that are required to carry state-of-the-art research in image understanding.>

It is shown how random perturbation models can be set up for a vision algorithm sequence involving edge finding, edge linking, and gap filling. By starting with an appropriate noise model for the input data, the authors derive random perturbation models for the output data at each stage of their example sequence. These random perturbation models are useful for performing model-based theoretical comparisons of the performance of vision algorithms. Parameters of these random perturbation models are related to measures of error such as the probability of misdetection of feature units, probability of false alarm, and the probability of incorrect grouping. Since the parameters of the perturbation model at the output of an algorithm are indicators of the performance of the algorithm, one could utilize these models to automate the selection of various free parameters (thresholds) of the algorithm.>