We present a re-ranking algorithm for image retrieval by fusing multi-feature information. We utilize pair wise similarity scores between images to exploit the underlying relationships among images. The initial ranked list for a query from each feature is represented as an undirected graph, where edge strength comes from feature-specific image similarity. Graphs from multiple features are combined by a mixture Markov model. In addition, we utilize a probabilistic model based on the statistics of similarity scores of similar and dissimilar image pairs to determine the weight for each graph. The weight for a feature is query specific, where the ranked lists of different queries receive different weights. Our approach for calculating weights is data-driven and does not require any learning. A diffusion process is then applied to the fused graph to reduce noise and achieve better retrieval performance. Experiments demonstrate that our approach significantly improves performance over baseline methods and outperforms many state-of-the-art retrieval methods.

We present a novel method for matching ground-based query images to a georeferenced LIDAR 3D dataset acquired from an airborne platform in urban environments. We are addressing two main technical challenges: (i) different modalities between the query and the reference data (electro-optical vs. LIDAR) that impose unique challenges to the matching problem; (ii) very different viewing directions from which the query, respectively the LIDAR data were acquired. We make two main technical contributions in this paper. First, we present a method for automatically extracting features from LIDAR data that largely remain invariant to the projection in a 2D image and thus allow robust matching across modalities and change in viewpoint. Second, we describe a matching technique that finds the best 3D pose that relates the query input image to a rendered image of the 3D models. We present results of matching images to high-resolution LIDAR data covering five square kilometers over a city that demonstrate the power of the matching method proposed.

We describe a vehicle tracking algorithm using input from a network of nonoverlapping cameras. Our algorithm is based on a novel statistical formulation that uses joint kinematic and image appearance information to link local tracks of the same vehicles into global tracks with longer persistence. The algorithm can handle significant spatial separation between the cameras and is robust to challenging tracking conditions such as high traffic density, or complex road infrastructure. In these cases, traditional tracking formulations based on MHT, or JPDA algorithms, may fail to produce track associations across cameras due to the weak predictive models employed. We make several new contributions in this paper. Firstly, we model kinematic constraints between any two local tracks using road networks and transit time distributions. The transit time distributions are calculated dynamically as convolutions of normalized transit time distributions that are learned and adapted separately for individual roads. Secondly, we present a complete statistical tracker formulation, which combines kinematic and appearance likelihoods within a multi-hypothesis framework. We have extensively evaluated the algorithm proposed using a network of ground-based cameras with narrow field of view. The tracking results obtained on a large ground-truthed dataset demonstrate the effectiveness of the algorithm proposed.

We present a novel LIDAR streaming architecture for real-time, on-board processing using unmanned robots. We propose a two-level 3D data structure that allows pipelined and streaming processing of the 3D data as it arrives from a moving robot: (i) at the coarse level, the incoming 3D scans are stored in memory in a dense 3D voxel grid with a relatively large voxel size - this ensures buffering of the most recent data and the availability of sufficient 3D measurements within a specific processing volume at the next level; (ii) at the fine level, we employ a data chunking mechanism guided by the movement of the robot and a rolling dense 3D voxel grid for processing the data in the immediate vicinity of the robot, which enables reuse of previously computed features. The architecture proposed requires a very small memory footprint, minimal data copying, and allows a fast spatial access for 3D data, even at the finest resolutions. We illustrate the proposed streaming architecture on a real-time 3D structure characterization task for detecting doors and stairs in indoor environments and show qualitative results demonstrating the effectiveness of our approach.

We present a real-time pedestrian detection system based on structure and appearance classification. We discuss several novel ideas that contribute to having low-false alarms and high detection rates, while at the same time achieving computational efficiency: (i) At the front end of our system we employ stereo to detect pedestrians in 3D range maps using template matching with a representative 3D shape model, and to classify other background objects in the scene such as buildings, trees and poles. The structure classification efficiently labels substantial amount of non-relevant image regions and guides the further computationally expensive process to focus on relatively small image parts; (ii)We improve the appearance-based classifiers based on HoG descriptors by performing template matching with 2D human shape contour fragments that results in improved localization and accuracy; (iii) We build a suite of classifiers tuned to specific distance ranges for optimized system performance. Our method is evaluated on publicly available datasets and is shown to match or exceed the performance of leading pedestrian detectors in terms of accuracy as well as achieving real-time computation (10 Hz), which makes it adequate for in-vehicle navigation platform.

We present a novel building segmentation system for densely built areas, containing thousands of buildings per square kilometer. We employ solely sparse LIDAR (Light/Laser Detection Ranging) 3D data, captured from an aerial platform, with resolution less than one point per square meter. The goal of our work is to create segmented and delineated buildings as well as structures on top of buildings without requiring scanning for the sides of buildings. Building segmentation is a critical component in many applications such as 3D visualization, robot navigation and cartography. LIDAR has emerged in recent years as a more robust alternative to 2D imagery because it acquires 3D structure directly, without the shortcomings of stereo in un- textured regions and at depth discontinuities. Our main technical contributions in this paper are: (i) a ground segmentation algorithm which can handle both rural regions, and heavily urbanized areas, where the ground is 20% or less of the data, (ii) a building segmentation technique, which is robust to buildings in close proximity to each other, sparse measurements and nearby structured vegetation clutter, and (Hi) an algorithm for estimating the orientation of a boundary contour of a building, based on minimizing the number of vertices in a rectilinear approximation to the building outline, which can cope with significant quantization noise in the outline measurements. We have applied the proposed building segmentation system to several urban regions with areas of hundreds of square kilometers each, obtaining average segmentation speeds of less than three minutes per km2on a standard Pentium processor. Extensive qualitative results obtained by overlaying the 3D segmented regions onto 2D imagery indicate accurate performance of our system.

In an errors-in-variables (EIV) model, all the measurements are corrupted by noise. The class of EIV models with constraints separable into the product of two nonlinear functions, one solely in the variables and one solely in the parameters, is general enough to represent most computer vision problems. We show that the estimation of such nonlinear EIV models can be reduced to iteratively estimating a linear model having point dependent, i.e., heteroscedastic, noise process. Particular cases of the proposed heteroscedastic errors-in-variables (HEIV) estimator are related to other techniques described in the vision literature: the Sampson method, renormalization, and the fundamental numerical scheme. In a wide variety of tasks, the HEIV estimator exhibits the same, or superior, performance as these techniques and has a weaker dependence on the quality of the initial solution than the Levenberg-Marquardt method, the standard approach toward estimating nonlinear models.

We propose a new method for rapid 3D object indexing that combines feature-based methods with coarse alignment-based matching techniques. Our approach achieves a sublinear complexity on the number of models, maintaining at the same time a high degree of performance for real 3D sensed data that is acquired in largely uncontrolled settings. The key component of our method is to first index surface descriptors computed at salient locations from the scene into the whole model database using the Locality Sensitive Hashing (LSH), a probabilistic approximate nearest neighbor method. Progressively complex geometric constraints are subsequently enforced to further prune the initial candidates and eliminate false correspondences due to inaccuracies in the surface descriptors and the errors of the LSH algorithm. The indexed models are selected based on the MAP rule using posterior probability of the models estimated in the joint 3D-signature space. Experiments with real 3D data employing a large database of vehicles, most of them very similar in shape, containing 1,000,000 features from more than 365 models demonstrate a high degree of performance in the presence of occlusion and obscuration, unmodeled vehicle interiors and part articulations, with an average processing time between 50 and 100 seconds per query.

Histograms of shape signature or prototypical shapes, called shapemes, have been used effectively in previous work for 2D/3D shape matching and recognition. We extend the idea of shapeme histogram to recognize partially observed query objects from a database of complete model objects. We propose representing each model object as a collection of shapeme histograms and match the query histogram to this representation in two steps: 1) compute a constrained projection of the query histogram onto the subspace spanned by all the shapeme histograms of the model and 2) compute a match measure between the query histogram and the projection. The first step is formulated as a constrained optimization problem that is solved by a sampling algorithm. The second step is formulated under a Bayesian framework, where an implicit feature selection process is conducted to improve the discrimination capability of shapeme histograms. Results of matching partially viewed range objects with a 243 model database demonstrate better performance than the original shapeme histogram matching algorithm and other approaches.

A robust, multi-frame, progressive refinement framework for registering narrow field of view video to reference imagery is presented. A major strength of the approach is its effectiveness in the presence of dissimilar video and reference image appearance. Normalized oriented energy image pyramids are employed to enable alignment of images with global visual dissimilarities, yet local feature commonality. Local matching is then applied coarse-to-fine, along four dimensions: spatial frequency, local support, search range, and model order (a robust parametric model fit is used to reject outliers at each iteration). Globally optimal multi-frame alignment is obtained with respect to several constraints: frame-to-reference local matches, recovered frame-to-frame motion, and optional a priori estimates of sensor pose. The framework is described in detail and applied to two examples: aerial video to geographic reference image alignment (georegistration) and retinal slit lamp video to fundus image alignment.

Reliable estimation of the trifocal tensor is crucial for 3D reconstruction from uncalibrated cameras. The estimation process is based on minimizing the geometric distances between the measurements and the corrected data points, the underlying nonlinear optimization problem being most often solved with the Levenberg-Marquardt (LM) algorithm. We employ for this task the heteroscedastic errors-in-variables (HEIV) estimator and take into account both the singularity of the multivariate tensor constraint and the bifurcation which can appear for noisy data. In comparison to the Gold Standard method, the new approach is significantly faster while having the same performance, and it is less sensitive to initialization when the data is close to degenerate. Analytical expressions for the covariances of the parameter and corrected image point estimates are available for the HEIV estimator and thus the confidence regions of the corrected measurements can be delineated in the images.

An algorithm is presented for video georegistration, with a particular concern for aerial video, i.e., video captured from an airborne platform. The algorithm's input is a video stream with telemetry (camera model specification sufficient to define an initial estimate of the view) and geodetically calibrated reference imagery (coaligned digital orthoimage and elevation map). The output is a spatial registration of the video to the reference so that it inherits the available geodetic coordinates. The video is processed in a continuous fashion to yield a corresponding stream of georegistered results. Quantitative results of evaluating the developed approach with real world aerial video also are presented. The results suggest that the developed approach may provide valuable input to the analysis and interpretation of aerial video.

The Errors-in-Variables (EIV) model from statistics is often employed in computer vision though only rarely under this name. In an EIV model all the measurements are corrupted by noise while the a priori information is captured with a nonlinear constraint among the true (unknown) values of these measurements. To estimate the model parameters and the uncorrupted data, the constraint can be linearized, i.e., embedded in a higher dimensional space. We show that linearization introduces data-dependent (heteroscedastic) noise and propose an iterative procedure, the heteroscedastic EIV (HEIV) estimator to obtain consistent estimates in the most general, multivariate case. Analytical expressions for the covariances of the parameter estimates and corrected data points, a generic method for the enforcement of ancillary constraints arising from the underlying geometry are also given. The HEIV estimator minimizes the first order approximation of the geometric distances between the measurements and the true data points, and thus can be a substitute for the widely used Levenberg-Marquardt based direct solution of the original nonlinear problem. The HEIV estimator has however the advantage of a weaker dependence on the initial solution and a faster convergence. In comparison to Kanatani's renormalization paradigm (an earlier solution of the same problem) the HEIV estimator has more solid theoretical foundations which translate into better numerical behavior We show that the HEIV estimator can provide an accurate solution to most 3D vision estimation tasks, and illustrate its performance through two case studies: calibration and the estimation of the fundamental matrix.

An improved maximum likelihood estimator for ellipse fitting based on the heteroscedastic errors-in-variables (HEIV) regression algorithm is proposed. The technique significantly reduces the bias of the parameter estimates present in the direct least squares method, while it is numerically more robust than renormalization, and requires less computations than minimizing the geometric distance with the Levenberg-Marquardt optimization procedure. The HEIV algorithm also provides closed-form expressions for the covariances of the ellipse parameters and corrected data points. The quality of the different solutions is assessed by defining confidence regions in the input domain, either analytically or by bootstrap. The latter approach is exclusively data driven and it is used whenever the expression of the covariance for the estimates is not available.

A new method for 3D rigid motion estimation is derived under the most general assumption that the measurements are corrupted by inhomogeneous and anisotropic, i.e., heteroscedastic noise. This is the case, for example, when the motion of a calibrated stereo-head is to be determined from image pairs. Linearization in the quaternion space transforms the problem into a multivariate, heteroscedastic errors-in-variables (HEIV) regression, from which the rotation and translation estimates are obtained simultaneously. The significant performance improvement is illustrated, for real data, by comparison with the results of quaternion, subspace and renormalization based approaches described in the literature. Extensive use as made of bootstrap, an advanced numerical tool from statistics, both to estimate the covariances of the 3D data points and to obtain confidence regions for the rotation and translation estimates. Bootstrap enables an accurate recovery of these information using only the two image pairs serving as input.