We investigate combinations of 3D-2D matches of points and lines for a single calibrated camera that imply a nontrivial constraint on the camera’s position. Specifically, we address the minimal cases for which n point matches and m line matches constrain the camera center to lie on an algebraic surface in $$\mathbb {R}^3$$ . For two points $$(m,n) = (0,2)$$ , the constraint is well-known to be a self-intersecting torus. We complete the classification problem by deriving explicit, general equations for the cases $$(m,n) \in \{ (2,0), (1, 1) \}$$ for two lines and one point, one line, respectively. We also report preliminary experiments investigating the suitability of these constraints for pose estimation tasks.

Abstract Discriminating salient moving objects against complex, cluttered backgrounds, with occlusions and challenging environmental conditions like weather and illumination, is essential for stateful scene perception in autonomous systems. We propose a novel deep architecture, named DeepFTSG, for robust moving object detection that incorporates single and multi-stream multi-channel USE-Net trellis asymmetric encoders extending U-Net with squeeze and excitation (SE) blocks and a single shared decoder network for fusing multiple motion and appearance cues. DeepFTSG is a deep learning based approach that builds upon our previous hand-engineered flux tensor split Gaussian (FTSG) change detection video analysis algorithm which won the CDNet CVPR Change Detection Workshop challenge competition. DeepFTSG generalizes much better than top-performing motion detection deep networks, such as the scene-dependent ensemble-based FgSegNet_v2, while using an order of magnitude fewer weights. Short-term motion and longer-term change cues are estimated using general-purpose unsupervised methods—flux tensor and multi-modal background subtraction, respectively. DeepFTSG was evaluated using the CDnet-2014 change detection challenge dataset, the largest change detection video sequence benchmark with 12.3 billion labeled pixels, and had an overall F-measure of 97%. We also evaluated the cross-dataset generalization capability of DeepFTSG trained solely on CDnet-2014 short video segments and then evaluated on unseen SBI-2015, LASIESTA and LaSOT benchmark videos. On the unseen SBI-2015 dataset, DeepFTSG had an F-measure accuracy of 87%, more than 30% higher compared to the top-performing deep network FgSegNet_v2 and outperforms the recently proposed KimHa method by 17%. On the unseen LASIESTA, DeepFTSG had an F-measure of 88% and outperformed the best recent deep learning method BSUV-Net2.0 by 3%. On the unseen LaSOT with axis-aligned bounding box ground-truth, network segmentation masks were converted to bounding boxes for evaluation, DeepFTSG had an F-Measure of 55%, outperforming KimHa method by 14% and FgSegNet_v2 by almost 1.5%. When a customized single DeepFTSG model is trained in a scene-dependent manner for comparison with state-of-the-art approaches, then DeepFTSG performs significantly better, reaching an F-Measure of 97% on SBI-2015 (+ 10%) and 99% on LASIESTA (+ 11%). The source code, pre-trained weights, and video demo for DeepFTSG are available at https://github.com/CIVA-Lab/DeepFTSG .

We introduce a novel discrete cosine transform-based feature (DCTF) descriptor designed for both robustly matching features in aerial video and tracking features across wide-baseline oblique views in aerial wide area motion imagery (WAMI). Our DCTF descriptor preserves local structure more compactly in the frequency domain by utilizing the mathematical properties of the discrete cosine transform (DCT) and outperforms widely used the spatial-domain feature extraction methods, such as speeded up robust features (SURF) and scale-invariant feature transform (SIFT). The DCTF descriptor can be used in combination with other feature detectors, such as SURF and features from accelerated segment test (FAST), for which we provide experimental results. The performance of DCTF for image matching and feature tracking is evaluated on two city-scale aerial WAMI data sets (ABQ-215 and LA-351) and a synthetic aerial drone video data set digital imaging and remote sensing image generation (Rochester Institute of Technology (RIT)-DIRSIG). DCTF is a compact 120-D descriptor that is less than half the dimensionality of state-of-the-art deep learning-based approaches, such as SuperPoint, LF-Net, and DeepCompare, which requires no learning and is domain-independent. Despite its small size, the DCTF descriptor surprisingly produces the highest image matching accuracies ( F1= 0.76 and ABQ-215), the longest maximum and average feature track lengths, and the lowest tracking error (0.3 pixel, LA-351) compared with both handcrafted and deep learning features.

Detection of moving objects is a critical component of many computer vision tasks. Recently, deep learning architectures have been developed for supervised learning based moving object change detection. Some top performing architectures, like FgSegNet are single frame spatial appearance cue-based detection and tend to overfit to the training videos. We propose a novel compact multi-cue autoencoder deep architecture, Motion U-Net (MU-Net) for robust moving object detection that generalizes much better than FgSegNet and requires nearly 30 times fewer weight parameters. Motion and change cues are estimated using a multi-modal background subtraction module combined with flux tensor motion estimation. MU-Net was trained and evaluated on the CDnet-2014 change detection challenge video sequences and had an overall F-measure of 0.9369. We used the unseen SBI-2015 video dataset to assess generalization capacity where MU-Net had an F-measure of 0.7625 while FgSegNet_v2 was 0.3519, less than half the MU-Net accuracy. The source code of the Motion U-Net is available at https://github.com/CIVA-Lab/Motion-U-Net.

Regularization methods are used widely in image selective smoothing and edge preserving restoration of noisy images. Traditional methods utilize image gradients within regularization function for controlling the smoothing and can produce artifacts when noise levels are higher. In this paper, we consider a robust image adaptive exponent driven regularization for filtering noisy images with salient feature preservation. Our spatially adaptive variable exponent function depends on a continuous switch based on the eigenvalues of structure tensor which identifies noisy edges, and corners with higher accuracy. Structure tensor eigenvalues encode various image features and we consider a spatially varying continuous map which provides multiscale edge maps of natural images. By embedding the structure tensor-based exponent in a well-defined regularization model, we obtain denoising filters which are capable of obtaining good feature preserving image restoration. The GPU-based implementation computes the edge map in real time at 45-60 frames/s depending on the GPU card. Multiscale structure tensor-based spatially adaptive variable exponent provides reliable edge maps and compared with standard edge detectors it is robust under various noisy conditions. Moreover, filtering based on the multiscale variable exponent map method outperforms L0 sparse gradient-based image smoothing and related filters.

This paper presents, HitGraph, an FPGA framework to accelerate graph processing based on the edge-centric paradigm. HitGraph takes in an edge-centric graph algorithm and hardware resource constraints, determines design parameters and then generates a Register Transfer Level (RTL) FPGA design. This makes accelerator design for various graph analytics transparent and user-friendly by masking internal details of the accelerator design process. HitGraph enables increased data reuse and parallelism through novel algorithmic optimizations, including (1) an optimized data layout that reduces non-sequential external memory accesses, (2) an efficient update merging and filtering scheme to reduce the data communication between the FPGA and external memory, and (3) a partition skipping scheme to reduce redundant edge traversals for non-stationary graph algorithms. Based on our design methodology, we accelerate Sparse Matrix Vector Multiplication (SpMV), PageRank (PR), Single Source Shortest Path (SSSP), and Weakly Connected Component (WCC). Experimental results show that HitGraph sustains a high throughput of 2076 Million Traversed Edges Per Second (MTEPS) for SpMV, 2225 MTEPS for PR, 2916 MTEPS for SSSP, and 3493 MTEPS for WCC, respectively. Compared with highly-optimized multi-core implementations, HitGraph achieves up to 37.9× speedup. Compared with state-of-the-art FPGA frameworks, HitGraph achieves up to 50.7× throughput improvement.

Multi-object tracking is a fundamental computer vision task with a wide variety of real-life applications ranging from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging task due to complications caused by object appearance changes, complex object dynamics, clutter in the environment, and partial or full occlusions. In this paper, we propose a time-efficient detection-based multi-object tracking system using a three-step cascaded data association scheme that combines a fast spatial distance only short-term data association, a robust tracklet linking step using discriminative object appearance models, and an explicit occlusion handling unit relying not only on tracked objects' motion patterns but also on environmental constraints such as presence of potential occlud-ers in the scene. Our experiments on UA-DETRAC multi-object tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

A desirable smart traffic-monitoring and street-safety system can elicit and support the intervention of law enforcement agencies or medical staff. Recently, there has been a dramatically higher demand for such smart systems. To this end, the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S) was organized in conjunction with the 15th IEEE International Conference on Advanced Video and Signal-based Surveillance (AVSS 2018). Our goal is to advance the state-of-the-art detection and tracking algorithms and provide a comprehensive performance evaluation for them. We evaluate 5 submitted detection and 7 submitted tracking methods on the large-scale UA-DETRAC benchmark, and the results are shared publicly on the website http://detrac-db. rit.albany.edu. We expect this challenge to advance the research and development of new detection and tracking methods for transportation applications.

Automatic image classification systems for indirect immunofluorescence (IIF) labeling of human epithelial (HEp-2) cell specimens are needed to improve the efficient management of autoimmune diseases. In this paper, we propose to classify HEp-2 cell specimen imagery using a combination of local features and deep learning features extracted from the IIF images. Two local descriptors are used to capture texture information, namely: Rotation Invariant Co-occurrence among Local Binary Patterns (RIC-LBP) extending the LBP descriptor and Joint Motif Labels (JML) based on the Peano scan motif concept. Deep learning features are then extracted using the VGG-19 image classification network. Finally, all descriptors are combined using a late fusion approach with a Random Forests (RF) classifier with seven output classes. Experimental results show that our proposed framework achieves a mean class accuracy of 92.11% with five-fold cross validation using the RF classifier with 1000 trees on the HEp-2 specimen benchmark dataset, which outperforms the state-of-the-art accuracy on this dataset.

Inevitable camera motion during exposure does not augur well for free-hand photography. Distortions introduced in images can be of different types and mainly depend on the structure of the scene, the nature of camera motion, and the shutter mechanism of the camera. In this paper, we address the problem of registering images taken from global shutter and rolling shutter cameras and reveal the constraints on camera motion that admit registration, change detection, and rectification. Our analysis encompasses degradations arising from camera motion during exposure and differences in shutter mechanisms. We also investigate conditions under which camera motions causing distortions in reference and target image can be decoupled to yield the underlying latent image through RS rectification. We validate our approach using several synthetic and real examples.

Multiview learning has shown promising potential in many applications. However, most techniques are focused on either view consistency, or view diversity. In this paper, we introduce a novel multiview boosting algorithm, called Boost.SH, that computes weak classifiers independently of each view but uses a shared weight distribution to propagate information among the multiple views to ensure consistency. To encourage diversity, we introduce randomized Boost.SH and show its convergence to the greedy Boost.SH solution in the sense of minimizing regret using the framework of adversarial multiarmed bandits. We also introduce a variant of Boost.SH that combines decisions from multiple experts for recommending views for classification. We propose an expert strategy for multiview learning based on inverse variance, which explores both consistency and diversity. Experiments on biometric recognition, document categorization, multilingual text, and yeast genomic multiview data sets demonstrate the advantage of Boost.SH (85%) compared with other boosting algorithms like AdaBoost (82%) using concatenated views and substantially better than a multiview kernel learning algorithm (74%).

Multi-object tracking is an important computer vision task with wide variety of real-life applications from surveillance and monitoring to biomedical video analysis. Multi-object tracking is a challenging problem due to complications such as partial or full occlusions, factors affecting object appearance, object interaction dynamics, etc. and computational cost. In this paper, we propose a detection-based multi-object tracking system that uses a two-step data association scheme to ensure time efficiency while preserving tracking accuracy; a robust but discriminative object appearance model that compares object color attributes using a novel color correlation cost matrix; and a framework that handles occlusions through prediction. Our experiments on UA-DETRAC multi-object tracking benchmark dataset consisting of challenging real-world traffic videos show promising results against state-of-the-art trackers.

Persistent detection and tracking of moving vehicles in airborne imagery provide indispensable information for many traffic surveillance applications including traffic monitoring and management, navigation systems, activity recognition and event detection. This paper presents a collaborative Spatial Pyramid Context-aware detection and Tracking system (SPCT) for moving vehicles in dense urban aerial imagery. The proposed system is composed of one master tracker that usually relies on visual object features and two auxiliary trackers based on object temporal motion information that will be called dynamically to assist master tracker. SPCT utilizes image spatial context at different level to make the video tracking system resistant to occlusion, background noise and improve target localization accuracy. We chose a pre-selected seven-channel complementary features including RGB color, intensity and spatial pyramid of HoG (PHoG) and exploit integral histogram as building block to meet the demands of real-time performance. The extensive experiments on ARGUS and ABQ wide aerial video and comparison with state-of-the-art single object trackers confirm that combining complementary tracking cues in an intelligent fusion framework is essential to address the challenges of persistent tracking in low frame rate Wide Aerial Motion Imagery (WAMI).

In this paper, we address the problem of registering a distorted image and a reference image of the same scene by estimating the camera motion that had caused the distortion. We simultaneously detect the regions of changes between the two images. We attend to the coalesced effect of rolling shutter and motion blur that occurs frequently in moving CMOS cameras. We first model a general image formation framework for a 3D scene following a layered approach in the presence of rolling shutter and motion blur. We then develop an algorithm which performs layered registration to detect changes. This algorithm includes an optimisation problem that leverages the sparsity of the camera trajectory in the pose space and the sparsity of changes in the spatial domain. We create a synthetic dataset for change detection in the presence of motion blur and rolling shutter effect covering different types of camera motion for both planar and 3D scenes. We compare our method with existing registration methods and also show several real examples captured with CMOS cameras.

In the event of natural or man-made disasters, providing rapid situational awareness through video/image data collected at salient incident scenes is often critical to the first responders. However, computer vision techniques that can process the media-rich and data-intensive content obtained from civilian smartphones or surveillance cameras require large amounts of computational resources or ancillary data sources that may not be available at the geographical location of the incident. In this paper, we propose an incident-supporting visual cloud computing solution by defining a collection, computation, and consumption (3C) architecture supporting fog computing at the network edge close to the collection/consumption sites, which is coupled with cloud offloading to a core computation, utilizing software-defined networking (SDN). We evaluate our 3C architecture and algorithms using realistic virtual environment test beds. We also describe our insights in preparing the cloud provisioning and thin-client desktop fogs to handle the elasticity and user mobility demands in a theater-scale application. In addition, we demonstrate the use of SDN for on-demand compute offload with congestion-avoiding traffic steering to enhance remote user quality of experience in a regional-scale application. The optimization between fogs computing at the network edge with core cloud computing for managing visual analytics reduces latency, congestion, and increases throughput.

We describe a fast and efficient camera pose refinement and Structure from Motion (SfM) method for sequential aerial imagery with applications to georegistration and 3-D reconstruction. Inputs to the system are 2-D images combined with initial noisy camera metadata measurements, available from on-board sensors (e.g., camera, global positioning system, and inertial measurement unit). Georegistration is required to stabilize the ground-plane motion to separate camera-induced motion from object motion to support vehicle tracking in aerial imagery. In the proposed approach, we recover accurate camera pose and (sparse) 3-D structure using bundle adjustment for sequential imagery (BA4S) and then stabilize the video from the moving platform by analytically solving for the image-plane-to-ground-plane homography transformation. Using this approach, we avoid relying upon image-to-image registration, which requires estimating feature correspondences (i.e., matching) followed by warping between images (in a 2-D space) that is an error prone process for complex scenes with parallax, appearance, and illumination changes. Both our SfM (BA4S) and our analytical ground-plane georegistration method avoid the use of iterative consensus combinatorial methods like RANdom SAmple Consensus which is a core part of many published approaches. BA4S is very efficient for long sequential imagery and is more than 130 times faster than VisualSfM, 35 times faster than MavMap, and about 274 times faster than Pix4D. Various experimental results demonstrate the efficiency and robustness of the proposed pipeline for the refinement of camera parameters in sequential aerial imagery and georegistration.

Reducing the amount of user driven input for interactive image segmentation enables faster and more precise foreground extraction of objects. A sparse collection of labeled seed points sampled over image regions can be quickly provided by the user using a few mouse clicks. Seed points are used for training an Elastic Body Spline classifier mapping function. We evaluate the efficiency and accuracy of user defined point inputs compared to fully manual boundary drawing that can be time consuming and automatic image segmentation methods that may not have sufficient accuracy. We show that using an average of just 8 labeled pixels (i.e. sparse set of seed points) the proposed EBS foreground-background thresholding method can achieve 90 percent accuracy compared to manual ground truth on the Berkeley BSDS500 benchmark.

Capturing multiple images using the burst mode of handheld cameras can be a boon to obtain a high resolution (HR) image by exploiting the subpixel motion among the captured images arising from handshake. However, the caveat with mobile phone cameras is that they produce rolling shutter (RS) distortions that must be accounted for in the super-resolution process. We propose a method in which we obtain an RS-free HR image using HR camera trajectory estimated by leveraging the intra- and inter-frame continuity of the camera motion. Experimental evaluations demonstrate that our approach can effectively recover a super-resolved image free from RS artifacts.

Most Wide Area Motion Imagery (WAMI) based trackers use motion based cueing for detecting and tracking moving objects. The results are very high false alarm rates in urban environments with tall structures due to parallax effects. This paper proposes an accurate moving object detection method using a precise orthorectification approach for ground stabilization combined with accurate multiview depth maps to reduce the number of false positives induced by parallax effects by 90 percent. Proposed hybrid moving vehicle detection approach for large scale aerial urban imagery is based on fusion of motion detection mask obtained from median-based background subtraction and tall structures height mask provided by image depth map information. Using buildings mask enables us to improve the object level detection accuracy in terms of F-measure by 57 percent from 22.2% to 79.2%.

Human epithelial (HEp-2) cell specimens are obtained from indirect immunofluorescence (IIF) imaging for diagnosis and management of autoimmune diseases. Analysis of HEp2 cells is important and in this work we consider automatic cell segmentation and classification using spatial and texture pattern features and random forest classifiers. In this paper, we summarize our efforts in classification and segmentation tasks proposed in ICPR 2016 contest. For the cell level staining pattern classification (Task 1), we utilized texture features such as rotational invariant co-occurrence (RIC) versions of the well-known local binary pattern (LBP), median binary pattern (MBP), joint adaptive median binary pattern (JAMBP), and motif labels (ML) along with other optimized features. We report the classification results utilizing different classifiers such as the k-nearest neighbors (kNN), support vector machine (SVM), and random forest (RF). We obtained the best accuracy of 94.26% for six cell classes with RIC-LBP combined with a motif pattern co-occurrence labels (MCL). For specimen level staining pattern classification (Task 2) we utilize a combination RIC-LBP with RF classifier and obtain 80% accuracy for seven classes. For cell segmentation (Task 4), we use our optimized multiscale spatial feature bank along with RF classifier for pixel-wise labeling to achieve an F-measure of 84.26% for 1008 images.

The importance of the internet-of-things (IOT) is now an established reality. With that backdrop, the phenomenal emergence of cameras/sensors mounted on unmanned aerial, ground and marine vehicles (UAVs, UGVs, UMVs) and body worn cameras is a notable new development. The swarms of cameras and real-time computing thereof are at the heart of new technologies like connected cars, drone-based city-wide surveillance and precision agriculture, etc. Smart computer vision algorithms (with or without dynamic learning) that enable object recognition and tracking, supported by baseline video content summarization or 2D/3D image reconstruction of the scanned environment are at the heart of such new applications. In this article, we summarize our recent innovations in this space. We focus primarily on algorithms and architectural design considerations for video summarization systems.

We propose a framework for robust estimation of normal of a planar scene from a single motion blurred observation. We first reveal how feature points can be extracted from blur kernels and matched to generate several point correspondences. Although these points correspond to different homographies, the fact that they conform to the same normal yields a rank-3 constraint which we harness within a hierarchical clustering framework to estimate the normal accurately.

Elastic body splines (EBS) belong to a family of splines introduced for biomedical image registration. EBS models the elastic deformation of a homogeneous isotropic elastic body subjected to external forces. The task of interactive image segmentation is framed as a semi-supervised interpolation where the basis functions are learned using the user provided seed points to model and predict the labels for the unlabeled pixels. Seed points are sparse compared to other methods that may require scribbles and regions. The spline for interpolating labels is the EBS which we compare to our previous work using Gaussian Elastic Body splines (GEBS) [1] for the task of interactive image segmentation. Experimental results show that the EBS is about 14 percent better, in terms of accuracy, than GEBS and significantly better than random walk and graph cut based segmentation. EBS is also 2.5 times faster than GEBS.

It is estimated that less than five percent of videos are currently analyzed to any degree. In addition to petabyte-sized multimedia archives, continuing innovations in optics, imaging sensors, camera arrays, (aerial) platforms, and storage technologies indicates that for the foreseeable future existing and new applications will continue to generate enormous volumes of video imagery. Contextual video summarizations and activity maps offers one innovative direction to tackling this Big Data problem in computer vision. The goal of this work is to develop semi-automatic exploitation algorithms and tools to increase utility, dissemination and usage potential by providing quick dynamic overview geospatial mosaics and motion maps. We present a framework to summarize (multiple) video streams from unmanned aerial vehicles (UAV) or drones which have very different characteristics compared to structured commercial and consumer videos that have been analyzed in the past. Using both metadata geospatial characteristics of the video combined with fast low-level image-based algorithms, the proposed method first generates mini-mosaics that can then be combined into geo-referenced meta-mosaics imagery. These geospatial maps enable rapid assessment of hours long videos with arbitrary spatial coverage from multiple sensors by generating quick look imagery, composed of multiple mini-mosaics, summarizing spatiotemporal dynamics such as coverage, dwell time, activity, etc. The overall summarization pipeline was tested on several DARPA Video and Image Retrieval and Analysis Tool (VIRAT) datasets. We evaluate the effectiveness of the proposed video summarization framework using metrics such as compression and hours of viewing time.

A robust camera pose refinement approach for sequential wide-area airborne imagery is proposed in this letter. Image frames are sequentially acquired, and with each frame, its corresponding position and orientation are approximately available from airborne platform inertial measurement unit and GPS sensors. In the proposed structure from motion (SfM) approach the available approximation of camera parameters (from low-certainty sensors) is directly used in an optimization stage. The putative matches obtained from the sequential matching paradigm are also directly used in the optimization with no early stage filtering (e.g., no RANSAC). A robust function is proposed and used to deal with outliers (mismatches). The full pipeline has been run over a set of wide-area motion imagery data collected by an airplane flying over different cities in the U.S. The results prove the power and efficiency of the proposed pipeline. Effectiveness of the proposed robust function is compared with some popular robust functions such as Cauchy and Huber using synthetic data.

PC-TRIO is an indexed TCAM architecture for packet classification. In addition to index TCAMs, PC-TRIO uses wide SRAM words. On our packet classifier data sets, PC-TRIO reduced TCAM power by 96 percent and lookup time by 98 percent on an average, compared to PC-DUOS+[28]that does not use indexing or wide SRAMs. PC-DUOS+ was shown to be better than STCAM, which is a single TCAM architecture conventionally used for packet classification[28]. In this paper, we also extend PC-DUOS+ by augmenting it with wide SRAMs and index TCAMs using the same methodology as used in PC-TRIO, to obtain PC-DUOS+W. On ACL data sets, PC-DUOS+W reduced TCAM power by 86 percent and lookup time by 98 percent, compared to PC-DUOS+, which demonstrates the effectiveness of indexing and usage of wide SRAMs in reducing power and lookup time for packet classifiers.

Edge preserving regularization using partial differential equation (PDE)-based methods although extensively studied and widely used for image restoration, still have limitations in adapting to local structures. We propose a spatially adaptive multiscale variable exponent-based anisotropic variational PDE method that overcomes current shortcomings, such as over smoothing and staircasing artifacts, while still retaining and enhancing edge structures across scale. Our innovative model automatically balances between Tikhonov and total variation (TV) regularization effects using scene content information by incorporating a spatially varying edge coherence exponent map constructed using the eigenvalues of the filtered structure tensor. The multiscale exponent model we develop leads to a novel restoration method that preserves edges better and provides selective denoising without generating artifacts for both additive and multiplicative noise models. Mathematical analysis of our proposed method in variable exponent space establishes the existence of a minimizer and its properties. The discretization method we use satisfies the maximum-minimum principle which guarantees that artificial edge regions are not created. Extensive experimental results using synthetic, and natural images indicate that the proposed multiscale Tikhonov-TV (MTTV) and dynamical MTTV methods perform better than many contemporary denoising algorithms in terms of several metrics, including signal-to-noise ratio improvement and structure preservation. Promising extensions to handle multiplicative noise models and multichannel imagery are also discussed.

Elastic body splines (EBS) belonging to the family of 3D splines were recently introduced to capture tissue deformations within a physical model-based approach for non-rigid biomedical image registration [1]. EBS model the displacement of points in a 3D homogeneous isotropic elastic body subject to forces. We propose a novel extension of using elastic body splines for learning driven figure-ground segmentation. The task of interactive image segmentation, with user provided foreground-background labeled seeds or samples, is formulated as learning an interpolating pixel classification function that is then used to assign labels for all unlabeled pixels in the image. The spline function we chose to model the supervised pixel classifier is the Gaussian elastic body spline (GEBS) which can use sparse scribbles from the user and has a closed form solution enabling a fast on-line implementation. Experimental results demonstrate the applicability of the GEBS approach for image segmentation. The GEBS method for interactive foreground image labeling shows promise and outperforms a previous approach using the thin-plate spline model.

This paper addresses the challenging problem of recognition and classification of textured surfaces under illumination variation, geometric transformations and noisy sensor measurements. We propose a new texture operator, Adaptive Median Binary Patterns (AMBP) that extends our previous Median Binary Patterns (MBP) texture feature. The principal idea of AMBP is to hash small local image patches into a binary pattern text on by fusing MBP and Local Binary Patterns (LBP) operators combined with using self-adaptive analysis window sizes to better capture invariant microstructure information while providing robustness to noise. The AMBP scheme is shown to be an effective mechanism for non-parametric learning of spatially varying image texture statistics. The local distribution of rotation invariant and uniform binary pattern subsets extended with more global joint information are used as the descriptors for robust texture classification. The AMBP is shown to outperform recent binary pattern and filtering-based texture analysis methods on two large texture corpora (CUReT and KTH_TIPS2-b) with and without additive noise. The AMBP method is slightly superior to the best techniques in the noiseless case but significantly outperforms other methods in the presence of impulse noise.

We present a scheme for recovering the orientation of a planar scene from a single translation ally-motion blurred image. By leveraging the homography relationship among image coordinates of 3D points lying on a plane, and by exploiting natural correspondences among the extremities of the blur kernels derived from the motion blurred observation, the proposed method can accurately infer the normal of the planar surface. We validate our approach on synthetic as well as real planar scenes.

We propose algorithms for distributing the classifier rules to two ternary content addressable memories (TCAMs) and for incrementally updating the TCAMs. The performance of our scheme is compared against the prevalent scheme of storing classifier rules in a single TCAM in priority order. Our scheme results in an improvement in average lookup speed by up to 49% and an improvement in update performance by up to 3.84 times in terms of the number of TCAM writes.

The extensive availability of sophisticated image editing tools has rendered it relatively easy to produce fake images. Image splicing is a form of tampering in which an original image is altered by copying a portion from a different source. Because the phenomenon of motion blur is a common occurrence in hand-held cameras, we propose a passive method to automatically detect image splicing using blur as a cue. Specifically, we address the scenario of a static scene in which the cause of blur is due to hand shake. Existing methods for dealing with this problem work only in the presence of uniform space-invariant blur. In contrast, our method can expose the presence of splicing by evaluating inconsistencies in motion blur even under space-variant blurring situations. We validate our method on several examples for different scene situations and camera motions of interest.

We address the problem of automatically detecting occluded regions given a blurred/unblurred image pair of a scene taken from different viewpoints. The occlusion can be due to single or multiple objects. We present a unified framework for detecting occluder(s) that is reasonably robust to non-uniform motion blur as well as variations in camera pose (without the need for deblurring). We assume that the occluded pixels occupy only a relatively small area and that the camera motion trajectory is sparse in the camera motion space. We validate the performance of our algorithm with experiments on synthetic and real data.

Existing single image defogging techniques can restore contrast loss and yield a rough estimate of the depth map of a scene. The ubiquity of hand-held imaging devices has attracted considerable attention to motion blur but this has not been addressed in the context of images captured under foggy conditions. In this paper, we show how to restore foggy motion-blurred images using depth cues derived from fog itself. Initially, we address restoration of images blurred primarily due to in-plane translational camera motion. This is followed by a scheme for handling general camera motion blur with a projective blur model. We demonstrate that foggy road scene images can be segmented into road, left, right and sky planes, and that each of these planes can be deblurred individually.

Knowledge discovery from big data demands effective representation of data. However, big data are often characterized by high dimensionality, which makes knowledge discovery more difficult. Many techniques for dimensionality reudction have been proposed, including well-known Fisher's Linear Discriminant Analysis (LDA). However, the Fisher criterion is incapable of dealing with heteroscedasticity in the data. A technique based on the Chernoff criterion for linear dimensionality reduction has been proposed that is capable of exploiting heteroscedastic information in the data. While the Chernoff criterion has been shown to outperform the Fisher's, a clear understanding of its exact behavior is lacking. In this article, we show precisely what can be expected from the Chernoff criterion. In particular, we show that the Chernoff criterion exploits the Fisher and Fukunaga-Koontz transforms in computing its linear discriminants. Furthermore, we show that a recently proposed decomposition of the data space into four subspaces is incomplete. We provide arguments on how to best enrich the decomposition of the data space in order to account for heteroscedasticity in the data. Finally, we provide experimental results validating our theoretical analysis.

Visual feature-based tracking systems need to adapt to variations in the appearance of an object and in the scene for robust performance. Though these variations may be small for short time steps, they can accumulate over time and deteriorate the quality of the matching process across longer intervals. Tracking in aerial imagery can be challenging as viewing geometry, calibration inaccuracies, complex ight paths and background changes combined with illumination changes, and occlusions can result in rapid appearance change of objects. Balancing appearance adaptation with stability to avoid tracking non-target objects can lead to longer tracks which is an indicator of tracker robustness. The approach described in this paper can handle affine changes such as rotation by explicit orientation estimation, scale changes by using a multiscale Hessian edge detector and drift correction by using segmentation. We propose an appearance update approach that handles the 'drifting' problem using this adaptive scheme within a tracking environment that is comprised of a rich feature set and a motion model.

Visual object recognition and tracking can be formulated as an object-background classification problem. Since combining multi-modal information is known to exponentially quicken classification, often different features are used to create a set of representations for a pixel or target object. Each of the representations generates a probability of that pixel being part of the target object or scene background. Thus, how to combine these views to effectively exploit multi-modal information for classification becomes a key issue. We propose a margin based fusion technique for exploiting these heterogeneous features for classification, thus tracking. All representations contribute to classification on their learned confidence scores (weights). As a result of optimally combining multi-modal information or evidence, discriminant object and background information is preserved, while ambiguous information is discarded. We provide experimental results that show its performance against competing techniques.

In classification, a large number of features often make the design of a classifier difficult and degrades its performance. In such situations, feature selection or dimensionality reduction methods play an important role in building classifiers by significantly reducing the number of features. There are many dimensionality reduction techniques for classification in the literature. The most popular one is Fisher's linear discriminant analysis (LDA). For two class problems, LDA simply tries to separate class means as much as possible. For the multi-class case, linear reduction does not guarantee to capture all the relevant information for a classification task. To address this problem, a multi-class problem is cast into a binary problem. The objective becomes to find a subspace where the two classes are well separated. This formulation not only simplifies the problem but also works well in practice. However, it lacks theoretical justification. We show in this paper the connection between the above formulation and RELIEF, thereby providing a sound basis for observed benefits associated with this formulation. Experimental results are provided that corroborate with our analysis.

We propose an indexed TCAM architecture, PC-TRIO, for packet classifiers. PC-TRIO uses wide SRAMs and index TCAMs. On our classifier datasets, PC-TRIO on an average reduced TCAM power by 96% and lookup time by 98%, compared to PC-DUOS+ [23] that does not use indexing or wide SRAMs. We extend PC-DUOS+ by augmenting it with wide SRAMs and index TCAMs using the same methodology as used in PC-TRIO, to obtain PC-DUOS+W. On ACL datasets, PC-DUOS+W reduced TCAM power by 86% and lookup time by 98%, compared to PC-DUOS+.

We propose algorithms for distributing the classifier rules to two TCAMs (ternary content addressable memories) and for incrementally updating the TCAMs. The performance of our scheme is compared against the prevalent scheme of storing classifier rules in a single TCAM in priority order. Our scheme results in an improvement in average lookup speed by up to 48% and our experiments demonstrate an improvement in update performance by up to 2.8 times in terms of the number of TCAM writes.

Well known linear discriminant analysis (LDA) based on the Fisher criterion is incapable of dealing with heteroscedasticity in data. However, in many practical applications we often encounter heteroscedastic data, i.e., within-class scatter matrices can not be expected to be equal. A technique based on the Chernoff criterion for linear dimensionality reduction has been proposed recently. The technique extends well-known Fisher's LDA and is capable of exploiting information about heteroscedasticity in the data. While the Chernoff criterion has been shown to outperform the Fisher's, a clear understanding of its exact behavior is lacking. In addition, the criterion, as introduced, is rather complex, making it difficult to clearly state its relationship to other linear dimensionality reduction techniques. In this paper, we show precisely what can be expected from the Chernoff criterion and its relations to the Fisher criterion and Fukunaga-Koontz transform. Furthermore, we show that a recently proposed decomposition of the data space into four subspaces is incomplete. We provide arguments on how to best enrich the decomposition of the data space in order to account for heteroscedasticity in the data. Finally, we provide experimental results validating our theoretical analysis.

Well known linear discriminant analysis (LDA) based on the Fisher criterion is incapable of dealing with heteroscedasticity in data. However, in many practical applications we often encounter heteroscedastic data, i.e., within class scatter matrices can not be expected to be equal. A technique based on the Chernoff criterion for linear dimensionality reduction has been proposed recently. The technique extends well-known Fisher's LDA and is capable of exploiting information about heteroscedasticity in the data. While the Chernoff criterion has been shown to outperform the Fisher's, a clear understanding of its exact behavior is lacking. In addition, the criterion, as introduced, is rather complex, thereby making it difficult to clearly state its relationship to other linear dimensionality techniques. In this paper, we show precisely what can be expected from the Chernoff criterion and its relations to the Fisher criterion and Fukunaga-Koontz transform. Furthermore, we show that a recently proposed decomposition of the data space into four subspaces is incomplete. We provide arguments on how to best enrich the decomposition of the data space in order to account for heteroscedasticity in the data.

In this paper, we study the Papoulis–Gerchberg (PG) method and its applications to domains of image restoration such as super-resolution (SR) and inpainting. We show that the method performs well under certain conditions. We then suggest improvements to the method to achieve better SR and inpainting results. The modification applied to the SR process also allows us to apply the method to a larger class of images by doing away with some of the restrictions inherent in the classical PG method. We also present results to demonstrate the performance of the proposed techniques.

We present a new approach aimed at fast multiframe registration of airborne video collected by moving platforms such as unmanned aerial vehicles. Registration is used to enable separating the moving objects from the stationary background, which is similar to estimating the egomotion of the sensor. The proposed registration algorithm is used to match the moving background and remap video frames into a common co-ordinate system in order to stabilize that segment of video. The major modules include dense feature detection, sparse prominent edge and corner-based feature block identification, feature block region matching to extract control points, confidence weighted robust projective transformation estimation and image warping to register a segment of frames within a given temporal window. The proposed method is shown to produce good results with small image registration error.

Persistent object tracking in complex and adverse environments can be improved by fusing information from multiple sensors and sources. We present a new moving object detection and tracking system that robustly fuses infrared and visible video within a level set framework. We also introduce the concept of the flux tensor as a generalization of the 3D structure tensor for fast and reliable motion detection without eigen-decomposition. The infrared flux tensor provides a coarse segmentation that is less sensitive to illumination variations and shadows. The Beltrami color metric tensor is used to define a color edge stopping function that is fused with the infrared edge stopping function based on the grayscale structure tensor. The min fusion operator combines salient contours in either the visible or infrared video and drives the evolution of the multispectral geodesic active contour to refine the coarse initial flux tensor motion blobs. Multiple objects are tracked using correspondence graphs and a cluster trajectory analysis module that resolves incorrect merge events caused by under-segmentation of neighboring objects or partial and full occlusions. Long-term trajectories for object clusters are estimated using Kalman filtering and watershed segmentation. We have tested the persistent object tracking system for surveillance applications and demonstrate that fusion of visible and infrared video leads to significant improvements for occlusion handling and disambiguating clustered groups of objects

We design and implement an efficient architecture for geometric computation of the global position of an airborne video camera from images of known landmarks. A robust Hough transform-like method facilitated by a class of CORDIC-structured computations is implemented. It empowers aerial surveillance systems to navigate effectively when the GPS and inertial navigation sensors are out of order or GPS functionality is disrupted by jamming etc. Our architecture exploits parallelism, data correlation and uses pipelining with minimal double precision division for superior performance.

This paper proposes a new image representation scheme for database indexing. The application domain is "image-based" indexing as opposed to "content-based" indexing. In other words, given a database of images and associated data, we want to search the database for a known image so as to retrieve the data associated with the image. A concrete example is to search a database of aerial images so as to retrieve the GPS coordinates of a known scene. Our representation is based on a hierarchy of color histograms and offers a compromise between methods based on a global histogram and methods based on multiple histograms corresponding to segmented regions. Our method retains positional information better than the above schemes and naturally leads to a multi-level comparison strategy where mismatches are quickly discarded at higher levels. Our method offers a simple, easily implementable solution to the specific problem of image-based indexing.

A piecewise-affine image registration method is proposed to compute the displacement field in an image sequence of an aerodynamic object experiencing heavy winds in a wind tunnel. Our method is useful for tracking objects whose net 3-D motion is fully characterized by a non-rigid motion and a significant component that is common to all parts of the object. A set of control points has been introduced on the object surface to help divide each image into a set of triangles and enable tracking each triangular area. The computed velocity field is piecewise affine for each triangle and is continuous across the boundary between any two adjacent triangles. Of interest is the accuracy of the registration process, and its estimate based on local moments and shape distortion that can be computed from the original image and the motion compensated version of its registered pair. The method has been applied to several images and the experimental results are presented.

This paper presents the outline of a conceptual control and communications architectural framework enabling agile manufacturing enterprises. A distributed intelligence control architecture model that imparts autonomous intelligence capabilities of "perception-decision-action" to machines and equipment within a manufacturing shop floor is presented. A cluster of such intelligent machines, under the supervision of the workcell controller enables flexibility and dynamic reconfigurability of manufacturing systems. In addition, the paper discusses a conceptual communications architecture framework, based on emerging technologies such as CORBA, Java, XML, etc., that may be used effectively for integrating flexible manufacturing systems into the overall manufacturing system, enabling enterprise agility.

We present a new technique for finding the optical center of a camera. The proposed technique uses spheres as calibration objects. We prove a new key result, that the major axis of the ellipse image resulting from the perspective projection of a sphere would pass through the optical center of the image, and use this result to find the optical center of the camera. Spheres being isotropic, simplify the process of calibration greatly. The proposed technique has been tested on a large number of images and we present the results of some of the test cases.

This paper investigates a fundamental problem of determining the position, orientation, and velocity field of the Gulf Stream in time-varying imagery. The authors propose an approximation method to characterize the deformation of these image motions for the purpose of estimating the velocity field of these images. The technique is focused on the interpretation of the change in the extracted features of the Gulf Stream. The underlying technique employs a triangulation of the region by a simplicial approximation of the velocity field on each triangle. A generalized computational framework, an outline of the mathematical foundation, and an implementation are presented.