Currently, there are many approaches for vehicle classification, but there is no specific study on automated, rear view, and video-based robust vehicle classification. The rear view is important for intelligent transportation systems since not all states in the United States require a frontal license plate on a vehicle. The classification of vehicles, from their rear views, is challenging since vehicles have only subtle appearance differences and there are changing illumination conditions and the presence of moving shadows. In this paper, we present a novel multi-class vehicle classification system that classifies a vehicle into one of four possible classes (sedan, minivan, SUV, and a pickup truck) from its rear view video, using physical and visual features. For a given geometric setup of the camera on highways, we make physical measurements on a vehicle. These measurements include visual rear ground clearance, the height of the vehicle, and the distance between the license plate and the rear bumper. We call these distances as the physical features. The visual features, also called appearance-based features, are extracted using convolutional neural networks from the input images. We achieve a classification accuracy of 93.22% and 91.52% using physical and visual features, respectively. Furthermore, we achieve a higher classification accuracy of 94.81% by fusing both the features together. The results are shown on a dataset consisting of 1831 rear view videos of vehicles and they are compared with various approaches, including deep learning techniques.

Person reidentification is a problem of recognizing a person across non-overlapping camera views. Pose variations, illumination conditions, low resolution images, and occlusion are the main challenges encountered in reidentification. Due to the uncontrolled environment in which the videos are captured, people could appear in different poses and due to which the appearance of a person could vary significantly. The walking direction of a person can provide a good estimation of their pose. Therefore, in this paper, we propose a reidentification system which adaptively selects an appropriate distance metric based on context of walking direction using reinforcement learning. Though experiments, we show that such a dynamic strategy outperforms static strategy learned or designed offline.

Facial emotion recognition in unconstrained settings is a difficult task. They key problems are that people express their emotions in ways that are different from other people, and, for large datasets, there are not enough examples of a specific person to model his/her emotion. A model for predicting emotions will not generalize well to predicting the emotions of a person who has not been encountered during the training. We propose a system that addresses these issues by matching a face video to references of emotion. It does not require examples from the person in the video being queried. We compute the matching scores without requiring fine registration. The method is called one-shot emotion score. We improve classification rate of interdataset experiments over a baseline system by 23% when training on MMI and testing on CK+.

This paper presents a make and model recognition system for passenger vehicles. We propose a two-step efficient alignment mechanism to account for view point changes. The 2D alignment problem is solved as two separate one dimensional shortest path problems. To avoid the alignment of the query with the entire database, reference views are used. These views are generated iteratively from the database. To improve the alignment performance further, use of two references is proposed: a universal view and type specific showcase views. The query is aligned with universal view first and compared with the database to find the type of the query. Then the query is aligned with type specific showcase view and compared with the database to achieve the final make and model recognition. We report results on database of 1500 vehicles with more than 250 makes and models.

Affective computing-the emergent field in which computers detect emotions and project appropriate expressions of their own-has reached a bottleneck where algorithms are not able to infer a person's emotions from natural and spontaneous facial expressions captured in video. While the field of emotion recognition has seen many advances in the past decade, a facial emotion recognition approach has not yet been revealed which performs well in unconstrained settings. In this paper, we propose a principled method which addresses the temporal dynamics of facial emotions and expressions in video with a sampling approach inspired from human perceptual psychology. We test the efficacy of the method on the Audio/Visual Emotion Challenge 2011 and 2012, CohnKanade and the MMI Facial Expression Database. The method shows an average improvement of 9.8 percent over the baseline for weighted accuracy on the Audio/Visual Emotion Challenge 2011 video-based frame-level subchallenge testing set.

State-of-the-art approaches have yet to deliver a feature representation for facial emotion recognition that can be applied to non-trivial unconstrained, continuous video data sets. Initially, research advanced with the use of Gabor energy filters. However, in recent work more attention has been given to other features. Gabor energy filters lack generalization needed in unconstrained situations. Additionally, they result in an undesirably high feature vector dimensionality. Nontrivial data sets have millions of samples; feature vectors must be as low dimensional as possible. We propose a novel texture feature based on Gabor energy filters that offers generalization with a background texture suppression component and is as compact as possible due to a maximal response representation and local histograms. We improve performance on the non-trivial Audio/Visual Emotion Challenge 2012 grandchallenge data set.

This paper focuses on a challenging pattern recognition problem of significant industrial impact, i.e., classifying vehicles from their rear videos as observed by a camera mounted on top of a highway with vehicles traveling at high speed. To solve this problem, this paper presents a novel feature called structural signature. From a rear-view video, a structural signature recovers the vehicle side profile information, which is crucial in its classification. As a vehicle moves away from a camera, its surfaces deform differently based on their relative orientation to the camera. This information is used to extract the structure of a vehicle, which captures the relative orientation of vehicle surfaces and the road surface. This paper presents a complete system that computes structural signatures and uses them for classification of passenger vehicles into sedans, pickups, and minivans/sport utility vehicles in highway videos. It analyzes the performance of the proposed system on a large video data set.

Recognition of a vehicle make is of interest in the fields of law enforcement and surveillance. In this paper, we develop a canonical correlation analysis (CCA) based method for vehicle logo super-resolution to facilitate the recognition of the vehicle make. From a limited number of high-resolution logos, we populate the training dataset for each make using gamma transformations. Given a vehicle logo from low-resolution source (i.e., surveillance or traffic camera recordings), the learned models yield super-resolved results. By matching the low-resolution image and the generated high-resolution images, we select the final output that is closest to the low-resolution image in the histogram of oriented gradients (HOG) feature space. Experimental results show that our approach outperforms the state-of-the-art super-resolution methods in qualitative and quantitative measures. Furthermore, the super-resolved logos help to improve the accuracy in the subsequent recognition tasks significantly.

Facial emotion recognition, the inference of an emotion from apparent facial expressions, in unconstrained settings is a typical case where algorithms perform poorly. A property of the AVEC2012 data set is that individuals in testing data are not encountered in training data. In these situations, conventional approaches suffer because models developed from training data cannot properly discriminate unforeseen testing samples. Additional information beyond the feature vectors is required for successful detection of emotions. We propose two similarity metrics that address the problems of a conventional approach: neutral similarity, measuring the intensity of an expression; and temporal similarity, measuring changes in an expression over time. These similarities are taken to be the energy of facial expressions, measured with a SIFT-based warping process. Our method improves correlation by 35.5% over the baseline approach on the frame-level sub-challenge.

The performance of a recognition system is usually experimentally determined. Therefore, one cannot predict the performance of a recognition system a priori for a new dataset. In this paper, a statistical model to predict the value of k in the rank-k identification rate for a given bio- metric system is presented. Thus, one needs to search only the topmost k match scores to locate the true match object. A geometrical probability distribution is used to model the number of non match scores present in the set of similarity scores. The model is tested in simulation and by using a public dataset. The model is also indirectly validated against the previously published results. The actual results obtained using publicly available database are very close to the predicted results which validates the proposed model.

Branch-and-bound methods are used in various data analysis problems, such as clustering, seriation and feature selection. Classical approaches of branch-and-bound based clustering search through combinations of various partitioning possibilities to optimize a clustering cost. However, these approaches are not practically useful for clustering of image data where the size of data is large. Additionally, the number of clusters is unknown in most of the image data analysis problems. By taking advantage of the spatial coherency of clusters, we formulate an innovative branch-and-bound approach, which solves clustering problem as a model-selection problem. In this generalized approach, cluster parameter candidates are first generated by spatially coherent sampling. A branch-and-bound search is carried out through the candidates to select an optimal subset. This paper formulates this approach and investigates its average computational complexity. Improved clustering quality and robustness to outliers compared to conventional iterative approach are demonstrated with experiments.

An efficient and robust framework is proposed for two-view multiple structure-and-motion segmentation of unknown number of rigid objects. The segmentation problem has three unknowns, namely the object memberships, the corresponding fundamental matrices, and the number of objects. To handle this otherwise recursive problem, hypotheses for fundamental matrices are generated through local sampling. Once the hypotheses are available, a combinatorial selection problem is formulated to optimize a model selection cost which takes into account the hypotheses likelihoods and the model complexity. An explicit model for outliers is also added for robust segmentation. The model selection cost is minimized through the branch-and-bound technique of combinatorial optimization. The proposed branch-and-bound approach efficiently searches the solution space and guarantees optimality over the current set of hypotheses. The efficiency and the guarantee of optimality of the method is due to its ability to reject solutions without explicitly evaluating them. The proposed approach was validated with synthetic data, and segmentation results are presented for real images.

Segmentation problems are one of the most important areas of research in computer vision. While segmentation problems are generally solved with clustering paradigms, they formulate the problem as recursive. Additionally, most approaches need the number of clusters to be known beforehand. This requirement is unreasonable for majority of the computer vision problems. This paper analyzes the model selection perspective which can overcome these limitations. Under this framework multiple hypotheses for cluster centers are generated using spatially coherent sampling. An optimal subset of these hypotheses is selected according to a model selection criterion. The selection can be carried out with a branch-and-bound procedure. The worst case complexity of any branch-and-bound algorithm is exponential. However, the average complexity of the algorithm is significantly lower. In this paper, we develop a framework for analysis of average complexity of the algorithm from the statistics of model selection costs.

An efficient and robust framework for two-view multiple structure and motion segmentation is proposed. To handle this otherwise recursive problem, hypotheses for the models are generated by local sampling. Once these hypotheses are available, a model selection problem is formulated which takes into account the hypotheses likelihoods and model complexity. An explicit model for outliers is also added for robust model selection. The model selection criterion is optimized through branch-and-bound technique of combinatorial optimization which guaranties optimality over current set of hypotheses by efficient search of solution space.

A split-and-merge framework based on a maximum variance criterion is proposed for disparity clustering. The proposed algorithm transforms low-level stereo disparity information to mid-level planar surface information which can be used further to carry out high-level computer vision tasks such as shape classification. Unlike conventional clustering, the proposed algorithm assumes that the number of clusters is unknown. Instead, a maximum variance criterion is applied to extract planar surfaces from the disparity image. The split phase of the algorithm creates clusters based on spatial continuity and the merge phase combines these clusters such that variance per cluster does not exceeded an allowable value. For efficient maximum variance clustering, a greedy branch-and-bound procedure is introduced. Efficiency of the approach is verified through experiments.

An iterative segmentation-estimation framework for segmentation of planar surfaces in the disparity space is proposed. Disparity of a scene is modeled by approximating various surfaces in the scene to be planar. Uncertainty in the stereo disparity of these surfaces is modeled with a hidden Markov random field. Planar surface labels are the hidden variables of the model. Surface labels are estimated during the segmentation phase of the framework with help of underlying plane parameters and the spatial coherency prior. After segmentation, the planar surfaces are separated into spatially continuous regions. Each spatially continuous region is treated as a hypothesis for the underlying plane. Hypotheses for each plane are then combined to form a multihypothesis Gaussian mixture prior. The underlying planar surface parameters are estimated with maximum a posteriori estimation from the multihypothesis prior. The iterative process is continued until the labels converge or satisfactory segmentation is achieved. Experimental results are presented with synthetic as well as real-life scenes to demonstrate the success of the proposed method.

An iterative split-and-merge framework for the segmentation of planar surfaces in the disparity space is presented. Disparity of a scene is modeled by approximating various surfaces in the scene to be planar. In the split phase, the number of planar surfaces along with the underlying plane parameters is assumed to be known from the initialization or from the previous merge phase. Based on these parameters, planar surfaces in the disparity image are labeled to minimize the residuals between the actual disparity and the modeled disparity. The labeled planar surfaces are separated into spatially continuous regions which are treated as candidates for the merging that follows. The regions are merged together under a maximum variance constraint while maximizing the merged area. A multistage branch-and-bound algorithm is proposed to carry out this optimization efficiently. Each stage of the branch-and-bound algorithm separates a planar surface from the set of spatially continuous regions. The multistage merging estimates the number of planar surfaces and their labeling. The splitting and the multistage merging is repeated till convergence is reached or satisfactory results are achieved. Experimental results are presented for variety of stereo image data.

An iterative Segmentation-Estimation framework for segmentation of planar surfaces in the disparity space is implemented on a Digital Signal Processor (DSP). Disparity of a scene is modeled by approximating various surfaces in the scene to be planar. The surface labels are estimated during the segmentation phase of the framework with help of the underlying plane parameters. After segmentation, planar surfaces are separated into spatially continuous regions. The largest of these regions is used to compute the estimates for the plane parameters. The iterative process is continued till convergence. The algorithm was optimized and implemented on TMS320DM642 based embedded system that operates at 3 to 5 frames per second on images of size 320 x 240.

In this paper a probabilistic technique for compensation of intensity loss in the confocal microscopy images is presented. Confocal microscopy images are modeled as a mixture of two Gaussians, one representing the background and another corresponding to the foreground. Images are segmented into foreground and background by applying expectation maximization (EM) algorithm to the mixture. Final intensity compensation is carried out by scaling and shifting the original intensities with help of parameters estimated for the foreground. Since foreground is separated to calculate the compensation parameters, the method is effective even when image structure changes from frame to frame. As intensity decay function (IDF) is not used, complexity associated with estimation of IDF parameters is eliminated. Also, images can be compensated out of order as only information from the reference image is required for compensation of any image. These properties make our method an ideal tool for intensity compensation of confocal microscopy images which can suffer intensity loss due to absorption/scattering of light as well as photobleaching and can change structure from optical/temporal section to section due to change in the depth of specimen or due to a living specimen. The proposed method was tested with number of image stacks and results for one of the stacks are presented here to demonstrate the effectiveness of the method.

The goal of this paper is to present a weighted likelihood discriminant for minimum error shape classification. Different from traditional maximum likelihood (ML) methods, in which classification is based on probabilities from independent individual class models as is the case for general hidden Markov model (HMM) methods, proposed method utilizes information from all classes to minimize classification error. The proposed approach uses a HMM for shape curvature as its 2-D shape descriptor. We introduce a weighted likelihood discriminant function and present a minimum classification error strategy based on generalized probabilistic descent method. We show comparative results obtained with our approach and classic ML classification with various HMM topologies alongside Fourier descriptor and Zernike moments-based support vector machine classification for a variety of shapes.

In this paper, we present a novel occlusion detection scheme for hidden Markov modeled shapes. First, hidden Markov model (HMM) is built using multiple examples of the shape. A reference path for the shape is built from the HMM, which is nothing but optimal path followed by the most likely example. The reference path stores temporal information about the entire shape, while the HMM only retains relationship between temporal information. For the shape of interest, its optimal path through HMM is calculated and warped to match the reference path using dynamic time warping (DTW). Occluded part of the shape is detected by identifying imbalance among various components of the matching cost. Detection results obtained for two shape data sets are presented for varying degrees of occlusion.

The goal of this paper is to present a weighted likelihood discriminant for minimum error shape classification. Different from traditional maximum likelihood (ML) methods, in which classification is based on probabilities from independent individual class models as is the case for general hidden Markov model (HMM) methods, proposed method utilizes information from all classes to minimize classification error. The proposed approach uses a HMM for shape curvature as its 2D shape descriptor. In this contribution, we introduce a weighted likelihood discriminant function and present a minimum error classification strategy based on generalized probabilistic descent (GPD) method. We believe our sound theory based implementation reduces classification error by combining HMM with GPD theory. We show comparative results obtained with our approach and classic ML classification along with Fourier descriptor and Zernike moments based classification for fighter planes and vehicle shapes.

In this paper, we present an automatic moving object extraction and classification system. For automatic extraction of object taken by a moving camera, a novel technique is proposed, in which optical flow handles the background modeling and camera motion estimation, and frame difference information yields the exact object shape. We also use forward region boundary based change detection approach for frame difference. This approach assures change detection for uniform intensity regions. For classification, weighted likelihood discriminant based shape classifier is designed. Unlike maximum likelihood (ML) methods, our proposed method utilizes information from all classes to design the classifier. In the description phase of the classifier, curvature features are extracted from the shape and are utilized to build a hidden Markov model (HMM). The HMM provides a robust ML description of the shape. In the discrimination phase, a weighted likelihood discriminant function is introduced, which weights the likelihoods of curvature at individual points of the shape to minimize the classification error. The weights are estimated by generalized probabilistic descent (GPD) method. To demonstrate the performance of the proposed method, we present results achieved for car shapes extraction and classification.

The goal of this communication is to present a weighted likelihood discriminant for minimum error shape classification. Different from traditional Maximum Likelihood (ML) methods in which classification is carried out based on probabilities from independent individual class models as is the case for general hidden Markov model (HMM) methods, our proposed method utilizes information from all classes to minimize classification error. Proposed approach uses a Hidden Markov Model as a curvature feature based 2D shape descriptor. In this contribution we present a Generalized Probabilistic Descent (GPD) method to weight the curvature likelihoods to achieve a discriminant function with minimum classification error. In contrast with other approaches, a weighted likelihood discriminant function is introduced. We believe that our sound theory based implementation reduces classification error by combining hidden Markov model with generalized probabilistic descent theory. We show comparative results obtained with our approach and classic maximum-likelihood calculation for fighter planes in terms of classification accuracies.

In this paper, we propose a technique for automatic extraction and localization of multiple moving objects captured by a moving stereo camera. Depth information obtained from the stereo matching is applied to initialize a background mask which is refined further by clustering optical flow. Simultaneous estimate for camera motion is also obtained and video frames are compensated accordingly. Refined background mask is used to model displaced frame difference (DFD) of the background. The objects are detected as outliers to this model. We also present a region boundary based change detection approach for frame difference. This approach assures change detection for uniform intensity regions. Finally we present experimental results for indoor and outdoor stereo video sequences captured with moving camera

Motion segmentation is a classic and on-going research topic which is an important pre-stage for many video processes. The reliability of the motion field calculation directly determines either success or failure of such segmentation. Due to the aperture property of optical flow calculation, motion estimation at a moving object boundary remains a challenging ill-posed problem. We propose a reliable optical-flow estimation method from the view of missing data reconstruction. Furthermore, to overcome the occlusion problem at motion boundaries, we apply a motion segmentation scheme which integrates with spatial segmentation. Experimental results of the proposed and other methods are presented.