Multiple views play a vital role in 3D pose estimation tasks. Ideally, multi-view 3D pose estimation tasks should directly utilize naturally collected videos for pose estimation. However, due to the constraints of video synchronization, existing methods often use expensive hardware devices to synchronize the initiation of cameras, which restricts most 3D pose collection scenarios to indoor settings. Some recent works learn deep neural networks to align desynchronized datasets derived from synchronized cameras and can only produce frame-level accuracy. For fractional frame video synchronization, this work proposes an Inter-Frame and Intra-Frame Desynchronized Dataset (IFID), which labels fractional time intervals between two video clips. IFID is the first dataset that annotates inter-frame and intra-frame intervals, with a total of 382,500 video clips annotated, making it the largest dataset to date. We also develop a novel model based on the Transformer architecture, named InSynFormer, for synchronizing inter-frame and intra-frame. Extensive experimental evaluations demonstrate its promising performance. The dataset and source code of the model are available at https://github.com/yuxuan-cser/InSynFormer.

Person re-identification (ReID) aims at matching identities across disjoint cameras. Its fundamental difficulty lies in associating images across individual cameras, where a key clue,i.e., identity appearance, is prone to the environmental factors of cameras and, consequently, subject to distinct image distributions due to the environmental differences between cameras. To associate images from training cameras, ReID methods strongly demand expensive inter-camera annotations for learning the relations between the distribution of these cameras, yet trained models are still not guaranteed to transfer well to unseen cameras. This problem significantly limits the application of ReID. This paper rethinks the working mechanism of conventional ReID approaches and puts forward a new solution. With an effective operator named Camera-based Batch Normalization (CBN), we guarantee an invariant input distribution independent of all cameras. Thus, the training and testing procedures are always conducted under the same input distribution. This alignment brings three benefits. First, ReID models enjoy better abilities to generalize across testing scenarios with unseen cameras and transfer across multiple training sets. Second, it makes better use of intra-camera annotations, which have been undervalued before due to the lack of cross-camera information. Ideally, the cost of inter-camera annotations can be largely reduced. Third, cross-modality tasks can be better defined through aligning visible/infrared cameras’ distributions. Experiments on a wide range of ReID tasks demonstrate the effectiveness of our approach.

Estimating 3D poses of multiple humans in real-time is a classic but still challenging task in computer vision. Its major difficulty lies in the ambiguity in cross-view association of 2D poses and the huge state space when there are multiple people in multiple views. In this paper, we present a novel solution for multi-human 3D pose estimation from multiple calibrated camera views. It takes 2D poses in different camera coordinates as inputs and aims for the accurate 3D poses in the global coordinate. Unlike previous methods that associate 2D poses among all pairs of views from scratch at every frame, we exploit the temporal consistency in videos to match the 2D inputs with 3D poses directly in 3-space. More specifically, we propose to retain the 3D pose for each person and update them iteratively via the cross-view multi-human tracking. This novel formulation improves both accuracy and efficiency, as we demonstrated on widely-used public datasets. To further verify the scalability of our method, we propose a new large-scale multi-human dataset with 12 to 28 camera views. Without bells and whistles, our solution achieves 154 FPS on 12 cameras and 34 FPS on 28 cameras, indicating its ability to handle large-scale real-world applications. The proposed dataset will be released at https://github.com/longcw/crossview_3d_pose_tracking.

Learning to predict multiple attributes of a pedestrian is a multi-task learning problem. To share feature representation between two individual task networks, conventional methods like Cross-Stitch [1] and Sluice [2] network learn a linear combination of features or feature subspaces. However, linear combination rules out the complex interdependency between channels. Moreover, spatial information exchanging is less-considered. In this paper, we propose a novel Co-Attentive Sharing (CAS) module which extracts discriminative channels and spatial regions for more effective feature sharing in multi-task learning. The module consists of three branches, which leverage different channels for between-task feature fusing, attention generation and task-specific feature enhancing, respectively. Experiments on two pedestrian attribute recognition datasets show that our module outperforms the conventional sharing units and achieves superior results compared to the state-of-the-art approaches using many metrics.

It remains very challenging to build a pedestrian detection system for real world applications, which demand for both accuracy and speed. This work presents a novel hierarchical knowledge distillation framework to learn a lightweight pedestrian detector, which significantly reduces the computational cost and still holds the high accuracy at the same time. Following the `teacher-student' diagram that a stronger, deeper neural network can teach a lightweight network to learn better representations, we explore multiple knowledge distillation architectures and reframe this approach as a unified, hierarchical distillation framework. In particular, the proposed distillation is performed at multiple hierarchies, multiple stages in a modern detector, which empowers the student detector to learn both low-level details and high-level abstractions simultaneously. Experiment result shows that a student model trained by our framework, with 6 times compression in number of parameters, still achieves competitive performance as the teacher model on the widely used pedestrian detection benchmark.

Multi-object tracking (MOT) has wide applications in the fields of video analysis and signal processing. A major challenge in MOT is how to associate the noisy detections into long and continuous trajectories. In this letter, we address the association problem at the tracklet-level, and mainly focus on the appearance representation designed for tracklets. A multitask convolutional neural network is proposed to learn the discriminative features and spatial-temporal attentions jointly. In particular, we decompose an object in a static image with spatial attentions, and then aggregate multiple features in a tracklet based on the temporal attentions. Appearance misalignment that caused by occlusion and inaccurate bounding is then mitigated by multi-feature aggregation. Experimental results on two challenging MOT benchmarks have demonstrated the effectiveness of the proposed method and shown significant improvement on the quality of tracking identities.

It remains very challenging for a single deep model to detect pedestrians of different sizes appears in an image. One typical remedy for the small pedestrian detection is to up-sample the input and pass it to the network multiple times. Unfortunately this strategy not only exponentially increases the computational cost but also probably impairs the model effectiveness. In this work, we present a deep architecture, refereed to as ZoomNet, which performs small pedestrian detection by deep aggregation learning without up-sampling the input. ZoomNet learns and aggregates deep feature representations at multiple levels and retains the spatial information of the pedestrian from different scales. ZoomNet also learns to cultivate the feature representations from the classification task to the detection task and obtains further performance improvements. Extensive experimental results demonstrate the state-of-the-art performance of ZoomNet. The source code of this work will be made public available to facilitate further studies on this problem.

Online multi-object tracking is a fundamental problem in time-critical video analysis applications. A major challenge in the popular tracking-by-detection framework is how to associate unreliable detection results with existing tracks. In this paper, we propose to handle unreliable detection by collecting candidates from outputs of both detection and tracking. The intuition behind generating redundant candidates is that detection and tracks can complement each other in different scenarios. Detection results of high confidence prevent tracking drifts in the long term, and predictions of tracks can handle noisy detection caused by occlusion. In order to apply optimal selection from a considerable amount of candidates in real-time, we present a novel scoring function based on a fully convolutional neural network, that shares most computations on the entire image. Moreover, we adopt a deeply learned appearance representation, which is trained on large-scale person re-identification datasets, to improve the identification ability of our tracker. Extensive experiments show that our tracker achieves real-time and state-of-the-art performance on a widely used people tracking benchmark.

Coherent image regions can be used as good features for many computer vision tasks, such as object tracking, segmentation, and recognition. Most of previous region extraction methods, however, are not suitable for online applications because of their either heavy computations or unsatisfactory results. We propose a seed-based region growing and merging approach to generate simultaneously coherent and discriminative image regions. We present a quadtree-based seed initialization algorithm to adaptively place seeds into different image areas and then grow them into regions by a color- and edge-guided growing procedure. To merge these regions in different levels, we propose to use the generalized boundary strength to measure the quality of region merging result. In addition, we present a region merging algorithm of linear time complexity to perform efficient and effective region merging. Overall, our new approach simultaneously holds these advantages: 1) it is extremely fast with linear complexity in both time and space, which takes less than 50 ms to process an HVGA image; 2) it can give a direct control of the region number and well adapt to image regions with various sizes and shapes; and 3) it provides a tree-structured representation of the regions and thus can model the image from multiple scales. We evaluate the proposed approach on the standard benchmarks with extensive comparisons with the state-of-the-art methods. The experimental results demonstrate its good comprehensive performances. Example applications using the extracted regions as features for online object tracking and multiclass object segmentation also exhibit its potential for many computer vision tasks.

In this paper, we propose a novel online multi-object tracking (MOT) framework, which exploits features from multiple convolutional layers. In particular, we use the top layer to formulate a category-level classifier and use a lower layer to identify instances from one category under the intuition that lower layers contain much more details. To avoid the computational cost caused by online fine-tuning, we train our appearance model with an offline learning strategy using the historical appearance reserved for each object. We evaluate the proposed tracking framework on a popular MOT benchmark to demonstrate the effectiveness and the state-of-the-art performance of our tracker.

In computer vision tasks, such as age estimation problem, a typical deep learning model is insufficient to represent all the transformations between data and their labels. In this paper, we present a novel deep neural network called cluster convolutional neural network (Cluster-CNN) to estimate ages from facial images, which is based on the concept that clustering rich CNN features is able to assist the network to efficiently tackle the nonlinearity of this task. In particular, given a facial image, we first coarsely normalize a face to a standard size according to the distance between two eyes. Then the normalized face is fed into a Cluster-CNN to get the prediction. The cluster module we propose in this paper is able to capture multimodal transformations, and is differentiable, thus making it capable to be optimized in a unified backpropagation fashion. We evaluate the proposed method on the popular MORPH II dataset. To the best of our knowledge, our model outperforms all the other previous methods by a large margin.

The basic goal of person re-identification (re-id) tasks is to identify a person across disjoint camera views. It has been deeply explored in video surveillance but still remains a very challenging problem. In this paper, we introduce a novel model for re-id tasks with two components: an expressive feature fusion strategy that consists of high-level convolution features and the low-level optical information, and an improved recurrent attention model that performs a coarse-to-fine feature selection. To the best of our knowledge, experiments show that our model achieves the best performance on several benchmark datasets compared with all the other state-of-the-art approaches.

The problem of person re-identification, identifying the same person appeared in different camera views, is an important and challenging task in computer vision that has high potential application in areas like visual surveillance. In this paper we introduce a new feature fusion strategy for person reidentification that combines low-level Weighted Histograms of Overlapping Stripes (WHOS) features with mid-level color name descriptors and we adopt KISSME algorithm for person matching. Experiments on several public person reidentification datasets (VIPeR, i-LIDS and CAVIAR4REID) demonstrate that our approach achieves much better results compared with other state-of-the-art approaches.

Human detection in RGB-D images is an important yet very challenging task in computer vision. In this paper, we propose a novel human detection approach in RGB-D images, which integrates ROI (region-of-interest) generation, depth-size relationship estimation and a human detector. Our approach has the following advantages: 1) ROI generation and depth-size relationship estimation take full advantage of color and depth information to fast reject about 70% negative samples while maintaining a high recall rate; 2) the cascade-structured human detector can seamlessly concatenate features extracted from both color and depth images; and 3) our method can detect human at a speed of more than 30 fps on 640 χ 480 images on a single laptop CPU without any GPU acceleration. Experiments on challenging public datasets demonstrate the effectiveness of our method.

Crowd counting is a very challenging task in crowded scenes due to heavy occlusions, appearance variations and perspective distortions. Current crowd counting methods typically operate on an image patch level with overlaps, then sum over the patches to get the final count. In this paper, we propose an end-to-end convolutional neural network (CNN) architecture that takes a whole image as its input and directly outputs the counting result. While making use of sharing computations over overlapping regions, our method takes advantages of contextual information when predicting both local and global count. In particular, we first feed the image to a pre-trained CNN to get a set of high level features. Then the features are mapped to local counting numbers using recurrent network layers with memory cells. We perform the experiments on several challenging crowd counting datasets, which achieve the state-of-the-art results and demonstrate the effectiveness of our method.

As humans, we love to rank things. Top ten lists exist for everything from movie stars to scary animals. Ambiguities (i.e., ties) naturally occur in the process of ranking when people feel they cannot distinguish two items. Human reported rankings derived from star ratings abound on recommendation websites such as Yelp and Netflix. However, those websites differ in star precision which points to the need for ranking systems that adapt to an individual user's preference sensitivity. In this work we propose an adaptive system that allows for ties when collecting ranking data. Using this system, we propose a framework for obtaining computer-generated rankings. We test our system and a computer-generated ranking method on the problem of evaluating human attractiveness. Extensive experimental evaluations and analysis demonstrate the effectiveness and efficiency of our work.

In this paper, we aim to track objects that undergo abrupt appearance changes and heavy occlusions. To address these problems, we propose an online structure learning algorithm which contains two layers, block-based online random forest classifiers (BORFs) and online structure models (OSMs). BORFs are able to handle occlusion problems since they model local appearances of the target. To further improve the accuracy and reliability, the algorithm utilizes relational models as context information to combine BORFs into the online structure models. Capturing the discriminative parts of targets with online learnt structures, OSMs help locate targets accurately even when they are heavily occluded. In addition, OSMs guide the block occlusion reasoning and the update scheme of BORFs and relational models, which can handle appearance changes and drift problems effectively. Experiments on challenging videos show that the proposed tracker performs better than several state-of-the-art algorithms which demonstrate the effectiveness of our approach.

This paper proposes a novel hierarchical summarization system for surveillance video in which synopsis video and original video are at the lower layer and dynamic collage at the higher layer. A user can efficiently browse the video content through dynamic zooming into collage images and further skimming synopsis video or original video for details, enabling an interactive interface for video navigation. We call this system Dynamic VideoBook because its structure is akin to a book, that is, dynamic collage images resemble the cover and the contents while synopsis video gives an abridged edition of the video. Unlike previous collage techniques for films or entertainment videos, collage for surveillance video is challenging since such video imposes restrictions on spatiotemporal relationship and focus on multiple foreground objects. So we put forward a new collage method which better conveys the spatiotemporal motion and structurally extracts the visual contents from the video simultaneously. And we also enable dynamic zooming in to reveal more information in real-time. In addition, by mapping the upper layer of sketchy visualization to the lower layer of visual data, this system can efficiently explore and browse the video content, highlight one motion or interactively search one specific fragment.

Hair plays an important role in human appearance. However, hair segmentation is still a challenging problem partially due to the lack of an effective model to handle its arbitrary shape variations. In this paper, we present a part-based model robust to hair shape and environment variations. The key idea of our method is to identify local parts by promoting the effectiveness of the part-based model. To this end, we propose a measurable statistic, called Subspace Clustering Dependency (SC-Dependency), to estimate the co-occurrence probabilities between local shapes. SC-Dependency guarantees output reasonability and allows us to evaluate the effectiveness of part-wise constraints in an information-theoretic way. Then we formulate the part identification problem as an MRF that aims to optimize the effectiveness of the potential functions. Experiments are performed on a set of consumer images and show our algorithm's capability and robustness to handle hair shape variations and extreme environment conditions.

Text detection in natural scenes is fundamental for text image analysis. In this paper, we propose a context-based approach for robust and fast text detection. Our main contribution is that we introduce a new concept of key region, which is described with context according to stroke properties, appearance consistency and specific spatial distribution of text line. With such context descriptors, we adopt SVM to learn a context-based classifier to find key regions in candidate regions. Therein, candidate regions are connected components generated by local binarization algorithm in the areas, which are detected by an offline learned text patch detector. Experimental results on two benchmark datasets demonstrate that our approach has achieved competitive performances compared with the state-of-the-art algorithms including the stroke width transform (SWT) [1] and the hybrid approach based on CRFs [2] with speedup rates of about 1.7x~4.4x.

Adaptation of pre-trained boosted pedestrian detectors to specific scenes is an important yet difficult task in computer vision. To address this problem, a feature reselection strategy is proposed in this paper. The proposed method identifies weak classifiers which do not well adapt to the specific scene, and replaces them with retrained weak classifiers. This feature reselection strategy has the following advantages: 1) it does not need original offline training data, but only uses a few online samples from the target scene; 2) the adapted detector preserves the generality of the generic detector, resulting in very few false positives; and 3) it can adapt a generic detector to a specific scene with very fast speed due to its parallel nature. Experiments on challenging pedestrian detection datasets demonstrate that our proposed strategy can significantly improve the performance of pre-trained boosted detectors in specific scenes with very low computation cost and very little labeling work.

Vehicle detection in traffic scenes is a fundamental task for intelligent transportation system and has many practical applications as diverse as traffic monitoring, intelligent scheduling and autonomous navigation. In recent years, the number of detection approaches in monocular images has grown rapidly. However, most of them focus on detecting other objects (such as face, pedestrian, cat, dog, etc.) and also there lacks of vehicle datasets with various conditions for vehicle detection and comprehensive comparisons. To address these problems, we perform an extensive evaluation of many state-of-the-art detection approaches on vehicles. Our main contributions are: (1) we collect a large dataset of real-world vehicles in frontal/rear view with 30° ~ -30° yaw changes and 5° ~ 45° pitch changes under different weather conditions (snowy, rainy, sunny and cloudy) and illumination variations, and then (2) we evaluate six types of state-of-the-art features in Real AdaBoost framework on the adequate dataset collected by ourselves and a public dataset using the same evaluation protocol. Our study presents a fair comparison and deep analysis of these features in vehicle detection. From these experiments, we explore the characteristics of good features for vehicle detection. (3) Finally, we exploit these characteristics and propose a relatively effective and efficient detector, balancing performance, speed and memory cost which can be put into practical use.

In this paper, we propose a novel and effective structural patches tiling procedure which is able to generate high quality probabilistic masks to guide semantic segmentation.

In this paper, an automatic head-shoulder segmentation method for human photos based on graph cut with shape sketch constraint and border detection through learning is presented. We propose a new shape constraint method based upon graph cut for head-shoulder photos. First, a watershed algorithm is used to over segment the photo into superpixels; next, an iterative shape mask guided graph cut algorithm with sketch constraint is applied to the superpixel level graph to get a border that segments the head-shoulder from its background; finally, a border detector, which is trained by AdaBoost, is used to refine the border. Experiments on consumer photo images demonstrate its effectiveness.

Clothing is one of the most informative cues of human appearance. In this paper, we propose a novel multi-person clothing segmentation algorithm for highly occluded images. The key idea is combining blocking models to address the person-wise occlusions. In contrary to the traditional layered model that tries to solve the full layer ranking problem, the proposed blocking model partitions the problem into a series of pair-wise ones and then determines the local blocking relationship based on individual and contextual information. Thus, it is capable of dealing with cases with a large number of people. Additionally, we propose a layout model formulated as Markov Network which incorporates the blocking relationship to pursue an approximately optimal clothing layout for group people. Experiments demonstrated on a group images dataset show the effectiveness of our algorithm.

In this paper, we propose a method for dot text detection based on FAST points. This problem is different from general scene text detection because of discontinuous text stroke. Unlike many other methods which assume that text is horizontally oriented, our method is able to deal with slant dot text. We extract interesting patches from FAST points and define four features based on the stroke and gray value similarity of dot text to describe a patch. Then, we generate some candidate regions from these patches and utilize SVM to filter out non-dot text ones with the first and second order moments of FAST points in them. Experimental results show that the proposed method is effective and fast to detect dot text.

Detecting people in occlusion and articulated pose remains a big challenging problem in computer vision. To achieve a fast and accurate human detection algorithm, Node-Combined Part Detector (NCPD) Model is proposed in this paper. We make two major contributions: (1) We propose a novel method, torso-nodes combination, to integrate part detectors. (2) We adopt stable part detectors described by Associated Paring Comparison Features (APCF) and trained with Real-AdaBoost algorithm. This new human detection algorithm is not only much faster than the previous work but also maintaining competitive accuracy with the state-of-the-art human detection system. Besides, the algorithm performs better within low false alarm. For average time per image, our algorithm can achieve speedup rate of about 10x as compared with Deformable Part based Model (DPM) and over 125x as compared with Poselet Model.

Crowd counting which aims at obtaining the number of people within a scene is an important computer vision task. While most previous methods try to count people within one frame, this paper addresses this problem using the detection flow which is defined as a set of object detection responses along the temporal video sequence. We argue that counting based on detection flow provides a better way to estimate the crowd size with following merits: 1) it can greatly alleviate the common weakness of an object detector including miss detection and false alarms; 2) it is robust to temporal object occlusions and noises; 3) it is more competent to give specific descriptions of the crowd, e.g. crowd moving directions and target locations. Experiment results on PETS 2009 dataset demonstrate the potential of this method.

Background subtraction plays a key role in many surveillance systems. A good background subtractor should not only be able to robustly detect targets under different situations (e.g. moving and static), but also to adaptively maintain the background model against various influences (e.g. dynamic scenes and noises). This paper proposes a novel background modeling approach with these good characteristics. By introducing the “life span” concept into a background model, different properties of the scene are obtained through different life span models. Specifically, three different models, i.e., the Long Life Span Model, the Middle Life Span Model, and the Short Life Span Model, are online adaptively built and updated in a collaborative manner. Output of the system gives an adaptive, robust, and efficient estimation of the foreground region which can facility many practical applications. Experiment results on lots of surveillance videos demonstrate the superiority of the proposed method over competing approaches.

Multiple object tracking (MOT) is a very challenging task yet of fundamental importance for many practical applications. In this paper, we focus on the problem of tracking multiple players in sports video which is even more difficult due to the abrupt movements of players and their complex interactions. To handle the difficulties in this problem, we present a new MOT algorithm which contributes both in the observation modeling level and in the tracking strategy level. For the observation modeling, we develop a progressive observation modeling process that is able to provide strong tracking observations and greatly facilitate the tracking task. For the tracking strategy, we propose a dual-mode two-way Bayesian inference approach which dynamically switches between an offline general model and an online dedicated model to deal with single isolated object tracking and multiple occluded object tracking integrally by forward filtering and backward smoothing. Extensive experiments on different kinds of sports videos, including football, basketball, as well as hockey, demonstrate the effectiveness and efficiency of the proposed method.

In this paper, a novel human articulated pose estimation method based on AdaBoost algorithm is presented. The human articulated pose is estimated by locating major human joint positions. We learn the classifiers on a normalized image for classifying each pixel position into a certain category. Two different kinds of classifiers, bottom-up joint position classifier and top-down skeleton classifier, are combined to achieve final results. HOG (Histogram of Oriented Gradient) feature is used for training both classifiers. Our human pose estimation system consists of three models, human detection, view classification, and pose estimation. The implemented system can automatically estimate human pose of different views. Experiment results are reported to show our proposed method can work on relatively small-size human images without using human silhouettes as a prerequisite, which is very efficient, robust and accurate enough for potential applications in visual surveillance.

This paper focuses on the problem of tracking multiple humans in dense environments which is very challenging due to recurring occlusions between different humans. To cope with the difficulties it presents, an offline boosted multi-view upper-body detector is used to automatically initialize a new human trajectory and is capable of dealing with partial human occlusions. What is more, an online learning process is proposed to learn discriminative human observations, including discriminative interest points and color patches, to effectively track each human when even more occlusions occur. The offline and online observation models are neatly integrated into the particle filter framework to robustly track multiple highly interactive humans. Experiments results on CAVIAR dataset as well as many other challenging real-world cases demonstrate the effectiveness of the proposed method.

In this paper, a novel feature named adaptive contour feature (ACF) is proposed for human detection and segmentation. This feature consists of a chain of a number of granules in oriented granular space (OGS) that is learnt via the AdaBoost algorithm. Three operations are defined on the OGS to mine object contour feature and feature co-occurrences automatically. A heuristic learning algorithm is proposed to generate an ACF that at the same time define a weak classifier for human detection or segmentation. Experiments on two open datasets show that the ACF outperform several well-known existing features due to its stronger discriminative power rooted in the nature of its flexibility and adaptability to describe an object contour element.

This paper presents an online detection-based two-stage multi-object tracking method in dense visual surveillances scenarios with a single camera. In the local stage, a particle filter with observer selection that could deal with partial object occlusion is used to generate a set of reliable tracklets. In the global stage, the detection responses are collected from a temporal sliding window to deal with ambiguity caused by full object occlusion to generate a set of potential tracklets. The reliable tracklets generated in the local stage and the potential tracklets generated within the temporal sliding window are associated by Hungarian algorithm on a modified pairwise tracklets association cost matrix to get the global optimal association. This method is applied to the pedestrian class and evaluated on two challenging datasets. The experimental results prove the effectiveness of our method.

This paper studies face recognition and person-specific face image retrieval in unconstrained environments. The proposed method consists of two parts: offline and online learning. In offline stage, we take advantage of both global and local features in a Bayesian framework for generic face recognition. In online stage, the offline learned classifier is adapted according to the query images of a given person, from which a person-specific face image retriever can be obtained. Our method is applied to the ldquolabeled faces in the wildrdquo dataset, which is more realistic than usual face recognition datasets. We show that the combination of offline and online learning can yield very promising results.

In this paper a novel ranking-based face recognition (FR) scheme is proposed. Compared with classical two-class (intra/extra person) and multi-class (each person a single class) schemes, the ranking-based method only takes into account the most relevant information in training data to find a solution, and therefore is more consistent with the objective of FR. In our approach, given a feature set and its similarity measure, all interested image pairs will be ordered by similarity. The solution to FR then becomes to explore a ranking function that can rank each intra-personal similarity prior to its relevant extra-personal similarities, which can be readily solved by rank boost algorithm. Furthermore in this paper, a logit-rank boost algorithm is proposed which can achieve better recognition performance, and a pruning technique is adopted to deal with the large amount of data that results in further improvement in recognition accuracy. Extensive experimental results on a consumer image collection and the FERET dataset are reported to show the effectiveness of our approach.

Real-time face alignment in video is very critical in many applications such as facial expression analysis, driver fatigue monitoring, etc. This paper presents a real time algorithm for face alignment in video that combines Active Shape Model (ASM) based face alignment and spatial-temporal continuity based tracking strategy. To guarantee the correctness of the tracked shape in each frame, a verification procedure is introduced so that when inter-frame shape tracking failed the intra-frame ASM algorithm can be restored to initialize a new shape for tracking. Experiments show that the implemented system can run totally automatic with a quite good accuracy that may have many practical applications.

For an image, texture unit (TU) is a small complete unit which characterizes the local texture of a given pixel and its neighborhood. Recently, TU-based approaches have been widely used in face recognition. This paper proposes a novel face representation and recognition approach based on TU. We make three major contributions: (1) we introduce a novel TU feature, Local Gabor Quarternary Pattern (LGQP), which incorporates both Gabor magnitude and phase information in a single TU code; (2) similarity measure of two TU images is treated as a tracking problem between two images, and we present a novel point-to-point matching (PPM) approach for TU similarity measure; (3) based on an integral histogram technique, the PPM similarity can be computed very rapidly. Experimental results on CMU-PIE and FERET data sets show that our method is able to reach very promising results.

In this paper, we present an approach for eye state recognition and closed-eye photo correction. For eye state recognition, AdaBoosted cascade open-eye detectors of different scales are trained. For closed-eye photo correction, a PCA generative model of concatenated corresponding closed-eye and open-eye texture patterns is built, and given a closed-eye texture pattern, an algorithm is proposed to recover its corresponding open-eye one for closed-eye replacement. Experiments on popular consumer images show our open-eye detectors achieved 94.71% correct recognition rate, and the closed-eye photo correction looks very natural.

Face alignment is a critical problem in many face related applications such as facial expression analysis, face recognition, etc. This paper presents a novel real time face alignment algorithm based on active shape model (ASM). In our algorithm, local textures of each label point is used to predict the displacement of each label point by applying boost non-linear regressions on the local search stage of the ASM framework. Experiments on different datasets show that our algorithm is much faster in speed and more robust to the initialization than previous ASM method.

Tracking objects in low frame rate (LFR) video or with abrupt motion poses two main difficulties which most conventional tracking methods can hardly handle: 1) poor motion continuity and increased search space; 2) fast appearance variation of target and more background clutter due to increased search space. In this paper, we address the problem from a view which integrates conventional tracking and detection, and present a temporal probabilistic combination of discriminative observers of different lifespans. Each observer is learned from different ranges of samples, with different subsets of features, to achieve varying levels of discriminative power at varying cost. An efficient fusion and temporal inference is then done by a cascade particle filter which consists of multiple stages of importance sampling. Experiments show significantly improved accuracy of the proposed approach in comparison with existing tracking methods, under the condition of LFR data and abrupt motion of both target and camera.

Tracking object in low frame rate video or with abrupt motion poses two main difficulties which conventional tracking methods can barely handle: 1) poor motion continuity and increased search space; 2) fast appearance variation of target and more background clutter due to increased search space. In this paper, we address the problem from a view which integrates conventional tracking and detection, and present a temporal probabilistic combination of discriminative observers of different lifespans. Each observer is learned from different ranges of samples, with different subsets of features, to achieve varying level of discriminative power at varying cost. An efficient fusion and temporal inference is then done by a cascade particle filter which consists of multiple stages of importance sampling. Experiments show significantly improved accuracy of the proposed approach in comparison with existing tracking methods, under the condition of low frame rate data and abrupt motion of both target and camera.

In recent years, boosting has been successfully applied to many practical problems in pattern recognition and computer vision fields such as object detection and tracking. As boosting is an offline training process with beforehand collected data, once learned, it cannot make use of any newly arriving ones. However, an offline boosted detector is to be exploited online and inevitably there must be some special cases that are not covered by those beforehand collected training data. As a result, the inadaptable detector often performs badly in diverse and changeful environments which are ordinary for many real-life applications. To alleviate this problem, this paper proposes an incremental learning algorithm to effectively adjust a boosted strong classifier with domain-partitioning weak hypotheses to online samples, which adopts a novel approach to efficient estimation of training losses received from offline samples. By this means, the offline learned general-purpose detectors can be adapted to special online situations at a low extra cost, and still retains good generalization ability for common environments. The experiments show convincing results of our incremental learning approach on challenging face detection problems with partial occlusions and extreme illuminations.

Different from appearance-based methods, clustering feature points only by their motion coherence is an emerging category of approach to detecting and tracking individuals among crowds. This paper reformalizes the problem and models a novel objective function for clustering with potential functions as in conditional random field approach. The merits include: (1) it integrates motion, spatial, temporal information; (2) the parameters are automatically obtained by supervised learning; (3) the objective function is based on feature-pair information, which enables effective learning on small amount of training data, as well as very fast online processing speed. Detection ROC curves are given on several datasets (including the CAVIAR set).

In this paper we propose a domain partitioning RankBoost approach for face recognition. This method uses Local Gabor Binary Pattern Histogram (LGBPH) features for face representation, and adopts RankBoost to select the most discriminative features. Unlike the original RankBoost algorithm in Freund et al. (2003), weak hypotheses in our method make their predictions based on a partitioning of the similarity domain. Since the domain partitioning approach handles the loss function of a ranking problem directly, it can achieve a higher convergence speed than the original approach. Furthermore, in order to improve the algorithm's generalization ability, we introduce some constraints to the weak classifiers being searched. Experiment results on FERET database show the effectiveness of our approach.

Rotation invariant multiview face detection (MVFD) aims to detect faces with arbitrary rotation-in-plane (RIP) and rotation-off-plane (ROP) angles in still images or video sequences. MVFD is crucial as the first step in automatic face processing for general applications since face images are seldom upright and frontal unless they are taken cooperatively. In this paper, we propose a series of innovative methods to construct a high-performance rotation invariant multiview face detector, including the Width-First-Search (WFS) tree detector structure, the Vector Boosting algorithm for learning vector-output strong classifiers, the domain-partition-based weak learning method, the sparse feature in granular space, and the heuristic search for sparse feature selection. As a result of that, our multiview face detector achieves low computational complexity, broad detection scope, and high detection accuracy on both standard testing sets and real-life images.

In this paper, we propose a novel tree-structured multiview face detector (MVFD), which adopts the coarse-to-fine strategy to divide the entire face space into smaller and smaller subspaces. For this purpose, a newly extended boosting algorithm named vector boosting is developed to train the predictors for the branching nodes of the tree that have multicomponents outputs as vectors. Our MVFD covers a large range of the face space, say, +/-45/spl deg/ rotation in plane (RIP) and +/-90/spl deg/ rotation off plane (ROP), and achieves high accuracy and amazing speed (about 40 ms per frame on a 320 /spl times/ 240 video sequence) compared with previous published works. As a result, by simply rotating the detector 90/spl deg/, 180/spl deg/ and 270/spl deg/, a rotation invariant (360/spl deg/ RIP) MVFD is implemented that achieves real time performance (11 fps on a 320 /spl times/ 240 video sequence) with high accuracy.

In this paper, a multi-view face pose classification method is introduced. Each face region is normalized by two eye-centers and mouth center, and then multi-class classifiers are trained for face pose classification. The face pose classification is organized into a tree structure that deals with off-image plane pose variation. As a specific application in video, it can estimate the face pose in each frame that results in a face pose variation trajectory, which may have important applications such as in security monitoring of car drivers. Experiment results on a very large set compared with a PCA reconstruction method as a benchmark are reported to show its effectiveness.

We propose a robust face alignment algorithm with a novel discriminative local texture model. Different from the conventional descriptive PCA local texture model in ASM, classifiers using LUT-type Haar-like features are trained from a large data set as local texture model. The strong discriminative power of the classifier greatly improves the accuracy and robustness of local searching on faces with expression variation and ambiguous contours. A Bayesian framework is configured for shape parameter optimization and the algorithm is implemented in a hierarchical structure for efficiency. Extensive experiments are reported to show its accuracy and robustness.

We propose an omni-directional face detection method based on the confidence-rated AdaBoost algorithm, called real AdaBoost, proposed by R.E. Schapire and Y. Singer (see Machine Learning, vol.37, p.297-336, 1999). To use real AdaBoost, we configure the confidence-rated look-up-table (LUT) weak classifiers based on Haar-type features. A nesting-structured framework is developed to combine a series of boosted classifiers into an efficient object detector. For omni-directional face detection, our method has achieved a rather high performance and the processing speed can reach 217 ms per 320/spl times/240 image. Experiment results on the CMU+MIT frontal and the CMU profile face test sets are reported to show its effectiveness.

In this paper, a novel method to learn the intrinsic object structure for robust visual tracking is proposed. The basic assumption is that the parameterized object state lies on a low dimensional manifold and can be learned from training data. Based on this assumption, firstly we derived the dimensionality reduction and density estimation algorithm for unsupervised learning of object intrinsic representation, the obtained non-rigid part of object state reduces even to 2 dimensions. Secondly the dynamical model is derived and trained based on this intrinsic representation. Thirdly the learned intrinsic object structure is integrated into a particle-filter style tracker. We will show that this intrinsic object representation has some interesting properties and based on which the newly derived dynamical model makes particle-filter style tracker more robust and reliable. Experiments show that the learned tracker performs much better than existing trackers on the tracking of complex non-rigid motions such as fish twisting with self-occlusion and large inter-frame lip motion. The proposed method also has the potential to solve other type of tracking problems.

In this paper a new condensation style contour tracking method called probabilistic dynamic contour (PDC) is proposed for lip tracking: a novel mixture dynamic model is designed to represent shape more compactly and to tolerate larger motions between frames, a measurement model is designed to include multiple visual cues. The proposed PDC tracker has the advantage that it is conceptually general but effectively suitable for lip tracking with the designed dynamic and measurement model. The new tracker improves the traditional condensation style tracker in three aspects: Firstly, the dynamic model is partially derived from the image sequence, so the tracker does not need to learn the dynamics in advance. Secondly, the measurement model is easy to be updated during tracking, which avoids modeling the foreground object in prior. Thirdly, to improve the tracker's speed, a compact representation of shape and a noise model are proposed to reduce the samples required to represent the posterior distribution. An experiment on lip contour tracking shows that the proposed method tracks contours robustly as well as accurately compared to the existing tracking method.

A face detection algorithm integrating template matching and support vector machines (SVM) is presented. Two types of templates: eyes-in-whole and face itself, are used for coarse filtering, and the SVM classifier is used for classification. A bootstrap method is used to collect non-face samples for SVM training under a template matching constrained subspace, which greatly reduces the complexity of training the SVM. Comparative experimental results demonstrate its effectiveness.

This paper addresses the subject of stereo matching between the corners of two perspectives. Similarity-based matching is prone to errors, and the existing algorithms reject outliers but never correct them. Frequently, in fact, the correct correspondence is not found at the single point with the largest similarity; but it lies among a few points with large value. So we propose a new algorithm, which first selects several candidates for each corner and then optimizes the whole match with global constraints. The algorithm increases remarkably not only the percentage of correctness, but also the number of correct matches. To expedite the optimization, we apply group evolution. A simple directional constraint is used as criteria for evaluation, which avoids the estimation of epipolar lines. The principles and applicable cases are presented. Results are provided for corners, retrieved both manually and automatically, in real images.