Fair face recognition is all about learning invariant feature that generalizes to unseen faces in any demographic group. Unfortunately, face datasets inevitably capture the imbalanced demographic attributes that are ubiquitous in real-world observations, and the model learns biased feature that generalizes poorly in the minority group. We point out that the bias arises due to the confounding demographic attributes, which mislead the model to capture the spurious demographic-specific feature. The confounding effect can only be removed by causal intervention, which requires the confounder annotations. However, such annotations can be prohibitively expensive due to the diversity of the demographic attributes. To tackle this, we propose to generate diverse data partitions iteratively in an unsupervised fashion. Each data partition acts as a self-annotated confounder, enabling our Invariant Feature Regularization (Inv-Reg) to deconfound. Inv-Reg is orthogonal to existing methods, and combining Inv-Reg with two strong baselines (Arcface and CIFP) leads to new state-of-the-art that improves face recognition on a variety of demographic groups. Code is available at https://github.com/milliema/InvReg.

Pose variation is one key challenge in face recognition. As opposed to current techniques for pose invariant face recognition, which either directly extract pose invariant features for recognition, or first normalize profile face images to frontal pose before feature extraction, we argue that it is more desirable to perform both tasks jointly to allow them to benefit from each other. To this end, we propose a Pose Invariant Model (PIM) for face recognition in the wild, with three distinct novelties. First, PIM is a novel and unified deep architecture, containing a Face Frontalization sub-Net (FFN) and a Discriminative Learning sub-Net (DLN), which are jointly learned from end to end. Second, FFN is a well-designed dual-path Generative Adversarial Network (GAN) which simultaneously perceives global structures and local details, incorporated with an unsupervised cross-domain adversarial training and a "learning to learn" strategy for high-fidelity and identity-preserving frontal view synthesis. Third, DLN is a generic Convolutional Neural Network (CNN) for face recognition with our enforced cross-entropy optimization strategy for learning discriminative yet generalized feature representation. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks demonstrate the superiority of the proposed model over the state-of-the-arts.

Learning from synthetic faces, though perhaps appealing for high data efficiency, may not bring satisfactory performance due to the distribution discrepancy of the synthetic and real face images. To mitigate this gap, we propose a 3D-Aided Deep Pose-Invariant Face Recognition Model (3D-PIM), which automatically recovers realistic frontal faces from arbitrary poses through a 3D face model in a novel way. Specifically, 3D-PIM incorporates a simulator with the aid of a 3D Morphable Model (3D MM) to obtain shape and appearance prior for accelerating face normalization learning, requiring less training data. It further leverages a global-local Generative Adversarial Network (GAN) with multiple critical improvements as a refiner to enhance the realism of both global structures and local details of the face simulator’s output using unlabelled real data only, while preserving the identity information. Qualitative and quantitative experiments on both controlled and in-the-wild benchmarks clearly demonstrate superiority of the proposed model over state-of-the-arts.

Video-based person re-identification plays a central role in realistic security and video surveillance. In this paper, we propose a novel accumulative motion context (AMOC) network for addressing this important problem, which effectively exploits the long-range motion context for robustly identifying the same person under challenging conditions. Given a video sequence of the same or different persons, the proposed AMOC network jointly learns appearance representation and motion context from a collection of adjacent frames using a two-stream convolutional architecture. Then, AMOC accumulates clues from motion context by recurrent aggregation, allowing effective information flow among adjacent frames and capturing dynamic gist of the persons. The architecture of AMOC is end-to-end trainable, and thus, motion context can be adapted to complement appearance clues under unfavorable conditions (e.g., occlusions). Extensive experiments are conduced on three public benchmark data sets, i.e., the iLIDS-VID, PRID-2011, and MARS data sets, to investigate the performance of AMOC. The experimental results demonstrate that the proposed AMOC network outperforms state-of-the-arts for video-based re-identification significantly and confirm the advantage of exploiting long-range motion context for video-based person re-identification, validating our motivation evidently.

We investigate the problem of person search in the wild in this work. Instead of comparing the query against all candidate regions generated in a query-blind manner, we propose to recursively shrink the search area from the whole image till achieving precise localization of the target person, by fully exploiting information from the query and contextual cues in every recursive search step. We develop the Neural Person Search Machines (NPSM) to implement such recursive localization for person search. Benefiting from its neural search mechanism, NPSM is able to selectively shrink its focus from a loose region to a tighter one containing the target automatically. In this process, NPSM employs an internal primitive memory component to memorize the query representation which modulates the attention and augments its robustness to other distracting regions. Evaluations on two benchmark datasets, CUHK-SYSU Person Search dataset and PRW dataset, have demonstrated that our method can outperform current state-of-the-arts in both mAP and top-1 evaluation protocols.

Synthesizing realistic profile faces is promising for more efficiently training deep pose-invariant models for large-scale unconstrained face recognition, by populating samples with extreme poses and avoiding tedious annotations. However, learning from synthetic faces may not achieve the desired performance due to the discrepancy between distributions of the synthetic and real face images. To narrow this gap, we propose a Dual-Agent Generative Adversarial Network (DA-GAN) model, which can improve the realism of a face simulator's output using unlabeled real faces, while preserving the identity information during the realism refinement. The dual agents are specifically designed for distinguishing real v.s. fake and identities simultaneously. In particular, we employ an off-the-shelf 3D face model as a simulator to generate profile face images with varying poses. DA-GAN leverages a fully convolutional network as the generator to generate high-resolution images and an auto-encoder as the discriminator with the dual agents. Besides the novel architecture, we make several key modifications to the standard GAN to preserve pose and texture, preserve identity and stabilize training process: (i) a pose perception loss; (ii) an identity perception loss; (iii) an adversarial loss with a boundary equilibrium regularization term. Experimental results show that DA-GAN not only presents compelling perceptual results but also significantly outperforms state-of-the-arts on the large-scale and challenging NIST IJB-A unconstrained face recognition benchmark. In addition, the proposed DA-GAN is also promising as a new approach for solving generic transfer learning problems more effectively.

Faces in the wild are usually captured with various poses, illuminations and occlusions, and thus inherently multimodally distributed in many tasks. We propose a conditional Convolutional Neural Network, named as c-CNN, to handle multimodal face recognition. Different from traditional CNN that adopts fixed convolution kernels, samples in c-CNN are processed with dynamically activated sets of kernels. In particular, convolution kernels within each layer are only sparsely activated when a sample is passed through the network. For a given sample, the activations of convolution kernels in a certain layer are conditioned on its present intermediate representation and the activation status in the lower layers. The activated kernels across layers define the sample-specific adaptive routes that reveal the distribution of underlying modalities. Consequently, the proposed framework does not rely on any prior knowledge of modalities in contrast with most existing methods. To substantiate the generic framework, we introduce a special case of c-CNN via incorporating the conditional routing of the decision tree, which is evaluated with two problems of multimodality - multi-view face identification and occluded face verification. Extensive experiments demonstrate consistent improvements over the counterparts unaware of modalities.

An unsupervised approach to segment the swimmer in hydrodynamic video sequences is presented in this paper. Detecting and segmenting objects in hydrodynamic scenes is challenging as both the desired foreground and unwanted background are driven by complex nonlinear dynamics. These dynamics typically comprise a chaotic mix of vortices, flows and turbulence. Medium introduced reflection and light diffraction creates the camouflage, rendering the swimmer detection even more difficult. On the other hand, spatial regularities and temporal distortions within these sequences often exhibit signature-like patterns. The novelty of our approach is to combine texture modeling and photometric features in order to identify the foreground subject, which lacks the visual characteristics of hydrodynamic content. Segmentation results are then temporally propagated from one frame to another to reduce the effects of camouflage and confine the segmentation to desired region. Simulation results demonstrate ability of the algorithm to separate human body in hydrodynamic sequences for silhouette based underwater motion capture and analysis.

In this paper we present a novel model-based hybrid technique for user localization and drift-free tracking in urban environments. In outdoor augmented reality, instantaneous 6-DoF user localization is achieved with position and orientation sensors such as GPS and gyroscopes. Initial pose obtained with these sensors is dominated by large positional errors due to coarse granularity of GPS data. Subsequent tracking is also erroneous as gyroscopes are prone to drifts and often need recalibration. We propose to use model-to-image registration technique to refine initial rough estimate for accurate user localization. Large positional errors in user localization are mitigated by aligning silhouettes of the model with that of the camera image using shape context descriptors as they are invariant to translation, scale and rotational errors. Once initialized, drift-free tracking is achieved by combining frame-to-frame and model-to-frame feature tracking. Frame-to-frame tracking is done by matching corners whereas edges are used for model-to-frame silhouette tracking. Final camera pose is obtained with M-estimators.

This paper presents a novel approach for user positioning, robust tracking and online 3D mapping for outdoor augmented reality applications. As coarse user pose obtained from GPS and orientation sensors is not sufficient for augmented reality applications, sub-meter accurate user pose is then estimated by a one-step silhouette matching approach. Silhouette matching of the rendered 3D model and camera data is carried out with shape context descriptors as they are invariant to translation, scale and rotational errors, giving rise to a non-iterative registration approach. Once the user is correctly positioned, further tracking is carried out with camera data alone. Drifts associated with vision based approaches are minimized by combining different feature modalities. Robust visual tracking is maintained by fusing frame-to-frame and model-to-frame feature matches. Frame-to-frame tracking is accomplished with corner matching while edges are used for model-to-frame registration. Results from individual feature tracker are fused using a pose estimate obtained from an extended Kalman filter (EKF) and a weighted M-estimator. In scenarios where dense 3D models of the environment are not available, online 3D incremental mapping and tracking is proposed to track the user in unprepared environments. Incremental mapping prepares the 3D point cloud of the outdoor environment for tracking.

A fully automatic 3D-2D articulated registration algorithm for aligning a whole-body human geometry to a 2D X-ray image is presented. Domain prior in the form of a hierarchical model encapsulating the kinematic structure of the human body is exploited. A deformation operator is then defined with respect to the hierarchical model. An iterative search using ICP is performed to estimate the optimal deformation parameters. Empirical results indicate good matching rates of 94% and above. Requiring minimal user intervention, the method is suited for large-scale data acquisition in human anthropometric studies.

Despite leaps in motion capture technology, the dichotomy between unencumbered vision-based motion recovery and the prevailing marker-assisted motion capture solution remains largely pertinent. This paper bridges the gap by introducing an attachment-free, full-body motion capture technique that employs multiple high-speed cameras and a customized bodysuit. The flexible bodysuit allows free movements while providing marker data for guided reconstruction of pose and motion. Our method overcomes several practical problems in existing systems, including long preparation time and displaced markers. The network of markerized pattern provides connectivity information and enhances robustness in feature tracking, model matching and 3D reconstruction.

Reliable transmission of compressed video is not possible without error recovery mechanisms as the existing communication channels are lossy. In this paper, we present an effective method of increasing the error resilience of the coded video data for transmission over any lossy channel for videoconferencing applications. The proposed content-based robust video coding method takes into account the domain knowledge of the video, typical of videoconferencing applications, to code "some" macroblocks in INTRA mode so that the effect of transmission errors on subjective quality are minimized.

Human face images form the important database in police departments, banks, security kiosks, and they are also found in abundance in day-to-day life. In these databases the important content, of course, is the face region. We present a highly efficient system that detects the human faces in the wavelet transform for discriminative quantization to achieve high perceptual quality content-based image coding technique. The proposed method gives superior subjective performance over JPEG without sacrificing the performance in the rate-distortion spectrum.

In this paper we present a new hybrid compression scheme for videoconferencing and videotelephony applications at very low bit rates (i.e., 32 kbits/s). In these applications, human face is the most important region within a frame and should be coded with high fidelity. To preserve perceptually important information at low bit rates, such as face regions, skin-tone is used to detect and adaptively quantize these regions. Novel features of this coder are the use of overlapping block motion compensation in combination with discrete wavelet transform, followed by zerotree entropy coding with new scanning procedure of wavelet blocks such that the rest of the H.263 framework can be used. At the same total bit-rate, coarser quantization of the background enables the face region to be quantized finely and coded with higher quality. The simulation results demonstrates comparable objective and superior subjective performance when compared with H.263 video coding standard, while providing the advanced feature like scalability functionalities.

Two new schemes are presented for finding human faces in a photograph. The first scheme approximates the unknown distributions of the face and the face-like manifolds wing higher order statistics (HOS). An HOS-based data clustering algorithm is also proposed. In the second scheme, the face to non-face and non-face to face transitions are learnt using a hidden Markov model (HMM). The HMM parameters are estimated corresponding to a given photograph and the faces are located by examining the optimal state sequence of the HMM. Experimental results are presented on the performance of both the schemes.

The embedded zerotree of the wavelet coefficient (EZW) algorithm has become an effective way of compressing images. The use of multistage vector quantizer (MSVQ) provides successive-approximation coding for vectors. A new algorithm is proposed that, provides good quality of reconstructed images at very low bit rates. The algorithm uses successive-approximation quantization of both scalars and vectors on the wavelet coefficients of the image. Successive-approximation quantization of scalars and vectors is done using EZW and MSVQ algorithms respectively. The EZW algorithm is applied to wavelet coefficients belonging to coarser level subbands and MSVQ is applied to vectors of wavelet coefficients belonging to finer level subbands. The proposed method further uses static Huffman coding to achieve more compression.