Fusion has been widely used in machine learning community, especially for problems dealing with multiple input sources and classifiers. The general strategy for information fusion in deep neural network is to directly concatenate the embedding features on the latent space of input sources. However, it is very hard to capture the relative importance of fused sources. It is also impossible to learn the correlation among fused multimodalities inputs, e.g., intra-class and inter-class similarities. Besides, most existing deep learning fusion approaches use universal fusion weights strategy, which cannot fully exploit the relative importance of different inputs. In order to address these problems, in this work we propose an Adaptive Weighted Deep Fusion scheme (AWDF) to capture potential relationships among various input sources. It integrates the feature level and decision level fusion in one framework. Furthermore, in order to address the limitations of existing fusing models with fixed weights, we propose a new scheme named Cross Decision Weights Method (CDWM). It can dynamically learn the weight for each input branch during the fusion process instead of utilizing pre-defined weights. To evaluate the performance of AWDF, we conduct experiments on three different real-world datasets: Wild Business Terms (WBT) Dataset, Iceberg Detection Dataset and CareerCon Dataset. Our experimental results demonstrate the superiority of AWDF over other fusion approaches.

Portrait segmentation has gained more and more attractions in recent years due to the popularity of selfie images. Compared to general semantic segmentation problems, portrait segmentation focuses on facial areas with higher requirements especially over the boundaries. To improve the performance of portrait segmentation, we propose a boundary-sensitive deep neural network (BSN) for better accuracy among the portrait boundaries. BSN introduces three novel techniques. First, an individual boundary-sensitive mask is proposed by dilating the contour line and assigning the boundary pixels with multi-class labels. Second, a global boundary-sensitive mask is employed as a position sensitive prior to further constrain the overall shape of the segmentation map. Third, we train a boundary-sensitive attribute classifier jointly with the segmentation network to reinforce the network with semantic boundary shape information. We have evaluated BSN on the state-of-the-art public portrait segmentation datasets, i.e., the PFCN dataset, as well as the portrait images collected from other three popular image segmentation datasets: COCO, COCO-Stuff, and PASCAL VOC. Our method achieves the superior quantitative and qualitative performance over state-of-the-arts on the evaluated datasets, especially obtains better visualization effect on the portrait boundary region.

Kinship verification aims to identify the kin relation between two given face images. It is a very challenging problem due to the lack of training data and facial similarity variations between kinship pairs. In this work, we build a novel appearance and shape based deep learning pipeline. First we adopt the knowledge learned from general face recognition network to learn general facial features. Afterwards, we learn kinship oriented appearance and shape features from kinship pairs and combine them for the final prediction. We have evaluated the model performance on a widely used popular benchmark and demonstrated the superiority over the state-of-the-art.

Deploying deep neural networks on mobile devices is a challenging task. Current model compression methods such as matrix decomposition effectively reduce the deployed model size, but still cannot satisfy real-time processing requirement. This paper first discovers that the major obstacle is the excessive execution time of non-tensor layers such as pooling and normalization without tensor-like trainable parameters. This motivates us to design a novel acceleration framework: DeepRebirth through "slimming" existing consecutive and parallel non-tensor and tensor layers. The layer slimming is executed at different substructures: (a) streamline slimming by merging the consecutive non-tensor and tensor layer vertically; (b) branch slimming by merging non-tensor and tensor branches horizontally. The proposed optimization operations significantly accelerate the model execution and also greatly reduce the run-time memory cost since the slimmed model architecture contains less hidden layers. To maximally avoid accuracy loss, the parameters in new generated layers are learned with layer-wise fine-tuning based on both theoretical analysis and empirical verification. As observed in the experiment, DeepRebirth achieves more than 3x speed-up and 2.5x run-time memory saving on GoogLeNet with only 0.4% drop on top-5 accuracy in ImageNet. Furthermore, by combining with other model compression techniques, DeepRebirth offers an average of 106.3ms inference time on the CPU of Samsung Galaxy S5 with 86.5% top-5 accuracy, 14% faster than SqueezeNet which only has a top-5 accuracy of 80.5%.

Machine lipreading is a special type of automatic speech recognition (ASR) which transcribes human speech by visually interpreting the movement of related face regions including lips, face, and tongue. Recently, deep neural network based lipreading methods show great potential and have exceeded the accuracy of experienced human lipreaders in some benchmark datasets. However, lipreading is still far from being solved, and existing methods tend to have high error rates on the wild data. In this paper, we propose LCANet, an end-to-end deep neural network based lipreading system. LCANet encodes input video frames using a stacked 3D convolutional neural network (CNN), highway network and bidirectional GRU network. The encoder effectively captures both short-term and long-term spatio-temporal information. More importantly, LCANet incorporates a cascaded attention-CTC decoder to generate output texts. By cascading CTC with attention, it partially eliminates the defect of the conditional independence assumption of CTC within the hidden neural layers, and this yields notably performance improvement as well as faster convergence. The experimental results show the proposed system achieves a 1.3% CER and 3.0% WER on the GRID corpus database, leading to a 12.3% improvement compared to the state-of-the-art methods.

Face detection is a widely studied problem over the past few decades. Recently, significant improvements have been achieved via the deep neural network, however, it is still challenging to directly apply these techniques to mobile devices for its limited computational power and memory. In this work, we present a proposal generation acceleration framework for real-time face detection. More specifically, we adopt a popular cascaded convolutional neural network (CNN) as the basis, then apply our acceleration approach on the basic framework to speed up the model inference time. We are motivated by the observation that the computation bottleneck of this framework arises from the proposal generation stage, where each level of the dense image pyramid has to go through the network. In this work, we reduce the number of image pyramid levels by utilizing both global and local facial characteristics (i.e., global face and facial parts). Experimental results on public benchmarks WIDER-face and FDDB demonstrate the satisfactory performance and faster speed compared to the state-of-the-arts.

Recurrent neural networks (RNNs) achieve cutting-edge performance on a variety of problems. However, due to their high computational and memory demands, deploying RNNs on resource constrained mobile devices is a challenging task. To guarantee minimum accuracy loss with higher compression rate and driven by the mobile resource requirement, we introduce a novel model compression approach DirNet based on an optimized fast dictionary learning algorithm, which 1) dynamically mines the dictionary atoms of the projection dictionary matrix within layer to adjust the compression rate 2) adaptively changes the sparsity of sparse codes cross the hierarchical layers. Experimental results on language model and an ASR model trained with a 1000h speech dataset demonstrate that our method significantly outperforms prior approaches. Evaluated on off-the-shelf mobile devices, we are able to reduce the size of original model by eight times with real-time model inference and negligible accuracy loss.

Human age estimation plays an important role inhuman facial image analysis. Aging feature representation is one of the widely studied problems in this topic. Convolutional map (bio-inspired features, or BIF) has been proven to be the most successful framework, but its manual crafted filters cannot easily capture the complicated facial aging pattern. In this paper, we adopt this convolutional map framework but propose a novel feature learning approach based on convolutional sparse coding (CSC) that can automatically learn to characterize aging signatures. Compared to other popular feature learning approaches like deep convolutional neural network (CNN), we verify that our learning approach can extract localized subtle aging features like CNN, and also significantly reduce the model size. Moreover, we employ the standard deviation (STD)pooling to summarize the aging feature. Finally, the extracted features are fed into a discriminative manifold learning model to obtain more discriminative low-dimensional representations and further improve the computational efficiency. We evaluate our approach over the standard benchmark datasets. The experimental results demonstrate that our approach impressively out performs the state-of-the-art results. The proposed age estimation scheme also performs well in the cross-database age estimation task.

Facial aging is a complicated process which usually affects the facial appearance (e.g., wrinkles). Variations of facial appearance pose a big challenge to the automatic face recognition problem. How to eliminate the influence of aging factors to the verification performance is a very challenging problem. Multi-task learning has provided a principled framework for jointly learning multiple related tasks to improve generalization performance. In this paper, we leverage this powerful technique to improve the task of cross-age face verification. We present an end-to-end learning framework for cross-age face verification by designing a multi-task deep neural network architecture that exploits the intrinsic low-dimensional representation shared between the tasks of face verification and age estimation. We show that the algorithm effectively balances feature sharing and feature exclusion between the two given tasks. We evaluate the proposed framework on two standard benchmarks. Experimental results demonstrate that our algorithm has significant improvement over the state-of-theart (2.2% EER on MORPH and 7.8% EER on FG-NET, by more than 50.0% and 59.70% performance gain respectively).

Objects in fine-grained categories always share a high degree of shape similarity, making both “localizing discriminative parts” and “learning appearance descriptors” extremely difficult. We propose a framework to leverage 2D+3D cues to handle above two challenges. Towards the goal of image alignment to localize discriminative parts, traditional methods rely on either manual part annotation or image segmentation. Instead, our framework leverages each image's 3D camera pose estimation to align images; Towards the goal of “learning appearance descriptors” confined with small training data and memory/computation cost, we propose an unsupervised Convolutional Sparse Coding (CSC) + manifold learning that significantly reduces model complexity, but still successfully produces highly diverse feature filters like deep neural network. Our experimental results attest the advocated framework's accuracy is comparable to a deep network, demonstrating its great potential on mobile devices.

Human age provides key demographic information. It is also considered as an important soft biometric trait for human identification or search. Compared to other pattern recognition problems (e.g., object classification, scene categorization), age estimation is much more challenging since the difference between facial images with age variations can be more subtle and the process of aging varies greatly among different individuals. In this work, we investigate deep learning techniques for age estimation based on the convolutional neural network (CNN). A new framework for age feature extraction based on the deep learning model is built. Compared to previous models based on CNN, we use feature maps obtained in different layers for our estimation work instead of using the feature obtained at the top layer. Additionally, a manifold learning algorithm is incorporated in the proposed scheme and this improves the performance significantly. Furthermore, we also evaluate different classification and regression schemes in estimating age using the deep learned aging pattern (DLA). To the best of our knowledge, this is the first time that deep learning technique is introduced and applied to solve the age estimation problem. Experimental results on two datasets show that the proposed approach is significantly better than the state-of-the-art.

Kinship verification has become a very active topic recently. Many kinship verification algorithms have been proposed, however, many problems still need to be solved, such as how to locate familial traits of two individuals with kinship and how to make use of facial geometry information to verify kinship relation. To solve these problems, we propose one feature matching scheme to locate familial traits between two facial images. We also advocate a method using geometry information for kinship verification. This approach achieves more than 17% improvement compared with the state of the art. In addition, preliminary results show that the proposed problems can be accurately addressed when fusing appearance and geometry feature.

Object recognition is an important problem with a wide range of applications. It is also a challenging problem, especially for animal categorization as the differences among breeds can be subtle. In this paper, based on statistical techniques for landmark-based shape representation, we propose to model the shape of dog breed as points on the Grassmann manifold. We consider the dog breed categorization as the classification problem on this manifold. The proposed scheme is tested on a dataset including 8,351 images of 133 different breeds. Experimental results demonstrate the advocated scheme outperforms state of the art approaches by nearly 20%.

Automatic human age estimation has attracted a great deal of interest in the past few years. Although many advancements have been made by researchers, there are still many challenges: such as age estimation across different image acquisition methods, different expressions, gender and races. The influence due to race and gender seems to be the most common issue, because collecting a large amount of face images with comprehensive racial diversities seems impractical. The performance will degrade when estimating face images of races that differ from the training set. In this work, we present a new scheme to mitigate the influences of race and gender in the problem of age estimation. Our system will contribute a robust solution to solve the problem of age estimation across races and genders. This study is essential for developing a practical age estimation system (with mixture of races and gender.) To evaluate the performance of the proposed algorithm, we run comprehensive experiments on one widely used big database - MORPH-II, which contains more than 55, 000 images. On an average, an improvement of more than 20% has been achieved using the proposed scheme.

This paper proposes a novel scheme for face recognition from visible images to depth images. In our proposed technique, we adopt Partial Least Square (PLS) to handle correlation mapping between 2D to 3D. A considerable performance improvement is observed compared to using Canonical Correlation Analysis (CCA). To further improve the performance, a fusion scheme based on PLS and CCA is advocated. We evaluate the advocated approach on a popular face dataset-FRGCV2.0. Experimental results demonstrate that the proposed scheme is an effective approach to perform 2D-3D face recognition.

In this paper, we study human age estimation in face images under significant expression changes. We will address two issues: (1) Is age estimation affected by facial expression changes and how significant is the influence? (2) How to develop a robust method to perform age estimation undergoing various facial expression changes? This systematic study will not only discover the relation between age estimation and expression changes, but also contribute a robust solution to solve the problem of cross-expression age estimation. This study is an important step towards developing a practical and robust age estimation system that allows users to present their faces naturally (with various expressions) rather than constrained to the neutral expression only. Two databases originally captured in the Psychology community are introduced to Computer Vision, to quantitatively demonstrate the influence of expression changes on age estimation, and evaluate the proposed framework and corresponding methods for cross-expression age estimation.