Video question answering (VideoQA) is challenging as it requires reasoning about natural language and multimodal interactive relations. Most existing methods apply attention mechanisms to extract interactions between the question and the video or to extract effective spatio-temporal relational representations. However, these methods neglect the implication of relations between intra- and inter-modal interactions for multimodal learning, and they fail to fully exploit the synergistic effect of multiscale semantics in answer reasoning. In this article, we propose a novel hierarchical synergy-enhanced multimodal relational network (HMRNet) to address these issues. Specifically, we devise (i) a compact and unified relation-oriented interaction module that explores the relation between intra- and inter-modal interactions to enable effective multimodal learning; and (ii) a hierarchical synergistic memory unit that leverages a memory-based interaction scheme to complement and fuse multimodal semantics at multiple scales to achieve synergistic enhancement of answer reasoning. With careful design of each component, our HMRNet has fewer parameters and is computationally efficient. Extensive experiments and qualitative analyses demonstrate that the HMRNet is superior to previous state-of-the-art methods on eight benchmark datasets. We also demonstrate the effectiveness of the different components of our method.

X-ray security check is one of the most effective security measures widely used in airports, high-speed trains, subways, and other important places. Due to the difference in imaging mechanism between X-ray images and visible images, X-ray contraband detection suffers from problems of large intra-class differences and small inter-class differences. The Softmax loss does not encourage compactness within class and separability between classes explicitly, resulting in insensitivity to target. We address this problem by proposing an attention mechanism based ACD-Net, which can quickly focus on the obscured objects in luggage. To optimize inter-class interval, we evaluate impacts of several cosine-based loss function on X-ray contraband detection performance, including Cosine Embedding Loss, L-Softmax Loss, CosFace and ArcFace. Then we propose a cosine loss function DA-Loss focusing on intra-class compactness and inter-class difference at the same time. Extensive experiments show the superiority of our proposed ACD-Net model with DALoss classification loss function, which has better separability of large intra-class differences and small inter-class differences, and outperform the state-of-the-arts.

Video question answering (VideoQA) challenges the joint learning of visual and linguistic knowledge. Whilst the dynamic video-question interaction is not well explored in previous methods, the search for answer clues from textual semantics in the interaction is not valued. To address these issues, this paper proposes a novel Two-Stream Heterogeneous Graph Network (TSHGNet) using Dynamic Interactive Learning (DIL) to accomplish effective reasoning between videos and questions. Inspired by the way people answer questions, the two-stream architecture of TSHGNet is designed to extract question-driven visual cues and video-driven textual semantics, respectively. Therein, for each stream, DIL gradually refines the comprehension of question semantics and the extraction of visual representations through heterogeneous graph interactions. Extensive experiments and qualitative analyses demonstrate improved performances of the proposed TSHGNet on three benchmark datasets in comparison with previous state-of-the-art methods, and the effectiveness of different components of our method.

Single-view 3D human pose estimation (HPE) based on Graph Convolutional Networks (GCNs) currently suffers from problems such as insufficient spatial feature representation, difficult fusion of various information, and depth ambiguity in 2D to 3D pose mapping. This paper proposes a framework for monocular 3D human pose learning based on spatio-temporal attention graph. Firstly, we build a spatial graph feature acquisition scheme to obtain spatial semantic feature of 3D human pose with strong representativeness, by constructing a global to local attention graph through a coarse-to-fine way. And then we capture contextual information of temporal related images in the sequence as attention factors, evaluate their influence on the target image and achieve effective integration of spatio-temporal characteristics to mitigate the depth ambiguity problem. Extensive experimental results on two challenging benchmark datasets (Human3.6M and HumanEva-I) show that our method can effectively improve the accuracy of 3D HPE and outperform the state-of-the-arts.

Regression-based face alignment involves learning a series of mapping functions to predict the true landmarks from an initial estimation of the alignment. Most existing approaches focus on learning efficacious mapping functions from some feature representations to improve performance. The issues related to the initial alignment estimation and the final learning objective, however, receive less attention. This work proposes a deep regression architecture with progressive reinitialization and a new error-driven learning loss function to explicitly address the above two issues. Given an image with a rough face detection result, the full face region is first mapped by a supervised spatial transformer network to a normalized form and trained to regress coarse positions of landmarks. Then, different face parts are further respectively reinitialized to their own normalized states, followed by another regression sub-network to refine the landmark positions. To deal with the inconsistent annotations in existing training datasets, we further propose an adaptive landmark-weighted loss function. It dynamically adjusts the importance of different landmarks according to their learning errors during training without depending on any hyper-parameters manually set by trial and error. A high level of robustness to annotation inconsistencies is thus achieved. The whole deep architecture permits training from end to end, and extensive experimental analyses and comparisons demonstrate its effectiveness and efficiency. The source code, trained models, and experimental results are made available at https://github.com/shaoxiaohu/Face_Alignment_DPR.git.

Face frontalization has been widely used in face recognition to alleviate distribution discrepancy between multi-view faces. Given a profile face, existing models learn to synthesize a frontal face from the whole region indistinguishably, often resulting in unsatisfactory frontalization caused by a lack of synthetic focus and disturbances of trivial backgrounds. This paper proposes a novel Deep Attention-based Face Frontalization (DAFF) method to address the above issues explicitly. We first inject the 3D spatial prior of the input face into an encoder-decoder model. This process locates the discriminative foreground for decomposing meaningful convolutional embeddings. After that, we propose a novel objective that served as the generator’s geometric guidance to pay more attention to the target’s essential regions. Therefore, we can leverage the attentional constraints to perform recovery refinement at both embedding and texture levels. Extensive experiments show that DAFF achieves satisfactory frontalization and competitive recognition performance under constrained and in-the-wild benchmarks.

Regression based facial landmark detection methods usually learns a series of regression functions to update the landmark positions from an initial estimation. Most of existing approaches focus on learning effective mapping functions with robust image features to improve performance. The approach to dealing with the initialization issue, however, receives relatively fewer attentions. In this paper, we present a deep regression architecture with two-stage re-initialization to explicitly deal with the initialization problem. At the global stage, given an image with a rough face detection result, the full face region is firstly re-initialized by a supervised spatial transformer network to a canonical shape state and then trained to regress a coarse landmark estimation. At the local stage, different face parts are further separately re-initialized to their own canonical shape states, followed by another regression subnetwork to get the final estimation. Our proposed deep architecture is trained from end to end and obtains promising results using different kinds of unstable initialization. It also achieves superior performances over many competing algorithms.

Supervised deep learning models like convolutional neural network (CNN) have shown very promising results for the face recognition problem, which often require a huge number of labeled face images. Since manually labeling a large training set is a very difficult and time-consuming task, it is very beneficial if the deep model can be trained from face samples with only weak annotations. In this paper, we propose a general framework to train a deep CNN model with weakly labeled facial images that are available on the Internet. Specifically, we first design a deep Multiscale-Inception CNN (MICNN) architecture to exploit the multi-scale information for face recognition. Then, we train an initial MICNN model with only a limited number of labeled samples. After that, we propose a dual-level sample selection strategy to further fine-tune the MICNN model with the weakly labeled samples from both the sample level and class level, which aims to skip outliers and select more samples from confusing class pairs during training. Extensive experimental results on the LFW and YTF benchmarks demonstrate the effectiveness of the proposed method.

Semi-supervised visual domain adaptation is devoted to adapting a model learned in source domain to target domain where there are only a few labeled samples. In this paper, we propose a semi-supervised cross-domain image recognition method which unifies the feature learning and recognition model training into a convolutional neural network framework. Based on a few labeled samples and massive unlabeled samples in the source and target domains, we specially design three branches for class label, domain label and similarity label prediction which simultaneously optimizes the network to generate image features that are domain invariance and inter-class discriminative. Experimental results demonstrate that our method is effective for learning robust cross-domain image recognition model, and achieves the state-of-the-art performance on the widely used visual domain adaptation benchmark.

Deep Convolutional Networks (DCNs) have achieved great success in face detection. Most architectures of the DCN-based methods, however, suffer from multiple separated steps and large-size models, which increase the training complexity and also slow down the testing speed. In this paper, we propose an efficient end-to-end architecture, called Hierarchical Bilinear Network (HBN), for fast and accurate face detection. It mainly consists of two parts: the Backbone Network and the Bilinear Network. The Backbone Network generates hierarchical feature maps for efficiently characterizing faces of different scales, while the Bilinear Network classifies the regions and regresses the face bounding-boxes on each feature map by introducing the Inception module and weights sharing. Benefited from the characters of the proposed architecture, it obtains a better comprehensive performance regarding the model effectiveness, running efficiency, and parameter size, compared with other DCN-based methods. Extensive experimental results demonstrate that our detector achieves competitive accuracy on both the FDDB database and the WIDER FACE database, while still runs in real time (about 69 FPS on a Titan Black GPU) with a tiny size (2.2 MB) model.

Deep convolutional neural networks (DCNN) have recently achieved state-of-the-art performance on handwritten Chinese character recognition (HCCR). However, most of DCNN models employ the softmax activation function and minimize cross-entropy loss, which may loss some inter-class information. To cope with this problem, we demonstrate a small but consistent advantage of using both classification and similarity ranking signals as supervision. Specifically, the presented method learns a DCNN model by maximizing the inter-class variations and minimizing the intra-class variations, and simultaneously minimizing the cross-entropy loss. In addition, we also review some loss functions for similarity ranking and evaluate their erformance. Our experiments demonstrate that the presented method achieves state-of-the-art accuracy on the well-known ICDAR 2013 offline HCCR competition dataset.

Facial landmark detection is a fundamental process included in many face analysis tasks. However, it faces great challenges for the facial images to be detected usually containing large variations, which will deteriorate the detection precision. Among such variations, partial occlusions and pose variations take great effects. In this study, the authors present a mixture of discriminative visibility‐aware models (MDVMs) for facial landmark detection, to improve the generalisation ability of the model to occlusion and pose variation. By adopting different structure constrains for different poses as well as selecting different appearance model for the occluded parts, the MDVMs method can efficiently address the problem of partial occlusion and pose variation. Experiment results demonstrate that their proposed MDVMs method outperforms the well‐known template‐based methods, and can obtain much more accurate and robust facial landmarks detection results under both occlusions and pose variations.