Recently, great progress has been achieved on facial landmark detection based on convolutional neural network, while it is still challenging due to partial occlusion and extreme head pose. In this paper, we propose a Cascaded Structure-Learning Network (CSLN) with using adversarial training to improve the performance of 2D facial landmark detection by taking the structure of facial landmarks into account. In the first stage, we improve the original stacked hourglass network, which applies a multi-branch module to capture different scales of features, a progressive convolution structure to compensate for the missing structural features in hourglass networks, and a pyramid inception structure to expand the receptive field. Specially, by introducing a discriminator, we use the adversarial training strategy to urge the improved hourglass network for generating more accurate heatmaps. The second stage, which is based on attention mechanism, optimizes the spatial correlations between different facial landmarks by reusing the structural features. Moreover, we propose a novel region loss, which can adaptively allocate proper weights to different regions. In this way, the network can focus more on those occluded landmarks. The experimental results on several datasets, i.e. 300W, COFW, and AFLW, show that our proposed method achieves superior performance compared with the state-of-the-art methods.

Facial landmark detection aims at localizing multiple keypoints for a given facial image, which usually suffers from variations caused by arbitrary pose, diverse facial expression and partial occlusion. In this paper, we develop a two-stage regression network for facial landmark detection on unconstrained conditions. Our model consists of a Structural Hourglass Network (SHN) for detecting the initial locations of all facial landmarks based on heatmap generation, and a Global Constraint Network (GCN) for further refining the detected locations based on offset estimation. Specifically, SHN introduces an improved Inception-ResNet unit as basic building block, which can effectively improve the receptive field and learn contextual feature representations. In the meanwhile, a novel loss function with adaptive weight is proposed to make the whole model focus on the hard landmarks precisely. GCN attempts to explore the spatial contextual relationship between facial landmarks and refine the initial locations of facial landmarks by optimizing the global constraint. Moreover, we develop a pre-processing network to generate features with different scales, which will be transmitted to SHN and GCN for effective feature representations. Different from existing models, the proposed method realizes the heatmap-offset framework, which combines the outputs of heatmaps generated by SHN and coordinates estimated by GCN, to obtain an accurate prediction. The extensive experimental results on several challenging datasets, including 300W, COFW, AFLW, and 300-VW confirm that our method achieve competitive performance compared with the state-of-the-art algorithms.

Recently, tremendous progress has been achieved on human pose estimation with the development of convolutional neural networks (CNNs). However, current methods still suffer from severe occlusion, back view, and large pose variation due to the lack of consideration of the spatial relationship between different joints, which can provide strong cues for localizing the hidden keypoints. In this work, we design a Structural Pose Network (SPN) to take full advantage of joint structure for human pose estimation under unconstrained environment. Specifically, the proposed model is composed of two subnets: Structure Residual Network (SRN) and Structure Improving Network (SIN). Given an input image, SRN first captures rich joint structure as priors through a multi-branch feature extraction module, following a hourglass network with pyramid residual units to enlarge the receptive field and further obtain structural feature representations. SIN, based on coordinate regression, can optimize the spatial relationship of different joints via the attention mechanism, thus refining the initial prediction from SRN. In addition, we propose a novel structure-consistency constraint, which can maintain the structural consistency between the joints and body parts via estimating whether the joints are located in their corresponding parts. At the same time, an online hard regions mining (OHRM) strategy is introduced to drive the network to pay corresponding attention to different body parts. The experimental results on three challenging datasets show that our method outperforms other state-of-the-art algorithms.