Heterogeneous Face Recognition (HFR) is a challenging task due to the significant intra-class variation between the query and gallery images. The reason behind this vast intra-class variation is the varying image capturing sensors and the varying image representation techniques. Visual, Infrared, thermal images are the output of different sensors and viewed sketches, and composite sketches are the output of different image representation techniques. Conventional deep learning models are trying to solve the problem. Still, progress is impeded due to small HFR data samples, task-specific models (one model trained for face sketch-photo matching can’t perform well for NIR-VIS face matching), joint learning of two different HFR scenarios are not possible by one single deep network, and models are not interpretable. In this paper, to solve these major problems, we presented a novel interpretable Local Frequency Binary Pattern (LFrBP) based continual learning shallow network for HFR. The model is divided into two parts. A modality-invariant CNN model using the LFrBP feature, fine-tuned with CNN, is presented in the first part. The second part is based on continual learning to jointly learn the two HFR scenarios (face sketch-photo and NIR-VIS face matching) using a single network. Recognition results on different challenging HFR databases depict the superiority of the proposed model over other state-of-the-art deep learning-based methods.

This paper presents a novel local image descriptor called Pattern of Local Gravitational Force (PLGF). It is inspired by Law of Universal Gravitation. PLGF is a hybrid descriptor, which is a combination of two feature components: one is the Pattern of Local Gravitational Force Magnitude (PLGFM), and another is Pattern of Local Gravitational Force Angle (PLGFA). PLGFM encodes the local gravitational force magnitude, and PLGFA encodes the local gravitational force angle that the center pixel exerts on all other pixels within a local neighborhood. We propose a novel noise resistance and the edge-preserving binary pattern called neighbors to center difference binary pattern (NCDBP) for gravitational force magnitude encoding. Finally, the histograms of the two components are concatenated to construct the PLGF descriptor. Experimental results on the existing face recognition databases, texture database, and biomedical image database show that PLGF is an effective image descriptor, and it outperforms other widely used existing descriptors. Even if in complicated variations like noise, and illumination with smaller databases, a combination of PLGF and convolutional neural network (CNN) performs consistently better than other state-of-the-art techniques.

This paper proposes a novel method called local-gravity-face (LG-face) for illumination-invariant and heterogeneous face recognition (HFR). LG-face employs a concept called the local gravitational force angle (LGFA). The LGFA is the direction of the gravitational force that the center pixel exerts on the other pixels within a local neighborhood. A theoretical analysis shows that the LGFA is an illumination-invariant feature, considering only the reflectance part of the local texture effect of the neighboring pixels. It also preserves edge information. Rank 1 recognition rates of 97.78% on the CMU-PIE database and 97.31% on the Extended Yale B database are achieved under varying illumination, demonstrating that LG-face is an effective method of illumination-invariant face recognition. For HFR, when faces appear in different modalities, LG-face produces a common feature representation. Rank 1 recognition rates of 99.96% on the CUFS database, 98.67% on the CUFSF database, and 99.78% on the CASIA-HFB database show that the LG-face is also an effective method for HFR. The proposed method also performs consistently in the presence of complicated variations and noise.