With the development of multimedia technology, online social media has become a major medium for people to access news, but meanwhile, it has also exacerbated the dissemination of multi-modal fake news. An automatic and efficient multi-modal fake news detection (MFND) method is urgently needed. Existing MFND methods usually conduct cross-modal information interaction at later stage, resulting in insufficient exploration of complementary information between modalities. Another challenge lies in the differences among news data from different domains, leading to the weak generalization ability in detecting news from various domains. In this work, we propose an efficient Cross-modal Prompt Learning with Semantic enhancement method for Domain-robust fake news detection (CPLSD). Specifically, we design an efficient cross-modal prompt interaction module, which utilizes prompt as medium to realize lightweight cross-modal information interaction in the early stage of feature extraction, enabling to exploit rich modality complementary information. We design a domain-general prompt generation module that can adaptively blend domain-specific news features to generate domain-general prompts, for improving the domain generalization ability of the model. Furthermore, an image semantic enhancement module is designed to achieve image-to-text translation, fully exploring the semantic discriminative information of the image modality. Extensive experiments conducted on three MFND benchmarks demonstrate the superiority of our proposed approach over existing state-of-the-art MFND methods.

Multi-modal fake news detection (MFND) has attracted increasing attention. However, due to information loading failure or access restriction, incomplete modality makes joint multi-modal information extraction be challenging. Existing MFND methods with missing-modality focus on specific missing-modality cases, and how to flexibly deal with various missing-modality cases of news in the real world has not been well studied. In this paper, we propose a novel fake news detection approach named Complementary Graph learning and Prompt-based cross-modal Generation network (CG-PG), which contains two main modules: a complementary graph learning module and a prompt-based cross-modal generation module. The complementary graph learning module explores structural complementary information in image and text graphs to implement cross-modal information propagation. To further recover the information loss caused by missing modalities, the prompt-based cross-modal generation module generates representations of the missing modality from available modalities and imposes task-related constraints on the representations with the missing-aware prompts. Experimental results on the public Weibo and Fakeddit datasets under various missing-modality cases show that CG-PG outperforms state-of-the-art related works.

Traditional unsupervised multi-source domain adaptation usually assumes that all source domain data can be utilized during training. Unfortunately, due to practical concerns such as privacy, data storage, and computational costs, data from different source domains are often isolated from each other. To address this issue, we propose a federated domain adaptation framework based on fine-grained alignment. This method achieves domain adaptation at the model level through iterative training of source and target domains, thereby avoiding the direct use of source domain data. Specifically, our approach employs specialized techniques at various stages—model construction, pseudo-label generation, and model training—to handle fine-grained features that are often overlooked. This enables the model to effectively remove irrelevant information and learn more discriminative features, thus narrowing the distribution gap between domains. Extensive experimental results demonstrate the effectiveness of our proposed method across multiple datasets.

In this paper, we investigate the Source-Free Domain Adaptation (SFDA), where a well-trained model adapts to an unlabeled target domain without access to source data. Previous SFDA methods mainly relied on convolutional neural networks, which struggle with domain shifts due to their local focus. To address this, we propose the Object-centric Perception Source-Free Transformer (OP-SFT), which leverages the self-attention mechanism of Transformers to focus on relevant target regions, improving adaptability to domain shifts. We also introduce self-supervised knowledge distillation to enhance semantic perception and a confidence-based k-means clustering method for more accurate pseudo-label generation. Extensive experiments demonstrate that our OP-SFT achieves significant adaptation performance across four widely-used domain adaptation benchmark datasets compared to other state-of-the-art baselines. The code is available at https://github.com/Weilong-Gao/OP-SFT.

Multi-source Domain Adaptation (MSDA) aims to adapt models trained on multiple labeled source domains to an unlabeled target domain. Recent MSDA methods based on Generative Adversarial Networks (GANs) implicitly capture the image distribution, which can lead to limited sample fidelity and result in misalignment of pixel-level information between the sources and the target domain. Moreover, when samples from different sources interact during training, significant misalignment across various source domains can occur. In this study, we introduce a novel MSDA framework called Adaptive Causal Diffusion Networks (ACDN) to address these challenges. ACDN integrates a diffusive domain adaptation model for effective, high-fidelity adaptation between the source and target domains, incorporating Granger-causal inference to ensure that the assigned weights for each source domain are closely related to their respective contributions to the decision-making process. Experimental results show that ACDN outperforms existing methods significantly across real-world domain adaptation benchmarks.

In data classification, class-balanced data is ideal, but real datasets are often imbalanced, necessitating rebalancing through methods like resampling. In recent years, some new generative model-based resampling methods have been proposed. However, when facing extreme class imbalance, where the minority class is strongly underrepresented and on its own does not contain enough information to conduct the generative process. Some deep learning methods have been proposed to solve extremely imbalanced classification problems, but some of them are only used for specific datasets. Therefore, we proposed a novel deep learning method that combines a generative strategy with multi-task joint learning, termed sample-pair learning network (SPLN), for extremely imbalanced classification. The network consists of data preprocessing and multi-task joint learning modules. During data preprocessing, the training set is expanded by constructing positive and negative sample-pairs, then rebalanced using a strategy combining attention and resampling, termed undersampling based on attention power values (APVUS). The multi-task joint learning module employs a Siamese convolutional subnetwork to measure the similarity between sample-pairs and a multi-layer perceptron to recognize the category of single samples. The module can reduce the risk of overfitting caused by excessive noise in the training set. Finally, we designed a voting model based on the Siamese convolutional subnetwork to infer the categories of test samples. Experimental results demonstrate that our approach outperforms state-of-the-art generative model-based methods and is effective and general for extremely imbalanced classification.

This paper proposes a novel normalizing flow learning based method IIAG-CoFlow for low-light image enhancement (LLIE), which consists of an inter-and intra-channel attention Transformer based conditional generator (IIAG) and a complete flow (CoFlow). On the one hand, IIAG is designed as a U-shape network, whose down-sampling and up-sampling layers are constructed by IIZAT (i.e., inter-and intra-channel and zero-map attention Transformer) and IIAT (i.e., inter-and intra-channel attention Transformer) respectively. IIAT is designed to calculate inter-channel attention and intra-channel attention independently. Based on IIAT, IIZAT is designed to perform parallel fusion of zero-map attention and intra-channel attention. On the other hand, based on existing normalizing flow, we bring in unconditional affine coupling layer and design 3 invertible linear transformation layers, to develop CoFlow. The height and width axes based cross attention network (HWCAN) is proposed to learn affine/linear transformation parameters for conditional feature-driven layers of CoFlow. Experiments show that IIAG-CoFlow outperforms existing SOTA LLIE methods on several benchmark low-light datasets, and real NTM images. The source codes and pre-trained models are available athttps://github.com/NJUPT-IPR-ChenTS/IIAG-CoFlow.

We observe that the low-light RGB images, as well as night traffic monitoring (NTM) images, contain lots of color pixels with zeros caused by the low-light, which means that the low-light images suffer both information weakness and information loss of zero-element pixels. In this paper, we propose a novel flow-based generative method JTE-CFlow for low-light image enhancement, which consists of a joint-attention transformer based conditional encoder (JTE) and a map-wise cross affine coupling flow (CFlow). Specifically, JTE executes short-range and long-range operations by RRDBs (i.e., residual-in-residual dense blocks) and JATs (i.e., joint-attention transformer blocks) in series connection. JAT achieves weak information amplification and information loss restoration of zero-element pixels by the integration of self-attention and specific-attention with sharing the same value vectors, where the query and key vectors of specific-attention are from the zero-map feature of the low-light image. On the other hand, CFlow develops a map-wise cross affine coupling (MCAC) layer to perform cross learning for the flow feature, and a multiplication coupling network (MCN) to learn the transformation parameters of MCAC. JTE-CFlow learns to map the subtraction of outputs of CFlow and JTE (i.e., the residual code) into a standard normal distribution, and the inverse network of CFlow takes the latent feature of the low-light image as its input to infer the enhanced image. Experiments show that JTE-CFlow outperforms most SOTA methods on 7 mainstream low-light datasets with the same architecture, and can be applied to enhance NTM images. The source code and pre-trained models are available athttps://github.com/NJUPT-IPR-HuYin/JTE-CFlow.

This paper proposes a fast HSI (hue, saturation, intensity) color space and an orthogonal triangular with column pivoting (QRCP) enhancement model to tackle the large size RGB (red, green, blue) night traffic monitoring (NTM) image. Firstly, the fast HSI (FHSI) is proposed to decompose the light and color information of the RGB image, whose hue is defined as the cosine value of the included angle, instead of the included angle in HSI. The saturation of FHSI is defined as the ratio of the projection vector length and the side length of the projection equilateral triangle, and a saturation correction model is further proposed to correct color distortion of the low-light image by adjusting the saturation of FHSI. FHSI is more concise and faster than HSI. Secondly, a novel QRCP enhancement (QRCPE) model is proposed to improve the light of the low-light image by enhancing the intensity of FHSI, which first strengthens diagonal elements of QRCP, and followed by controlling the normalization of strengthened diagonal elements of QRCP. Finally, the FHSI-QRCPE based RGB image can be obtained by transforming the processed FHSI to RGB. The experimental results on NTM, SICE, ExDark, and BDD 100K databases, indicate that the proposed FHSI-QRCPE is fast and efficient to tackle low-light image enhancement.

The real traffic monitoring driver face (TMDF) images are with complex multiple degradations, which decline face recognition accuracy in real intelligent transportation systems (ITS). This paper is the first to propose joint image-to-image (I2I) translation to enhance TMDF images of ITS. First, as TMDF images are without corresponding clear ones, identity preserving is critical for TMDF images under unpaired I2I translation. This paper proposes a fast diagonal symmetry pattern (FDSP) to preserve identity structure under unpaired I2I translation. Second, FDSP is introduced into CycleGAN to form FDSP-CG, which aims to learn the degradation mapping (i.e., FDSP-CG-d) from the clarity domain to the degradation domain. FDSP-CG-d can generate massive degradation/clarity image pairs for paired I2I translation training. Third, this paper proposes the dual residual block (DRB) to strengthen Pix2pix for rich face detail features learning (i.e., DRB-P2P), which learns the enhancement mapping from the degradation image to its clear version under paired I2I translation. Finally, the experiments on TMDF (i.e., the brevity name of the face database collected from real ITS) and Chinese famous face (CFF) databases, as well as CelebA and MegaFace databases, indicate that the proposed method can efficiently enhance TMDF images whose degradation variations are learned by FDSP-CG.

This paper proposes HSV (hue, saturation, value) with three sectors (HSV-3S) and two-dimensional gradient descent algorithm (2D-GDA) for high-saturation low-light image enhancement in night traffic monitoring (NTM). The saturation of HSV-3S is defined as the ratio of the projection vector length and twice length of the sector start vector, which results in that the saturation of HSV-3S is smaller than that of HSV, and a saturation weakening model is proposed to further decrease the saturation of HSV-3S. The hue of HSV-3S is defined as the cosine value of the included angle between the projection vector and the sector start vector in each of three sectors. HSV-3S is more concise and faster than HSV. Then, 2D-GDA extends the gradient descent algorithm to 2D image domain. 2D-GDA employs the iteration matrix with variable step values (i.e., the step values of the dark regions are less than those of the bright regions), which can improve the pixel distribution of the 2D-GDA enhanced image. Finally, the HSV-3S+2D- GDA based RGB image can be obtained by performing 2D-GDA on the value of HSV-3S with transforming the processed HSV-3S to RGB. The experimental results on NTM (i.e., the brevity name of the database collected from real ITS), LOL, ExDark and SICE databases, indicate that HSV-3S+2D-GDA is fast and efficient for high-saturation low-light image enhancement.

Existing face hallucination methods based on convolutional neural networks (CNNs) have achieved impressive performance on low-resolution (LR) faces in a normal illumination condition. However, their performance degrades dramatically when LR faces are captured in non-uniform illumination conditions. This paper proposes a Recursive Copy and Paste Generative Adversarial Network (Re-CPGAN) to recover authentic high-resolution (HR) face images while compensating for non-uniform illumination. To this end, we develop two key components in our Re-CPGAN: internal and recursive external Copy and Paste networks (CPnets). Our internal CPnet exploits facial self-similarity information residing in the input image to enhance facial details; while our recursive external CPnet leverages an external guided face for illumination compensation. Specifically, our recursive external CPnet stacks multiple external Copy and Paste (EX-CP) units in a compact model to learn normal illumination and enhance facial details recursively. By doing so, our method offsets illumination and upsamples facial details progressively in a coarse-to-fine fashion, thus alleviating the ambiguity of correspondences between LR inputs and external guided inputs. Furthermore, a new illumination compensation loss is developed to capture illumination from the external guided face image effectively. Extensive experiments demonstrate that our method achieves authentic HR face images in a uniform illumination condition with a 16× magnification factor and outperforms state-of-the-art methods qualitatively and quantitatively.

Existing face hallucination methods based on convolutional neural networks (CNN) have achieved impressive performance on low-resolution (LR) faces in a normal illumination condition. However, their performance degrades dramatically when LR faces are captured in low or non-uniform illumination conditions. This paper proposes a Copy and Paste Generative Adversarial Network (CPGAN) to recover authentic high-resolution (HR) face images while compensating for low and non-uniform illumination. To this end, we develop two key components in our CPGAN: internal and external Copy and Paste nets (CPnets). Specifically, our internal CPnet exploits facial information residing in the input image to enhance facial details; while our external CPnet leverages an external HR face for illumination compensation. A new illumination compensation loss is thus developed to capture illumination from the external guided face image effectively. Furthermore, our method offsets illumination and upsamples facial details alternatively in a coarse-to-fine fashion, thus alleviating the correspondence ambiguity between LR inputs and external HR inputs. Extensive experiments demonstrate that our method manifests authentic HR face images in a uniform illumination condition and outperforms state-of-the-art methods qualitatively and quantitatively.

This paper models the driver face recognition problem under the intelligent traffic monitoring systems as severe illumination variation face recognition with single sample problem. Firstly, in the point of view of numerical value sign, the current illumination invariant unit is derived from the subtraction of two pixels in the face local region, which may be positive or negative, we propose a generalized illumination robust (GIR) model based on positive and negative illumination invariant units to tackle severe illumination variations. Then, the GIR model can be used to generate several GIR images based on the local edge-region or the local block-region, which results in the edge-region based GIR (EGIR) image or the block-region based GIR (BGIR) image. For single GIR image based classification, the GIR image utilizes the saturation function and the nearest neighbor classifier, which can develop EGIR-face and BGIR-face. For multi GIR images based classification, the GIR images employ the extended sparse representation classification (ESRC) as the classifier that can form the EGIR image based classification (GIRC) and the BGIR image based classification (BGIRC). Further, the GIR model is integrated with the pre-trained deep learning (PDL) model to construct the GIR-PDL model. Finally, the performances of the proposed methods are verified on the Extended Yale B, CMU PIE, AR, self-built Driver and VGGFace2 face databases. The experimental results indicate that the proposed methods are efficient to tackle severe illumination variations.

In this paper, we present a novel high-frequency facial feature and a high-frequency based sparse representation classification to tackle single sample face recognition (SSFR) under varying illumination. Firstly, we propose the assumption that QRCP bases can represent intrinsic face surface features with different frequencies, and their corresponding energy coefficients describe illumination intensities. Based on this assumption, we take QRCP bases with corresponding weighting coefficients (i.e. the major components of energy coefficients) to develop the high-frequency facial feature of the face image, which is named as QRCP-face. The normalized QRCP-face (NQRCPface) is constructed to further constraint illumination effects by normalizing the weighting coefficients of QRCP-face. Moreover, we propose the adaptive QRCP-face (AQRCP-face) that assigns a special parameter to NQRCP-face via the illumination level estimated by the weighting coefficients. Secondly, we consider that the differences of pixel images cannot model the intraclass variations of generic faces with illumination variations, and the specific identification information of the generic face is redundant for the current SSFR with generic learning. To tackle above two issues, we develop a general high-frequency based sparse representation (GHSP) model. Two practical approaches separated high-frequency based sparse representation (SHSP) and unified high-frequency based sparse representation (UHSP) are developed. Finally, the performances of the proposed methods are verified on the Extended Yale B, CMU PIE, AR, LFW and our self-built Driver face databases. The experimental results indicate that the proposed methods outperform previous approaches for SSFR under varying illumination.

In practical application, the performances of face recognition are always affected by variations of expression, illumination and so on. To address this problem, an interval type-2 fuzzy linear discriminant analysis (IT2FLDA) method is proposed. In this paper, we first propose the supervised interva l type-2 fuzzy C-Means (IT2FCM) algorithm. Moreover, the supervised IT2FCM is incorporated into linear discriminant analysis (LDA). In this method, the membership degree matrix of training samples belonging to each class and means of each class are firstly calculated by the supervised IT2FCM algorithm. They are then applied to the definition of fuzzy within-class scatter matrix and fuzzy between-class scatter matrix, respectively. In doing so, means of each class that are estimated by the supervised IT2FCM can converge to a more desirable location than ones obtained by class sample average and fuzzy k-nearest neighbor (FKNN) method. Furthermore, the IT2FLDA is able to minimize the effects of uncertainties, find the optimal projective directions and make the feature subspace discriminating and robust, which inherits the benefits of the supervised IT2FCM and LDA. The experiment results show that the IT2FLDA improves the recognition rate and reduces sensitivity to variations when compared to results from the previous techniques.