The main challenge in few-shot learning (FSL) is overfitting of the model to limited training data. Recently developed vision-language models like CLIP have been employed to alleviate the overfitting issue and achieved state-of-the-art FSL performance. However, such approaches heavily depend on good alignments between images and associated texts, and therefore may not work as expected when image-text alignment is challenging in downstream tasks, e.g., fine-grained and cross domain image classification. In this study, a new CLIP-based fine-tuning and inference framework is proposed to particularly help the model more accurately recognize visually similar classes and also work well in new imaging domains. During fine-tuning, a modified contrastive loss with adaptively weighted negative pairs is proposed to effectively separate visually similar classes and cluster each class more compactly. During inference, an instance level adaptive ensemble strategy is proposed, utilizing the visual prototypes to adaptively complement the prediction from the CLIP's image-text alignment. Extensive experimental evaluations demonstrate the superiority of the proposed framework, outperforming current state-of-the-art methods by a decent margin on twelve public datasets. The source code will be released publicly.

Recent studies on out-of-distribution (OOD) detection focus on designing models or scoring functions that can effectively distinguish between unseen OOD data and in-distribution (ID) data. In this paper, we propose a simple yet novel ap- proach to OOD detection by leveraging the phenomenon that the average of feature vector elements from convolutional neural network (CNN) is typically larger for ID data than for OOD data. Specifically, the average of feature vector elements is used as part of the scoring function to further separate OOD data from ID data. We also provide mathematical analysis to explain this phenomenon. Experimental evaluations demonstrate that, when combined with a strong baseline, our method can achieve state-of-the-art performance on several OOD detection benchmarks. Furthermore, our method can be easily integrated into various CNN architectures and requires less computation. Source code address: https://github.com/SYSU-MIA-GROUP/statistical_discrepancy_ood.

Out-of-distribution (OOD) detection is vital for the safe application of intelligent systems in real-world scenarios. This paper proposes an enhancement to OOD detection by leveraging the consistency in cognition between two models, both pretrained on in-distribution (ID) data. Specifically, for a given test sample, we first apply Principal Component Analysis (PCA)-based projection on the feature vectors from each model. These obtained feature vectors (with correlation between dimensions decoupled by PCA projection) are then aligned using a multiple linear mapping, which is fitted using the least squares method on the training data. We hypothesize that the regression error for OOD data will be larger than that for ID data, making it a useful metric for OOD detection. Our experimental results demonstrate the effectiveness of this method. When combined with existing robust baselines, our approach achieves state-of-the-art performance in OOD detection.

Out-of-distribution (OOD) detection is a desired ability to ensure the reliability and safety of intelligent systems. A scoring function is often designed to measure the degree of any new data being an OOD sample. While most designed scoring functions are based on a single source of information (e.g., the classifier’s output, logits, or feature vector), recent studies demonstrate that fusion of multiple sources may help better detect OOD data. In this study, after detailed analysis of the issue in OOD detection by the conventional principal component analysis (PCA), we propose fusing a simple regularized PCA-based reconstruction error with other source of scoring function to further improve OOD detection performance. In particular, when combined with a strong energy score-based OOD method, the regularized reconstruction error helps achieve new state-of the-art OOD detection results on multiple standard benchmarks. The code is available at https://github.com/SYSUMIA-GROUP/pca-based-out-of-distribution-detection.