Test-time adaptation (TTA) leverages real-time updates with test samples to effectively address distribution shifts. Previous TTA methods have achieved stable results in scenarios with continual distribution change through techniques such as teacher-student model distillation and image augmentation. However, these test-time adaptation methods are primarily designed for scenarios with balanced class distributions. Yet, in practical applications, having imbalanced classes is more common, which can present a significant challenge. In test data streams characterized by distribution shifts and class imbalances, continual updates can cause the model to become biased towards extreme samples, reducing overall performance. There have only been a few attempts to reduce this influence—generally by sampling from imbalanced test streams to create a more balanced distribution. However, even with these strategies, the models still suffer from biases toward certain categories when the class imbalance is extreme. To overcome the influence of class imbalance in dynamic settings, we propose a Fuzzy Rule-based Test-Time Adaptation method (FTTA). The method incorporates fuzzy rules that allow weights to be assigned to each sample based on its degree of membership in different categories, which helps alleviate the problem of neglecting minor categories in traditional hard categorization methods. By using fuzzy rules, FTTA guides the model to select particular updates. Thus, it prevents the model from excessively favoring specific samples during iterative updates and maintains balance throughout the update process, improving overall accuracy and robustness. Experiments with five datasets in four-class imbalanced continual dynamic scenarios validate the proposed method with FTTA consistently achieving state-of-the-art performance.

Vision-language models, such as the Contrastive Language-Image Pre-Training (CLIP) model, have achieved significant success in image classification tasks. CLIP demonstrates high expressive power in few-shot learning scenarios due to its pairing of text and image encoders. However, CLIP still faces over-fitting when trained with a limited number of samples. To mitigate this, image augmentation techniques have been proposed in few-shot learning tasks to prevent over-fitting by enriching the dataset. Existing image augmentation methods, primarily designed for single-modal image models, focus solely on transformations within the image itself. However, for CLIP, merely increasing visual variety without considering textual content can reduce generalization ability and may even mislead the model. To address this issue, we introduce a novel image augmentation approach—Integrated Image-Text Augmentation (ITA)— for CLIP model in few-shot learning tasks. This method generates new and diverse augmented images to increase the diversity of the training data and reduce over-fitting. Additionally, ITA establishes an alignment between the augmented images and their textual descriptions. Through this alignment, the model not only learns to recognize visual elements in the images but also understands the semantic connections between these elements and the text descriptions. This dual-modal approach enhances the model’s flexibility and accuracy in processing few-shot learning tasks. Extensive experiments in few-shot image classification scenarios have demonstrated that ITA shows significant improvements compared to various image augmentation techniques.

In dynamic environments, the generalization capabilities of large-scale vision language models tend to decline. This is attributed to the evolving distribution of target domains over time, leading to misalignment between image and text pairings, affecting the model’s performance. Addressing this, Test-Time Adaptation (TTA) has been proposed to adapt pre-trained source models to these changing target domains during testing phases. However, traditional TTA approaches, which are designed for a single changing scenario and mainly depend on self-training and entropy minimization, are easily affected by extreme and novel samples in long-term environments, leading to error accumulation and catastrophic forgetting. Although previous Continual Test-Time Adaptation (Continual TTA) methods based on the teacher-student framework can effectively address long-term adaptation issues, they are not feasible for large-scale vision language models due to their high memory requirements. To overcome these challenges, we introduce a novel approach: Prompt-based memory bank for Continual Test-Time Adaptation (PCoTTA). PCoTTA uniquely freezes the CLIP image and text encoders, focusing on updating and storing trainable prompts, significantly reducing memory usage. By implementing a stable pseudo-label strategy and high gradient sensitivity updating, PCoTTA effectively learns new knowledge. In long-term dynamically changing environments, PCoTTA demonstrates high stability and accuracy and achieves a good balance between learning new information and retaining existing knowledge, significantly enhancing the adaptability and generalization capabilities of the CLIP model. Through extensive experimental comparisons, PCoTTA surpasses the current state-of-the-art methods, achieving an average 2% improvement in accuracy for both test-time adaptation and continual test-time adaptation tasks.

In the field of Vision-Language Models (VLM), the Contrastive Language-Image Pretraining (CLIP) model has yielded outstanding performance on many downstream tasks through prompt tuning. By integrating image and text representations, CLIP exhibits zero-shot generalization capabilities on unseen data. However, when new categories and distribution shifts occur, the pretrained text embeddings in CLIP may not align well with unseen images, potentially leading to a decrease in CLIP's zero-shot generalization performance. To address this issue, many existing methods use test samples to update the CLIP model during testing through a process known as Test-Time Adaptation (TTA). Previous TTA techniques, such as image augmentation, can lead to overfitting given outlying samples, while methods based on teacher-student distillation can increase memory use. Further, these methods significantly increase inference time, which is a crucial factor in the testing phase. To improve robustness, mitigate overfitting, and reduce bias toward outlying samples, we propose a novel method: Self-Text Distillation with Conjugate Pseudo-labels (SCP), designed to enhance CLIP's zero-shot generalization. SCP uses gradient information from conjugate pseudo-labels to enhance the model's robustness toward distribution shifts. It also innovates by using a fixed prompt list to distil learnable prompts from within the same model, acting as a self-regulation mechanism that minimizes overfitting. Additionally, SCP is a fully test-time adaptation method that does not require retraining. It directly improves CLIP's zero-shot generalization at test time without increasing either memory overheads or inference time. In evaluations across three zero-shot generalization scenarios, SCP surpasses existing state-of-the-art methods in performance and significantly reduces inference time.