The advancements in large language models (LLMs) have propelled the improvement of video understanding tasks by incorporating LLMs with visual models. However, most existing LLM-based models (e.g., VideoLLaMA, VideoChat) are constrained to processing short-duration videos. Recent attempts to understand long-term videos by extracting and compressing visual features into a fixed memory size. Nevertheless, those methods leverage only visual modality to merge video tokens and overlook the correlation between visual and textual queries, leading to difficulties in effectively handling complex question-answering tasks. To address the challenges of long videos and complex prompts, we propose AdaCM2, which, for the first time, introduces an adaptive cross-modality memory reduction approach to video-text alignment in an auto-regressive manner on video streams. Our extensive experiments on various video understanding tasks, such as video captioning, video question answering, and video classification, demonstrate that AdaCM2achieves state-of-the-art performance across multiple datasets while significantly reducing memory usage. Notably, it achieves a 4.5% improvement across multiple tasks in the LVU dataset with a GPU memory consumption reduction of up to 65%.

Domain adaptation leverages labeled data from a source domain to learn an accurate classifier for an unlabeled target domain. Since the data collected in practical applications usually contain noise, the weakly-supervised domain adaptation algorithm has attracted widespread attention from researchers that tolerates the source domain with label noises or/and features noises. Several weakly-supervised domain adaptation methods have been proposed to mitigate the difficulty of obtaining the high-quality source domains that are highly related to the target domain. However, these methods assume to obtain the accurate noise rate in advance to reduce the negative transfer caused by noises in source domain, which limits the application of these methods in the real world where the noise rate is unknown. Meanwhile, since source data usually comes from multiple domains, the naive application of single-source domain adaptation algorithms may lead to sub-optimal results. We hence propose a universal and scalable weakly-supervised domain adaptation method called PDCAS to ease restraints of such assumptions and make it more general. Specifically, PDCAS includes two stages: progressive distillation and domain alignment. In progressive distillation stage, we iteratively distill out potentially clean samples whose annotated labels are highly consistent with the prediction of model and correct labels for noisy source samples. This process is non-supervision by exploiting intrinsic similarity to measure and extract initial corrected samples. In domain alignment stage, we consider Class-Aligned Sampling which balances the samples for both source and target domains along with the global feature distributions to alleviate the shift of label distributions. Finally, we apply PDCAS in multi-source noisy scenario and propose a novel multi-source weakly-supervised domain adaptation method called MSPDCAS, which shows the scalability of our framework. Extensive experiments on Office-31 and Office-Home datasets demonstrate the effectiveness and robustness of our method compared to state-of-the-art methods.

Weakly-supervised domain adaptation has been introduced to address the source domain with label noise or/and feature noise. However, the existing weakly-supervised domain adaptation methods only work under ideal assumptions, which assume either the annotated data of the target domain can be accessed or the noise rate of all the classes is identical and already known. This limits their practical application. To tackle this, we propose a universal weakly-supervised domain adaptation method called PDCAS which relaxes the ideal assumptions and makes it more general. Specially, PD- CAS includes two stages: progressive distillation and domain alignment. In progressive distillation, we iteratively distill out potentially corrected samples whose annotated labels are consistent with the prediction of model. By exploiting intrinsic similarity to extract initial corrected samples, this process does not need any supervision. In domain alignment, besides taking the global feature distributions into consideration, we also adopt Class-Aligned Sampling which balances the samples for both source and target domains to alleviate the shift of label distributions. Extensive experiments on Office-31 and Office-Home datasets demonstrate the effectiveness and robustness of our method compared to state-of-the-art methods.

Domain adaptation aims to transfer knowledge across different domains and bridge the gap between them. While traditional knowledge transfer considers identical domain, a more realistic scenario is to transfer from a larger and more diverse source domain to a smaller target domain, which is referred to as partial domain adaptation (PDA). However, matching the whole source domain to the target domain for PDA might produce negative transfer. Samples in the shared classes should be carefully selected to mitigate negative transfer in PDA. We observe that the correlations between different target domain samples and source domain samples are diverse: classes are not equally correlated and moreover, different samples have different correlation strengthes even when they are in the same class. In this study, we propose ECDT, a novel PDA method that Exploits the Correlation Diversity for knowledge Transfer between different domains. We propose a novel method to estimate target domain label space that utilizes the label distribution and feature distribution of target samples, based on which outlier source classes can be filtered out and their negative effects on transfer can be mitigated. Moreover, ECDT combines class-level correlation and instance-level correlation to quantity sample-level transferability in domain adversarial network. Experimental results on three commonly used cross-domain object data sets show that ECDT is superior to previous partial domain adaptation methods.