Adapting pre-trained vision-language models to downstream tasks has emerged as a novel paradigm for zero-shot learning. Existing test-time adaptation (TTA) methods such as TPT attempt to fine-tune visual or textual representations to accommodate downstream tasks but still require expensive optimization costs. To this end, Training-free Dynamic Adapter (TDA) maintains a cache containing visual features for each category in a parameter-free manner and measures sample confidence based on prediction entropy of test samples. Inspired by TDA, this work aims to develop the first training-free adapter for zero-shot video classification. Capturing the intrinsic temporal relationships within video data to construct and maintain the video cache is key to extending TDA to the video domain. In this work, we propose a reliable and diverse Hierarchical Adapter for zero-shot video classification, which consists of Frame-level Cache Refiner and Video-level Cache Updater. Before each video sample enters the corresponding cache, it needs to be refined at frame level based on prediction entropy and temporal probability difference. Due to the limited capacity of the cache, we update the cache during inference based on the principle of diversity. Experiments on four popular video classification benchmarks demonstrate the effectiveness of Hierarchical Adapter. The code is available at https://github.com/Gwxer/Hierarchical-Adapter.

The core challenge in video compositional representation lies in jointly identifying atomic actions (verbs) and their associated objects (nouns), while generalizing to unseen verb-noun combinations. Existing approaches often adopt a shared backbone with multi-head classifiers, leading to semantic entanglement and recognition imbalance, especially under ViT-based paradigms. To tackle these issues, we propose a novel framework, Structural Pre-Decoupling and Co-Aligning (SODA), which structurally decouples the early-stage learning paths of verbs and nouns, mitigating semantic interference and recognition imbalance. A core component is the Divide-and-Conquer Disentanglement (DCD) module, comprising two parallel paths: an Object-Removed Verb (OR-V) path that explicitly suppresses object appearance by integrating frame-difference features for short-term motion and trajectory cues for long-term dynamics, and an Object-Centric Noun (OC-N) path that constructs adaptive key-frame representations in the guidance of dynamic features from OR-V. On top of these, a Compositional Co-Alignment (CCA) strategy is further introduced that aligns atomic and compositional representations in a shared semantic space through contrastive learning, capturing implicit commonsense associations of verb-noun pairs while preserving the discriminative power of each stream. Extensive experiments on both standard and zero-shot compositional action recognition benchmarks validate the effectiveness and generalization of our approach.