We study the patch selection problem for efficient transformer-based visual understanding wherein a sampler can be used to drop less informative patches at inference in order to speed up model execution on resource-constrained devices. As no labels are available on the saliency of the patches, existing works either try to solve an auxiliary task of locating distinctive patches or learn a policy network through the global image/video-level supervision. The former approach could drop redundant but important patches while the latter suffers from the weak supervision of a single class label per image/video. In this work, we observe that the attention weights in trained transformer-based models clearly highlight the salient regions in images and videos. Therefore, we propose a learned patch sampling framework called SAMPL that utilizes the attention weights as fine-grained patch-level supervision to learn a lightweight policy network for patch selection. To train SAMPL end-to-end with the transformer-based models, we introduce a new loss function based on the REINFORCE algorithm to match the distribution of patch selection probabilities and the attention scores. Experimental results on ImageNet, UCF101, Something-Something v2 and Kinetics-400 show that SAMPL can effectively increase the throughput by at least 1.5× while achieving competitive classification accuracy.

Incorporating large image-text foundation models such as CLIP has substantially improved the performance of the multimodal video retrieval task. However, how to practically sample the frames from a video and aggregate the frame features into a video representation is still an open research question. In particular, real-world deployment scenarios, such as embodiment within consumer electronics or cloud-based inference pipelines, require two key facets of retrieval (representation building and search) to be computationally light and fast. In this paper, we propose AdaCLIP, a computation- and latency-aware system for pragmatic multimodal video retrieval. AdaCLIP consists of a learning-based frame selection module to select informative frames and a query-independent frame aggregation module to obtain strong video representations from the frame features. Specifically, in the frame selection module, we introduce a differentiable Hard-Top-k algorithm to sample a subset of the frames while optimizing the performance of the video retrieval task in an end-to-end manner. Moreover, to be latency-aware, we also propose a query-independent lightweight approach, MLP-Score, to aggregate the frame features into the video representation, which offers up to 142x speedup on GPU and 822x speedup on CPU in similarity search time compared to query-dependent matching methods. Experimental results on several popular video retrieval datasets confirm the effectiveness of AdaCLIP.

Self-supervised methods have significantly closed the gap with end-to-end supervised learning for image classification [13], [24]. In the case of human action videos, however, where both appearance and motion are significant factors of variation, this gap remains significant [28], [58]. One of the key reasons for this is that sampling pairs of similar video clips, a required step for many self-supervised contrastive learning methods, is currently done conservatively to avoid false positives. A typical assumption is that similar clips only occur temporally close within a single video, leading to insufficient examples of motion similarity. To mitigate this, we propose SLIC, a clustering-based self-supervised contrastive learning method for human action videos. Our key contribution is that we improve upon the traditional intra-video positive sampling by using iterative clustering to group similar video instances. This enables our method to leverage pseudo-labels from the cluster assignments to sample harder positives and negatives. SLIC outperforms state-of-the-art video retrieval baselines by$+15.4\%$on top-1 recall on UCF101 and by$+5.7\%$when directly transferred to HMDB51. With end-to-end finetuning for action classi-fication, SLIC achieves 83.2% top-1 accuracy$(+0.8\%)$on UCF101 and 54.5% on HMDB51$(+1.6\%$,. SLIC is also competitive with the state-of-the-art in action classification after self-supervised pretraining on Kinetics400.