Contemporary deep learning, characterized by the training of cumbersome neural networks on massive datasets, confronts substantial computational hurdles. To alleviate heavy data storage burdens on limited hardware resources, numerous dataset compression methods such as dataset distillation (DD) and coreset selection have emerged to obtain a compact but informative dataset through synthesis or selection for efficient training. However, DD involves an expensive optimization procedure and exhibits limited generalization across unseen architectures, while coreset selection is limited by its low data keep ratio and reliance on heuristics, hindering its practicality and feasibility. To address these limitations, we introduce a newly versatile framework for dataset compression, namely Adaptive Dataset Quantization (ADQ). Specifically, we first identify the sub-optimal performance of naive Dataset Quantization (DQ), which relies on uniform sampling and overlooks the varying importance of each generated bin. Subsequently, we propose a novel adaptive sampling strategy through the evaluation of generated bins' representativeness score, diversity score and importance score, where the former two scores are quantified by the texture level and contrastive learning-based techniques, respectively. Extensive experiments demonstrate that our method not only exhibits superior generalization capability across different architectures, but also attains state-of-the-art results.

Vision Transformers (ViTs) and their variants have achieved remarkable success across a broad spectrum of computer vision tasks. However, their high computational cost and significant data requirements present challenges for deployment in resource-constrained environments. Current pruning methods for ViTs predominantly focus on reducing token counts, which often disrupt the inherent spatial structure of ViTs, hindering their adaptability to hardware. Furthermore, how to compress ViTs efficiently in few-shot scenarios remains an open question. Hence, we introduce a fine-grained block pruning framework for ViTs, named FBP-ViT. Unlike traditional block pruning techniques that indiscriminately remove entire blocks, FBP-ViT selectively eliminates Multi-Head Self-Attention (MSA) or Multi-Layer Perceptron (MLP) blocks, offering enhanced flexibility and efficiency. We unify pruning and finetuning, ensuring practicality in resource-constrained environments for real-world applications. We evaluate the proposed FBP-ViT across ViTs of varying sizes and architectures, demonstrating its effectiveness in improving computational efficiency while maintaining high performance. Specifically, with a speedup factor of 1.34, FBP-ViT preserves 80.39% top-1 accuracy on ImageNet-1k using DeiT-Base, achieving more precise and efficient pruning with tiny sets.