Contrastive Language-Image Pre-training (CLIP) has become a promising language-supervised visual pre-training framework. This paper aims to distill small CLIP models supervised by a large teacher CLIP model. We propose several distillation strategies, including relation, feature, gradient and contrastive paradigms, to examine the effectiveness of CLIP-Knowledge Distillation (KD). We show that a simple feature mimicry with Mean Squared Error loss works surprisingly well. Moreover, interactive contrastive learning across teacher and student encoders is also effective in performance improvement. We explain that the success of CLIP-KD can be attributed to maximizing the feature similarity between teacher and student. The unified method is applied to distill several student models trained on CC3M+12M. CLIP-KD improves student CLIP models consistently over zero-shot ImageNet classification and cross-modal retrieval bench-marks. When using ViT-U14 pretrained on Laion-400M as the teacher, CLIP-KD achieves 57.5% and 55.4% zero-shot top-1 ImageNet accuracy over ViT-B/16 and ResNet-50, surpassing the original CLIP without KD by 20.5% and 20.1% margins, respectively. Our code is released on https://github.com/winycg/CLIP-KD.

Unsupervised multitask pre-training has been the critical method behind the recent success of language models (LMs).However, supervised multitask learning still holds significant promise, as scaling it in the post-training stage trends towards better generalization.In this paper, we explore supervised multitask pretraining by proposing Instruction Pre-Training, a framework that scalably augments massive raw corpora with instruction-response pairs to pre-train LMs.The instruction-response pairs are generated by an efficient instruction synthesizer built on open-source models.In our experiments, we synthesize 200M instruction-response pairs covering 40+ task categories to verify the effectiveness of Instruction Pre-Training.In pre-training from scratch, Instruction Pre-Training not only consistently enhances pre-trained base models but also benefits more from further instruction tuning.In continual pre-training, Instruction Pre-Training enables Llama3-8B to be comparable to or even outperform Llama3-70B.Our model, code, and data are available at https://github.com/microsoft/LMOps.Ins: When is the finale of season 7? Let's think step by step.

Daixuan Cheng, Shaohan Huang, Junyu Bi, Yuefeng Zhan, Jianfeng Liu, Yujing Wang, Hao Sun, Furu Wei, Weiwei Deng, Qi Zhang. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Vision-Language Pretraining (VLP) has significantly improved the performance of various vision-language tasks with the matching of images and texts. In this paper, we propose VL-Match, a Vision-Language framework with Enhanced Token-level and Instance-level Matching. At the token level, a Vision-Language Replaced Token Detection task is designed to boost the substantial interaction between text tokens and images, where the text encoder of VLP works as a generator to generate a corrupted text, and the multimodal encoder of VLP works as a discriminator to predict whether each text token in the corrupted text matches the image. At the instance level, in the Image-Text Matching task that judges whether an image-text pair is matched, we propose a novel bootstrapping method to generate hard negative text samples that are different from the positive ones only at the token level. In this way, we can force the network to detect fine-grained differences between images and texts. Notably, with a smaller amount of parameters, VL-Match significantly outperforms previous SOTA on all image-text retrieval tasks.

Single Image Super-Resolution (SISR) based on deep learning methods has been widely studied for applications on remote sensing images. With limited remote sensing images, most of the existing SISR methods simply adopt the regular data augmentation approaches (such as flip) in natural images to improve model performance. Considering the fact that remote sensing images are all taken from a bird’s-eye view and objects appear in multiple directions, we first introduce rotation augmentation method in remote sensing images to promote diversity of samples dramatically, as rotation does not cause semantic problems like people standing upside down in natural images. However, image rotation at various angles implemented by interpolation will cause the inconsistent pixel distribution problem for the pixel level task. Thus, we propose Transformation Consistency Loss Function (TCLF) to narrow the gap between the augmented and original distribution, while expanding the feature space with rotation augmentation method. Extensive experiments are performed on UC-Merced Land-use dataset of 21 remote sensing scenes, and the results as well as ablation studies demonstrate our proposed method outperforms mainstream methods.