We propose a new paradigm to continually evolve pretrained models, denoted ColD Fusion.It provides the benefits of multitask learning but leverages distributed computation with limited communication and eliminates the need for shared data.Consequentially, ColD Fusion can give rise to a synergistic loop, where finetuned models can be recycled to continually improve the pretrained model they are based upon.We show that ColD Fusion yields comparable benefits to multitask training by producing a model that (a) attains strong performance on all of the datasets it was trained on; and (b) is a better starting point for finetuning on unseen datasets.We show that ColD Fusion outperforms RoBERTa and even previous multitask models.Specifically, when training and testing on 35 diverse datasets, ColD Fusion-based model outperforms RoBERTa by 2.33 points on average without any changes to the architecture.1

Previous studies observed that finetuned models may be better base models than the vanilla pretrained model.Such a model, finetuned on some source dataset, may provide a better starting point for a new finetuning process on a desired target dataset.Here, we perform a systematic analysis of this intertraining scheme, over a wide range of English classification tasks.Surprisingly, our analysis suggests that the potential intertraining gain can be analyzed independently for the target dataset under consideration, and for a base model being considered as a starting point.Hence, a performant model is generally strong, even if its training data was not aligned with the target dataset.Furthermore, we leverage our analysis to propose a practical and efficient approach to determine if and how to select a base model in real-world settings.Last, we release an updating ranking of best models in the HuggingFace hub per architecture.

Assaf Toledo, Shai Gretz, Edo Cohen-Karlik, Roni Friedman, Elad Venezian, Dan Lahav, Michal Jacovi, Ranit Aharonov, Noam Slonim. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.