Modern Natural Language Generation (NLG) models come with massive computational and storage requirements.In this work, we study the potential of compressing them, which is crucial for real-world applications serving millions of users.We focus on Knowledge Distillation (KD) techniques, in which a small student model learns to imitate a large teacher model, allowing to transfer knowledge from the teacher to the student.In contrast to much of the previous work, our goal is to optimize the model for a specific NLG task and a specific dataset.Typically in real-world applications, in addition to labeled data there is abundant unlabeled task-specific data, which is crucial for attaining high compression rates via KD.In this work, we conduct a systematic study of task-specific KD techniques for various NLG tasks under realistic assumptions.We discuss the special characteristics of NLG distillation and particularly the exposure bias problem.Following, we derive a family of Pseudo-Target (PT) augmentation methods, substantially extending prior work on sequence-level KD.We propose the Joint-Teaching method, which applies wordlevel KD to multiple PTs generated by both the teacher and the student.Finally, we validate our findings in an extreme setup with no labeled examples using GPT-4 as the teacher.Our study provides practical model design observations and demonstrates the effectiveness of PT training for task-specific KD in NLG.

Based on recent advances in natural language modeling and those in text generation capabilities, we propose a novel data augmentation method for text classification tasks. We use a powerful pre-trained neural network model to artificially synthesize new labeled data for supervised learning. We mainly focus on cases with scarce labeled data. Our method, referred to as language-model-based data augmentation (LAMBADA), involves fine-tuning a state-of-the-art language generator to a specific task through an initial training phase on the existing (usually small) labeled data. Using the fine-tuned model and given a class label, new sentences for the class are generated. Our process then filters these new sentences by using a classifier trained on the original data. In a series of experiments, we show that LAMBADA improves classifiers' performance on a variety of datasets. Moreover, LAMBADA significantly improves upon the state-of-the-art techniques for data augmentation, specifically those applicable to text classification tasks with little data.

User generated reviews is a highly informative source of information, that has recently gained lots of attention in the recommender systems community. In this work we propose a generative latent variable model that explains both observed ratings and textual reviews. This latent variable model allows to combine any traditional collaborative filtering method, together with any deep learning architecture for text processing. Experimental results on four benchmark datasets demonstrate its superiority comparing to all baseline recommender systems. Furthermore, a running time analysis shows that this approach is in order of magnitude faster that relevant baselines. Moreover, underlying our solution there is a general framework that may be further explored.

This paper introduces a new family of redundancy schemes for distributed storage systems, called replicated erasure codes (REC), which combine the storage-space efficiency of erasure codes and the repair-traffic efficiency of replication. A formal model for analyzing the storage and repair-traffic costs under availability and persistency constraints is also developed. It is shown that under parameters that characterize common P2P environments, REC generally achieves better results than each of the two methods separately.

We present a method and a tool for composing a reactive system and for accompanying the development and documentation process with a proof of its correctness. The approach is based on behavioral programming (BP) and the Z3 SMT solver. We show how program verification can be automated and streamlined by combining properties of individual modules, specified and verified separately, with application-independent specifications both of the BP semantics and of general theories. The method may yield an exponential acceleration of the verification process when compared with model-checking the composite application. We show that formalization of properties of independent modules in preparation for the correctness proofs can be useful as documentation for future development. We view this work as a further step towards making formal correctness proofs standard practice in the development of reactive systems, and carried out by programmers at large.

Live sequence charts (LSC) is a visual, executable, language for the modeling of reactive systems. Each chart depicts an inter-object scenario arising in the modeled system, partitioned into two: a monitored prechart, and a main chart. Despite the intuitive use of the language, complications arise when one wants to implement an LSC specification with decentralized components. In this paper, we introduce a variant of LSC, called distributed LSC (dLSC), which is targeted for the modeling and synthesis of decentralized systems, composed of several interacting components. While LSCs are commonly interpreted in terms of an interleaved execution of the scenarios in a sequential run, dLSCs employ partially ordered runs. We investigate the expressive power of dLSC compared to an established model of concurrent systems, namely, Petri nets, and show that dLSCs are, computationally, strictly more expressive than low-level Petri nets and subsumed by higher-level Petri nets. Specifically, we present an algorithm that synthesizes, given a dLSC specification, an equivalent token history net, which can serve as an executable implementation of the specification. Most importantly, the implementation is decentralized - components can be automatically extracted from the net. The synthesis of Petri-net components from a dLSC specification is supported by a tool.