With the goal of benchmarking generative systems beyond expert software development ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from scratch. Agents are provided with a specification document outlining the library’s API as well as a suite of interactive unit tests, with the goal of producing an implementation of this API accordingly. The implementation is validated through running these unit tests. As a benchmark, Commit0 is designed to move beyond static one-shot code generation towards agents that must process long-form natural language specifications, adapt to multi-stage feedback, and generate code with complex dependencies. Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate. Our experiments demonstrate that while current agents can pass some unit tests, none can yet fully reproduce full libraries. Results also show that interactive feedback is quite useful for models to generate code that passes more unit tests, validating the benchmarks that facilitate its use. We publicly release the benchmark, the interactive environment, and the leaderboard.

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

We address the problem of unsupervised abstractive summarization of collections of user generated reviews through self-supervision and control.We propose a self-supervised setup that considers an individual document as a target summary for a set of similar documents.This setting makes training simpler than previous approaches by relying only on standard log-likelihood loss and mainstream models.We address the problem of hallucinations through the use of control codes, to steer the generation towards more coherent and relevant summaries.Our benchmarks on two English datasets against graph-based and recent neural abstractive unsupervised models show that our proposed method generates summaries with a superior quality and relevance, as well as a high sentiment and topic alignment with the input reviews.This is confirmed in our human evaluation which focuses explicitly on the faithfulness of generated summaries.We also provide an ablation study showing the importance of the control setup in controlling hallucinations.

We consider the problem of multilingual unsupervised machine translation, translating to and from languages that only have monolingual data by using auxiliary parallel language pairs.For this problem the standard procedure so far to leverage the monolingual data is back-translation, which is computationally costly and hard to tune.In this paper we propose instead to use denoising adapters, adapter layers with a denoising objective, on top of pre-trained mBART-50.In addition to the modularity and flexibility of such an approach we show that the resulting translations are on-par with back-translating as measured by BLEU, and furthermore it allows adding unseen languages incrementally.

There is an ongoing debate in the NLP community whether modern language models contain linguistic knowledge, recovered through so-called probes. In this paper, we study whether linguistic knowledge is a necessary condition for the good performance of modern language models, which we call the rediscovery hypothesis. In the first place, we show that language models that are significantly compressed but perform well on their pretraining objectives retain good scores when probed for linguistic structures. This result supports the rediscovery hypothesis and leads to the second contribution of our paper: an information-theoretic framework that relates language modeling objectives with linguistic information. This framework also provides a metric to measure the impact of linguistic information on the word prediction task. We reinforce our analytical results with various experiments, both on synthetic and on real NLP tasks in English.

We propose a novel adapter layer formalism for adapting multilingual models.They are more parameter-efficient than existing adapter layers while obtaining as good or better performance.The layers are specific to one language (as opposed to bilingual adapters) allowing to compose them and generalize to unseen language-pairs.In this zero-shot setting, they obtain a median improvement of +2.77BLEU points over a strong 20-language multilingual Transformer baseline trained on TED talks.

Hady Elsahar, Matthias Gallé. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Byte-Pair Encoding (BPE) is an unsupervised sub-word tokenization technique, commonly used in neural machine translation and other NLP tasks.Its effectiveness makes it a de facto standard, but the reasons for this are not well understood.We link BPE to the broader family of dictionary-based compression algorithms and compare it with other members of this family.Our experiments across datasets, language pairs, translation models, and vocabulary size show that -given a fixed vocabulary size budget -the fewer tokens an algorithm needs to cover the test set, the better the translation (as measured by BLEU).

We present a solution to scale spectral algorithms for learning sequence functions. We are interested in the case where these functions are sparse (that is, for most sequences they return 0). Spectral algorithms reduce the learning problem to the task of computing an SVD decomposition over a special type of matrix called the Hankel matrix. This matrix is designed to capture the relevant statistics of the training sequences. What is crucial is that to capture long range dependencies we must consider very large Hankel matrices. Thus the computation of the SVD becomes a critical bottleneck. Our solution finds a subset of rows and columns of the Hankel that realizes a compact and informative Hankel submatrix. The novelty lies in the way that this subset is selected: we exploit a maximal bipartite matching combinatorial algorithm to look for a sub-block with full structural rank, and show how computation of this sub-block can be further improved by exploiting the specific structure of Hankel matrices.

We study the problem of handling the inter-turn pauses in a human-robot dialogue. In order to reduce the impression of elapsed time while the robot transcribes, understands and starts uttering a response we propose to automatically generate conversational fillers, to fill the silences. These fillers combine verbal utterances with body movements. We propose a Bayesian model that samples filler whose production duration time is close to the expected computational time needed by the robot. To increase the sensation of engagement, the fillers also include contextual information gathered during the dialogue (such as the name of the interlocutor), if this information is present with high confidence. We evaluate this approach with an indirect user study measuring time perception, comparing three different strategies to overcome the inter-turn time (silence, static filler and our approach). The results show that users prefer the dynamic fillers, even when the conversation is objectively shorter with one of the other strategies.

The World Wide Web contains a large quantity of community created knowledge of instructional nature. Similarly, in a commercial setting, databases of instructions are used by customer-care providers to guide clients in the resolution of issues. Most of these instructions are expressed in natural la nguage. Knowledge Bases including such information are valuable through the sum of their single entries. However, as each entry is created mostly independently, users (e.g. other community members) cannot take advantage of the accumulated knowledge that can be developed via the aggregation of related entries and of relevant information that can be found in external knowledge bases. In this paper we consider the problem of linking Knowledge Base entries to other relevant parts of the Knowledge Base and to third-party semi-structured knowledge sources. To achieve this, we propose (i) a new method to detect actionable phrases – text fragments that describe how to perform a certain action – and link them to other entries; and (ii) a new method to detect entities and link them to the Linked Open Data cloud. The method that we implemented for extracting actionable phrases achieves an F-score of 67.35%. We show that limiting the linking space to actionable phrases results in better linking quality than using coarser-grained spans of text, as proposed in other approaches to the task. In addition, we propose a new Linked Open Data linking method that uses a global optimization score to filter the set of possible candidates, increasing precision when compared to a standard method implemented in dbpedia-spotlight. Besides the above scientific contributions, we also present a detailed error analysis, and release our annotations to the community to foster future research on the subject.

We analyze the problem of reconstructing documents when we only have access to the n-grams (for n fixed) and their counts from the original documents. Formally, we are interested in recovering the longest contiguous substrings of whose presence in the original documents we are certain. We map this problem on a de Bruijn graph, where the n-grams form the edges and where every Eulerian cycles gives a plausible reconstruction. We define two rules that reduce this graph, preserving all possible reconstructions while at the same time increasing the length of the edge labels. From a theoretical perspective we prove that the iterative application of these rules gives an irreducible graph equivalent to the original one. We then apply this on the data from the Gutenberg project to measure the number and size of the obtained longest substrings. Moreoever, we analyze how the n-gram corpus could be noised to prevent reconstruction, showing empirically that removing low frequent n-grams has little impact. Instead, we propose another method consisting in adding strategically fictitious n-grams and show that a noised corpus like that is much harder to reconstruct while increasing only little the perplexity of a language model obtained through it.

n-gram representations of documents may improve over a simple bag-of-word representation by relaxing the independence assumption of word and introducing context. However, this comes at a cost of adding features which are non-descriptive, and increasing the dimension of the vector space model exponentially.