Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Large language models are able to perform a task by conditioning on a few input-output demonstrations -a paradigm known as incontext learning.We show that language models can explicitly infer an underlying task from a few demonstrations by prompting them to generate a natural language instruction that fits the examples.To explore this ability, we introduce the instruction induction challenge, compile a dataset consisting of 24 tasks, and define a novel evaluation metric based on executing the generated instruction.We discover that, to a large extent, the ability to generate instructions does indeed emerge when using a model that is both large enough and aligned to follow instructions; InstructGPT achieves 65.7% of human performance in our execution-based metric, while the original GPT-3 model reaches only 9.8% of human performance.This surprising result suggests that instruction induction might be a viable learning paradigm in and of itself, where instead of fitting a set of latent continuous parameters to the data, one searches for the best description in the natural language hypothesis space. 1

Instruction tuning enables pretrained language models to perform new tasks from inferencetime natural language descriptions.These approaches rely on vast amounts of human supervision in the form of crowdsourced datasets or user interactions.In this work, we introduce Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor.We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth.This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs.Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training on open-source manually-curated datasets, surpassing the performance of models such as T0++ and Tk-Instruct across various benchmarks.These results demonstrate the potential of model-generated data as a cost-effective alternative to crowdsourcing for dataset expansion and diversification. Example 1Instruction: You are given a science question (easy-level) and four answer options (associated with "A", "B", "C", "D").Your task is to find the correct answer based on scientific facts, knowledge, and reasoning.Do not generate anything else apart from one of the following characters: 'A', 'B, 'C', 'D'.There is only one correct answer for each question. Input: Which part of a bicycle BEST moves in a circle? (A) Seat (B) Frame (C) Foot pedal (D) KickstandConstraints: The output should be one of the following characters: 'A', 'B, 'C', 'D'. Example 2Instruction: You are given a negative review and your task is to convert it to a positive review by one or more making minimal changes.Avoid changing the context of the review.Input: we stood there in shock, because we never expected this. Constraints: None.Example 3 Instruction: In this task, you are given two sentences taken from a conversation, and your job is to classify whether these given sentences are sequential or not.We will mark the given sentence pair as 'True' if it's sequential, otherwise 'False'.The two sentences are spoken by two different people.

Ella Neeman, Roee Aharoni, Or Honovich, Leshem Choshen, Idan Szpektor, Omri Abend. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, Yossi Matias. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability.Inspired by recent work on evaluating factual consistency in abstractive summarization, we propose an automatic evaluation metric for factual consistency in knowledge-grounded dialogue using automatic question generation and question answering.Our metric, denoted Q 2 , compares answer spans using natural language inference (NLI), instead of token-based matching as done in previous work.To foster proper evaluation, we curate a novel dataset of dialogue system outputs for the Wizard-of-Wikipedia dataset, manually annotated for factual consistency.We perform a thorough meta-evaluation of Q 2 against other metrics using this dataset and two others, where it consistently shows higher correlation with human judgements.

Machine reading is an ambitious goal in NLP that subsumes a wide range of text understanding capabilities.Within this broad framework, we address the task of machine reading the time of historical events, compile datasets for the task, and develop a model for tackling it.Given a brief textual description of an event, we show that good performance can be achieved by extracting relevant sentences from Wikipedia, and applying a combination of taskspecific and general-purpose feature embeddings for the classification.Furthermore, we establish a link between the historical event ordering task and the event focus time task from the information retrieval literature, showing they also provide a challenging test case for machine reading algorithms. 1 1 Code and data are available at https://github.com/ltorroba/ machine-reading-historical-events.* Equal contribution.Year Event text OTD 2005 107 die in Amagasaki rail crash in Japan.1939 BMI (Broadcast Music Incorporated) formed.1864 General Sherman's armies reach Savannah & 12 day siege begins.WOTD 1887 Buffalo Bill Cody's Wild West Show opens in London.1399 Henry IV is proclaimed King of England.1943 First Flight of the Gloster Meteor, Britain's first combat jet aircraft.