Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through a language model API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

Abstract Many users of search systems are multilingual, that is, they are proficient in two or more languages. In order to better understand and support the language preferences and behaviors of such multilingual users, this paper presents a series of five large‐scale studies that specifically elicit language choices regarding search queries and result lists. Overall, the results from the studies indicate that users frequently make use of different languages (i.e., not just their primary language), especially when they are provided with choices (e.g., when provided with a secondary language query or result list choice). In particular, when presented with a mixed‐language list choice, participants choose this option to an almost equal extent compared to primary‐language‐only lists. Important factors leading to language choices are user‐, task‐ and system‐related, including proficiency, task topic, and result layout. Moreover, participants' subjective reasons for making particular choices indicate that their primary language is considered more comfortable, that the secondary language often has more relevant and trustworthy results, and that mixed‐language lists provide a better overview. These results provide crucial insights into multilingual user preferences and behaviors, and may help in the design of systems that can better support the querying and result exploration of multilingual users.

Evaluating the quality of a dialogue interaction between two agents is a difficult task, especially in open-domain chit-chat style dialogue.There have been recent efforts to develop automatic dialogue evaluation metrics, but most of them do not generalize to unseen datasets and/or need a human-generated reference response during inference, making it infeasible for online evaluation.Here, we propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances, and leverages the temporal transitions that exist between them.We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.

As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about---summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models. We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.

Combat medics care for battlefield casualties at point of injury (PoI), and stabilize and transport them to field care facilities. Casualty care requires complex decisions under austere and frequently dangerous conditions. Future operations are not expected to have immediate evacuation available, requiring more complex care and potential for complications including circulatory shock, which is a life-threatening state that can lead to organ failure and death if not detected and treated. We report on a project to develop decision support for combat medics (Role 1) and Battalion Aid Station/Field Hospital clinicians (Role 2) by combining resources of two proven intensive care unit (ICU) decision support systems. Machine learning algorithms are under development that will be used to detect the probability of shock in a casualty. Algorithms will be compared with actual clinical decisions in a “silent test.” A parallel effort is developing a software program for use with a vital signs sensor that can be installed on a DoD-approved version of the Samsung smart phone and tablet, then evaluated in the field by medics and clinicians. Faster, more accurate decisions can improve care for trauma patients, in circumstances where delays can increase morbidity and mortality.

The use of dialogue systems as a medium for human-machine interaction is an increasingly prevalent paradigm. A growing number of dialogue systems use conversation strategies that are learned from large datasets. There are well documented instances where interactions with these system have resulted in biased or even offensive conversations due to the data-driven training process. Here, we highlight potential ethical issues that arise in dialogue systems research, including: implicit biases in data-driven systems, the rise of adversarial examples, potential sources of privacy violations, safety concerns, special considerations for reinforcement learning systems, and reproducibility concerns. We also suggest areas stemming from these issues that deserve further investigation. Through this initial survey, we hope to spur research leading to robust, safe, and ethically sound dialogue systems.

Sequential data often possesses hierarchical structures with complex dependencies between sub-sequences, such as found between the utterances in a dialogue. To model these dependencies in a generative framework, we propose a neural network-based generative architecture, with stochastic latent variables that span a variable number of time steps. We apply the proposed model to the task of dialogue response generation and compare it with other recent neural-network architectures. We evaluate the model performance through a human evaluation study. The experiments demonstrate that our model improves upon recently proposed models and that the latent variables facilitate both the generation of meaningful, long and diverse responses and maintaining dialogue state.

Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, Joelle Pineau. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2017.

Humans interpret texts with respect to some background information, or world knowledge, and we would like to develop automatic reading comprehension systems that can do the same.In this paper, we introduce a task and several models to drive progress towards this goal.In particular, we propose the task of rare entity prediction: given a web document with several entities removed, models are tasked with predicting the correct missing entities conditioned on the document context and the lexical resources.This task is challenging due to the diversity of language styles and the extremely large number of rare entities.We propose two recurrent neural network architectures which make use of external knowledge in the form of entity descriptions.Our experiments show that our hierarchical LSTM model performs significantly better at the rare entity prediction task than those that do not make use of external resources.

We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.

The unprecedented increase in online search user diversity across the globe has led to new challenges for search engine providers. In particular, among these challenges is the need to better support individuals who are proficient in multiple languages. To investigate this particular user characteristic, this paper presents an analysis of multilingual search user behaviors through a series of large-scale studies using crowdsourcing. Results show that multilingual users make significant use of each of their languages when searching, and that there are significant differences in behaviors between querying and result selection. In addition, results show that language use strongly depends on a number of task factors and individual user characteristics. These results are discussed in terms of building novel adaptive multilingual search solutions that better support and adapt to users who have multiple language abilities.

We investigate evaluation metrics for dialogue response generation systems where supervised labels, such as task completion, are not available.Recent works in response generation have adopted metrics from machine translation to compare a model's generated response to a single target response.We show that these metrics correlate very weakly with human judgements in the non-technical Twitter domain, and not at all in the technical Ubuntu domain.We provide quantitative and qualitative results highlighting specific weaknesses in existing metrics, and provide recommendations for future development of better automatic evaluation metrics for dialogue systems.

An open challenge in constructing dialogue systems is developing methods for automatically learning dialogue strategies from large amounts of unlabelled data. Recent work has proposed Next-Utterance-Classification (NUC) as a surrogate task for building dialogue systems from text data. In this paper we investigate the performance of humans on this task to validate the relevance of NUC as a method of evaluation. Our results show three main findings: (1) humans are able to correctly classify responses at a rate much better than chance, thus confirming that the task is feasible, (2) human performance levels vary across task domains (we consider 3 datasets) and expertise levels (novice vs experts), thus showing that a range of performance is possible on this type of task, (3) automated dialogue systems built using state-of-the-art machine learning methods have similar performance to the human novices, but worse than the experts, thus confirming the utility of this class of tasks for driving further research in automated dialogue systems.

This paper introduces the Ubuntu Dialogue Corpus, a dataset containing almost 1 million multi-turn dialogues, with a total of over 7 million utterances and 100 million words.This provides a unique resource for research into building dialogue managers based on neural language models that can make use of large amounts of unlabeled data.The dataset has both the multi-turn property of conversations in the Dialog State Tracking Challenge datasets, and the unstructured nature of interactions from microblog services such as Twitter.We also describe two neural learning architectures suitable for analyzing this dataset, and provide benchmark performance on the task of selecting the best next response.

A technique has been developed for monitoring the structural damage accrued in ball grid array interconnects during operation in vibration environments. The technique uses resistance spectroscopy based state space vectors, rate of change of the state variable, and acceleration of the state variable in conjunction with extended Kalman filter, and is intended for the pre-failure time-history of the component. Condition monitoring using the presented technique can provide knowledge of impending failure in high reliability applications where the risks associated with loss-of-functionality are too high to bear. The methodology has been demonstrated on 96.5%Sn3.0%Ag0.5%Cu (SAC305) lead-free area-array electronic assemblies subjected to vibration. The future state of the system has been estimated based on a second order extended Kalman filter model and a Bayesian Framework. The measured state variable has been related to the underlying interconnect damage using plastic strain. Performance of the prognostication health management algorithm during the vibration test has been quantified using performance evaluation metrics. Model predictions have been correlated with experimental data. The presented approach is applicable to functional systems where corner interconnects in area-array packages may be often redundant. Prognostic metrics including α-λ metric, beta, and relative accuracy have been used to assess the performance of the damage proxies. The presented approach enables the estimation of residual life based on level of risk averseness.