Reinforcement Learning with Human Feedback (RLHF) is the most prominent method for Language Model (LM) alignment. However, RLHF is an unstable and data-hungry process that continually requires new high-quality LM-generated data for finetuning. We introduce Advantage-Leftover Lunch RL (A-LoL), a new class of offline policy gradient algorithms that enable RL training on any pre-existing data. By assuming the entire LM output sequence as a single action, A-LoL allows incorporating sequence-level classifiers or human-designed scoring functions as rewards. Subsequently, by using LM’s value estimate, A-LoL only trains on positive advantage (leftover) data points, making it resilient to noise. Overall, A-LoL is an easy-to-implement, sample-efficient, and stable LM training recipe. We demonstrate the effectiveness of A-LoL and its variants with a set of four different language generation tasks. We compare against both online RL (PPO) and recent preference-based (DPO, PRO) and reward-based (GOLD) offline RL baselines. On the commonly-used RLHF benchmark, Helpful and Harmless Assistant (HHA), LMs trained with A-LoL methods achieve the highest diversity while also being rated more safe and helpful than the baselines according to humans. Additionally, in the remaining three tasks, A-LoL could optimize multiple distinct reward functions even when using noisy or suboptimal training data.

We present a manually annotated corpus of 10,000 tweets containing public reports of five COVID-19 events, including positive and negative tests, deaths, denied access to testing, claimed cures and preventions. We designed slot-filling questions for each event type and annotated a total of 28 fine-grained slots, such as the location of events, recent travel, and close contacts. We show that our corpus can support fine-tuning BERT-based classifiers to automatically extract publicly reported events, which can be further collected for building a knowledge base. Our knowledge base is constructed over Twitter data covering two years and currently covers over 4.2M events. It can answer complex queries with high precision, such as “Which organizations have employees that tested positive in Philadelphia?” We believe our proposed methodology could be quickly applied to develop knowledge bases for new domains in response to an emerging crisis, including natural disasters or future disease outbreaks.

We present Stanceosaurus, a new corpus of 28,033 tweets in English, Hindi, and Arabic annotated with stance towards 251 misinformation claims.As far as we are aware, it is the largest corpus annotated with stance towards misinformation claims.The claims in Stanceosaurus originate from 15 fact-checking sources that cover diverse geographical regions and cultures.Unlike existing stance datasets, we introduce a more fine-grained 5class labeling strategy with additional subcategories to distinguish implicit stance.Pretrained transformer-based stance classifiers that are fine-tuned on our corpus show good generalization on unseen claims and regional claims from countries outside the training data.Cross-lingual experiments demonstrate Stanceosaurus' capability of training multilingual models, achieving 53.1 F1 on Hindi and 50.4 F1 on Arabic without any targetlanguage fine-tuning.Finally, we show how a domain adaptation method can be used to improve performance on Stanceosaurus using additional RumourEval-2019 data.We make Stanceosaurus publicly available to the research community and hope it will encourage further work on misinformation identification across languages and cultures.1 Source Country & Regions Lang #Claims #Tweets Irr.Sup.Ref. Dis.Que.Snopes USA (80%), INT'L (16.7%),Other (3.3%) en 30 3197 1051 428 229 1447 42 Poynter Europe (5%), INT'L (90%), Other (5%) en 20 2197 949 274 97 844 33 FullFact UK (30%), INT'L (55%), Other (15%) en 20 2379 806 300 179 1057 37 AFP Fact Check CAN Canada (55%), INT'L (30%), Other (15%) en 20 2078 746 252 130 910 40 AAP Fact Check Australia (10%), INT'L (65%), Other (25%) en 20 2302 739 374 136 1019 34 AFP Fact Check NZ New Zealand (15%), INT'L (75%), Other (10%) en 20 2227 879 194 81 1044 29 Blackdotresearch Singapore (30%), INT'L (55%), Other (15%) en 20 2307 842 248 113 1076 28 Factly India (45%), INT'L (55%) en 20 1979 889 190 117 734 49 Politifact USA (20%), INT'L (35%), Other (45%) en 20 2041 984 289 8 753 7 Alt News India (90.4%),INT'L (4.8%),Other (4.8%) hi 21 1730 550 489 172 500 19 Aajtak India (67%), Other (33%) hi 9 806 456 110 40 193 7 Hindi Newschecker India (56%), Other (44%) hi 9 781 195 313 46 219 8 MISBAR Arab World (58.3%),INT'L (8.3%),Other (33.4%) ar 12 2283 454 514 203 1031 81 Fatabyyano Arab World (28.5%),INT'L (57.1%),Other (14.4%) ar 7 986 234 163 49 522 18 Maharat Fact-o-meter INT'L (100%) ar 3 740

Dialogue models trained on human conversations inadvertently learn to generate toxic responses.In addition to producing explicitly offensive utterances, these models can also implicitly insult a group or individual by aligning themselves with an offensive statement.To better understand the dynamics of contextually offensive language, we investigate the stance of dialogue model responses in offensive Reddit conversations.Specifically, we create TOXICHAT, a crowd-annotated dataset of 2,000 Reddit threads and model responses labeled with offensive language and stance.Our analysis reveals that 42% of human responses agree with toxic comments, whereas only 13% agree with safe comments.This undesirable behavior is learned by neural dialogue models, such as DialoGPT, which we show are two times more likely to agree with offensive comments.To enable automatic detection of offensive language, we fine-tuned transformerbased classifiers on TOXICHAT that achieve 0.71 F 1 for offensive labels and 0.53 Macro-F 1 for stance labels.Finally, we quantify the effectiveness of controllable text generation (CTG) methods to mitigate the tendency of neural dialogue models to agree with offensive comments.Compared to the baseline, our best CTG model achieves a 19% reduction in agreement with offensive comments and produces 29% fewer offensive replies.Our work highlights the need for further efforts to characterize and analyze inappropriate behavior in dialogue models, in order to help make them safer. 1

Question answering (QA) is an important aspect of open-domain conversational agents, garnering specific research focus in the conversational QA (ConvQA) subtask.One notable limitation of recent ConvQA efforts is the response being answer span extraction from the target corpus, thus ignoring the natural language generation (NLG) aspect of high-quality conversational agents.In this work, we propose a method for situating QA responses within a SEQ2SEQ NLG approach to generate fluent grammatical answer responses while maintaining correctness.From a technical perspective, we use data augmentation to generate training data for an end-to-end system.Specifically, we develop Syntactic Transformations (STs) to produce question-specific candidate answer responses and rank them using a BERT-based classifier (Devlin et al., 2019).Human evaluation on SQuAD 2.0 data (Rajpurkar et al., 2018) demonstrate that the proposed model outperforms baseline CoQA and QuAC models in generating conversational responses.We further show our model's scalability by conducting tests on the CoQA dataset. 1

Neural conversation models tend to generate safe, generic responses for most inputs.This is due to the limitations of likelihoodbased decoding objectives in generation tasks with diverse outputs, such as conversation.To address this challenge, we propose a simple yet effective approach for incorporating side information in the form of distributional constraints over the generated responses.We propose two constraints that help generate more content rich responses that are based on a model of syntax and topics (Griffiths et al., 2005) and semantic similarity (Arora et al., 2016).We evaluate our approach against a variety of competitive baselines, using both automatic metrics and human judgments, showing that our proposed approach generates responses that are much less generic without sacrificing plausibility.A working demo of our code can be found at https://github.com/abaheti95/ DC-NeuralConversation.

A belt region is said to be k-barrier covered by a set of sensors if all paths crossing the width of the belt region intersect the sensing regions of at least k sensors. Barrier coverage can be achieved from a random initial deployment of mobile sensors by suitably relocating the sensors to form a barrier. Reducing the movement of the sensors is important in such scenarios due to the energy constraints of sensor devices. In this paper, we propose a centralized algorithm which achieves 1-barrier coverage by forming a non-linear barrier from a random initial deployment of sensors in a belt. The algorithm uses a novel idea of physical behavior of chains along with the concept of virtual force. Formation of non-linear barrier reduces the movement of the sensors needed as compared to linear barriers. Detailed simulation results are presented to show that the proposed algorithm achieves barrier coverage with less displacement of sensors compared to other existing algorithms in the literature.