Recent work has shown that large pretrained Language Models (LMs) can not only perform remarkably well on a range of Natural Language Processing (NLP) tasks but also start improving on reasoning tasks such as arithmetic induction, symbolic manipulation, and commonsense reasoning with increasing size of models (Wei et al., 2022;Chowdhery et al., 2022).However, it is still unclear what the underlying capabilities of these LMs are.Surprisingly, we find that these models have limitations on certain basic symbolic manipulation tasks such as copy, reverse, and addition.When the total number of symbols or repeating symbols increases, the model performance drops quickly.We investigate the potential causes behind this phenomenon and examine a set of possible methods, including explicit positional markers, fine-grained computation steps, and LMs with callable programs.Experimental results show that none of these techniques can solve the simplest addition induction problem completely.In the end, we introduce LMs with tutor, which demonstrates every single step of teaching.LMs with tutor is able to deliver 100% accuracy in situations of OOD and repeating symbols, shedding new insights on the boundary of large LMs in induction.* The first two authors (Jing and Hong) contributed equally to this work.

Commonly used information retrieval methods such as TF-IDF in open-domain question answering (QA) systems are insufficient to capture deep semantic matching that goes beyond lexical overlaps.Some recent studies consider the retrieval process as maximum inner product search (MIPS) using dense question and paragraph representations, achieving promising results on several informationseeking QA datasets.However, the pretraining of the dense vector representations is highly resource-demanding, e.g., requires a very large batch size and lots of training steps.In this work, we propose a sample-efficient method to pretrain the paragraph encoder.First, instead of using heuristically created pseudo questionparagraph pairs for pretraining, we use an existing pretrained sequence-to-sequence model to build a strong question generator that creates high-quality pretraining data.Second, we propose a simple progressive pretraining algorithm to ensure the existence of effective negative samples in each batch.Across three opendomain QA datasets, our method consistently outperforms a strong dense retrieval baseline that uses 6 times more computation for training.On two of the datasets, our method achieves more than 4-point absolute improvement in terms of answer exact match.

Existing work on automated hate speech classification assumes that the dataset is fixed and the classes are pre-defined.However, the amount of data in social media increases every day, and the hot topics changes rapidly, requiring the classifiers to be able to continuously adapt to new data without forgetting the previously learned knowledge.This ability, referred to as lifelong learning, is crucial for the realword application of hate speech classifiers in social media.In this work, we propose lifelong learning of hate speech classification on social media.To alleviate catastrophic forgetting, we propose to use Variational Representation Learning (VRL) along with a memory module based on LB-SOINN (Load-Balancing Self-Organizing Incremental Neural Network).Experimentally, we show that combining variational representation learning and the LB-SOINN memory module achieves better performance than the commonly-used lifelong learning techniques.

Existing models for extractive summarization are usually trained from scratch with a crossentropy loss, which does not explicitly capture the global context at the document level.In this paper, we aim to improve this task by introducing three auxiliary pre-training tasks that learn to capture the document-level context in a self-supervised fashion.Experiments on the widely-used CNN/DM dataset validate the effectiveness of the proposed auxiliary tasks.Furthermore, we show that after pretraining, a clean model with simple building blocks is able to outperform previous state-ofthe-art that are carefully designed.1

With social media becoming increasingly popular on which lots of news and real-time events are reported, developing automated question answering systems is critical to the effective-ness of many applications that rely on real-time knowledge. While previous datasets have concentrated on question answering (QA) for formal text like news and Wikipedia, we present the first large-scale dataset for QA over social media data. To ensure that the tweets we collected are useful, we only gather tweets used by journalists to write news articles. We then ask human annotators to write questions and answers upon these tweets. Unlike otherQA datasets like SQuAD in which the answers are extractive, we allow the answers to be abstractive. We show that two recently proposed neural models that perform well on formal texts are limited in their performance when applied to our dataset. In addition, even the fine-tuned BERT model is still lagging behind human performance with a large margin. Our results thus point to the need of improved QA systems targeting social media text.