Traditionally, a debate usually requires a manual preparation process, including reading plenty of articles, selecting the claims, identifying the stances of the claims, seeking the evidence for the claims, etc.As the AI debate attracts more attention these years, it is worth exploring the methods to automate the tedious process involved in the debating system.In this work, we introduce a comprehensive and large dataset named IAM, which can be applied to a series of argument mining tasks, including claim extraction, stance classification, evidence extraction, etc.Our dataset is collected from over 1k articles related to 123 topics.Near 70k sentences in the dataset are fully annotated based on their argument properties (e.g., claims, stances, evidence, etc.).We further propose two new integrated argument mining tasks associated with the debate preparation process: (1) claim extraction with stance classification (CESC) and (2) claim-evidence pair extraction (CEPE).We adopt a pipeline approach and an end-to-end method for each integrated task separately.Promising experimental results are reported to show the values and challenges of our proposed tasks, and motivate future research on argument mining. 1

Data augmentation is an effective solution to data scarcity in low-resource scenarios.However, when applied to token-level tasks such as NER, data augmentation methods often suffer from token-label misalignment, which leads to unsatsifactory performance.In this work, we propose Masked Entity Language Modeling (MELM) as a novel data augmentation framework for low-resource NER.To alleviate the token-label misalignment issue, we explicitly inject NER labels into sentence context, and thus the fine-tuned MELM is able to predict masked entity tokens by explicitly conditioning on their labels.Thereby, MELM generates high-quality augmented data with novel entities, which provides rich entity regularity knowledge and boosts NER performance.When training data from multiple languages are available, we also integrate MELM with codemixing for further improvement.We demonstrate the effectiveness of MELM on monolingual, cross-lingual and multilingual NER across various low-resource levels.Experimental results show that our MELM presents substantial improvement over the baseline methods. 1

While there is much research on cross-domain text classification, most existing approaches focus on one-to-one or many-to-one domain adaptation. In this paper, we tackle the more challenging task of domain generalization, in which domain-invariant representations are learned from multiple source domains, without access to any data from the target domains, and classification decisions are then made on test documents in unseen target domains. We propose a novel framework based on supervised contrastive learning with a memory-saving queue. In this way, we explicitly encourage examples of the same class to be closer and examples of different classes to be further apart in the embedding space. We have conducted extensive experiments on two Amazon review sentiment datasets, and one rumour detection dataset. Experimental results show that our domain generalization method consistently outperforms state-of-the-art domain adaptation methods.

Knowledge-enhanced language representation learning has shown promising results across various knowledge-intensive NLP tasks.However, prior methods are limited in efficient utilization of multilingual knowledge graph (KG) data for language model (LM) pretraining.They often train LMs with KGs in indirect ways, relying on extra entity/relation embeddings to facilitate knowledge injection.In this work, we explore methods to make better use of the multilingual annotation and language agnostic property of KG triples, and present novel knowledge based multilingual language models (KMLMs) trained directly on the knowledge triples.We first generate a large amount of multilingual synthetic sentences using the Wikidata KG triples.Then based on the intra-and inter-sentence structures of the generated data, we design pretraining tasks to enable the LMs to not only memorize the factual knowledge but also learn useful logical patterns.Our pretrained KMLMs demonstrate significant performance improvements on a wide range of knowledge-intensive crosslingual tasks, including named entity recognition (NER), factual knowledge retrieval, relation classification, and a newly designed logical reasoning task. 1

Yan Zhang, Ruidan He, Zuozhu Liu, Lidong Bing, Haizhou Li. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Many efforts have been made in solving the Aspect-based sentiment analysis (ABSA) task.While most existing studies focus on English texts, handling ABSA in resource-poor languages remains a challenging problem.In this paper, we consider the unsupervised crosslingual transfer for the ABSA task, where only labeled data in the source language is available and we aim at transferring its knowledge to the target language having no labeled data.To this end, we propose an alignment-free label projection method to obtain high-quality pseudolabeled data of the target language with the help of the translation system, which could preserve more accurate task-specific knowledge in the target language.For better utilizing the source and translated data, as well as enhancing the cross-lingual alignment, we design an aspect code-switching mechanism to augment the training data with code-switched bilingual sentences.To further investigate the importance of language-specific knowledge in solving the ABSA problem, we distill the above model on the unlabeled target language data which improves the performance to the same level of the supervised method.

Adapting pre-trained language models (PrLMs) (e.g., BERT) to new domains has gained much attention recently.Instead of fine-tuning PrLMs as done in most previous work, we investigate how to adapt the features of PrLMs to new domains without fine-tuning.We explore unsupervised domain adaptation (UDA) in this paper.With the features from PrLMs, we adapt the models trained with labeled data from the source domain to the unlabeled target domain.Self-training is widely used for UDA, and it predicts pseudo labels on the target domain data for training.However, the predicted pseudo labels inevitably include noise, which will negatively affect training a robust model.To improve the robustness of self-training, in this paper we present class-aware feature self-distillation (CFd) to learn discriminative features from PrLMs, in which PrLM features are self-distilled into a feature adaptation module and the features from the same class are more tightly clustered.We further extend CFd to a cross-language setting, in which language discrepancy is studied.Experiments on two monolingual and multilingual Amazon review datasets show that CFd can consistently improve the performance of self-training in cross-domain and cross-language settings.

BERT is inefficient for sentence-pair tasks such as clustering or semantic search as it needs to evaluate combinatorially many sentence pairs which is very time-consuming.Sentence BERT (SBERT) attempted to solve this challenge by learning semantically meaningful representations of single sentences, such that similarity comparison can be easily accessed.However, SBERT is trained on corpus with high-quality labeled sentence pairs, which limits its application to tasks where labeled data is extremely scarce.In this paper, we propose a lightweight extension on top of BERT and a novel self-supervised learning objective based on mutual information maximization strategies to derive meaningful sentence embeddings in an unsupervised manner.Unlike SBERT, our method is not restricted by the availability of labeled data, such that it can be applied on different domain-specific corpus.Experimental results show that the proposed method significantly outperforms other unsupervised sentence embedding baselines on common semantic textual similarity (STS) tasks and downstream supervised tasks.It also outperforms SBERT in a setting where in-domain labeled data is not available, and achieves performance competitive with supervised methods on various tasks.

Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements over various cross-lingual and low-resource tasks. Through training on one hundred languages and terabytes of texts, cross-lingual language models have proven to be effective in leveraging high-resource languages to enhance low-resource language processing and outperform monolingual models. In this paper, we further investigate the cross-lingual and cross-domain (CLCD) setting when a pretrained cross-lingual language model needs to adapt to new domains. Specifically, we propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features and domain-invariant features from the entangled pretrained cross-lingual representations, given unlabeled raw texts in the source language. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts. Experimental results show that our proposed method achieves significant performance improvements over the state-of-the-art pretrained cross-lingual language model in the CLCD setting.

Aspect-based sentiment analysis produces a list of aspect terms and their corresponding sentiments for a natural language sentence.This task is usually done in a pipeline manner, with aspect term extraction performed first, followed by sentiment predictions toward the extracted aspect terms.While easier to develop, such an approach does not fully exploit joint information from the two subtasks and does not use all available sources of training information that might be helpful, such as document-level labeled sentiment corpus.In this paper, we propose an interactive multi-task learning network (IMN) which is able to jointly learn multiple related tasks simultaneously at both the token level as well as the document level.Unlike conventional multi-task learning methods that rely on learning common features for the different tasks, IMN introduces a message passing architecture where information is iteratively passed to different tasks through a shared set of latent variables.Experimental results demonstrate superior performance of the proposed method against multiple baselines on three benchmark datasets.

Aspect-level sentiment classification aims to determine the sentiment polarity of a review sentence towards an opinion target. A sentence could contain multiple sentiment-target pairs; thus the main challenge of this task is to separate different opinion contexts for different targets. To this end, attention mechanism has played an important role in previous state-of-the-art neural models. The mechanism is able to capture the importance of each context word towards a target by modeling their semantic associations. We build upon this line of research and propose two novel approaches for improving the effectiveness of attention. First, we propose a method for target representation that better captures the semantic meaning of the opinion target. Second, we introduce an attention model that incorporates syntactic information into the attention mechanism. We experiment on attention-based LSTM (Long Short-Term Memory) models using the datasets from SemEval 2014, 2015, and 2016. The experimental results show that the conventional attention-based LSTM can be substantially improved by incorporating the two approaches.

We consider the cross-domain sentiment classification problem, where a sentiment classifier is to be learned from a source domain and to be generalized to a target domain.Our approach explicitly minimizes the distance between the source and the target instances in an embedded feature space.With the difference between source and target minimized, we then exploit additional information from the target domain by consolidating the idea of semi-supervised learning, for which, we jointly employ two regularizations -entropy minimization and self-ensemble bootstrapping -to incorporate the unlabeled target data for classifier refinement.Our experimental results demonstrate that the proposed approach can better leverage unlabeled data from the target domain and achieve substantial improvements over baseline methods in various experimental settings.

Aspect extraction is an important and challenging task in aspect-based sentiment analysis.Existing works tend to apply variants of topic models on this task.While fairly successful, these methods usually do not produce highly coherent aspects.In this paper, we present a novel neural approach with the aim of discovering coherent aspects.The model improves coherence by exploiting the distribution of word co-occurrences through the use of neural word embeddings.Unlike topic models which typically assume independently generated words, word embedding models encourage words that appear in similar contexts to be located close to each other in the embedding space.In addition, we use an attention mechanism to de-emphasize irrelevant words during training, further improving the coherence of aspects.Experimental results on real-life datasets demonstrate that our approach discovers more meaningful and coherent aspects, and substantially outperforms baseline methods on several evaluation tasks.