Current state-of-the-art sequence labeling models are typically based on sequential architecture such as Bi-directional LSTM (BiLSTM). However, the structure of processing a word at a time based on the sequential order restricts the full utilization of non-sequential features, including syntactic relationships, word co-occurrence relations, and document topics. They can be regarded as the corpus-level features and critical for sequence labeling. In this paper, we propose a Corpus-Aware Graph Aggregation Network. Specifically, we build three types of graphs, i.e., a word-topic graph, a word co-occurrence graph, and a word syntactic dependency graph, to express different kinds of corpus-level non-sequential features. After that, a graph convolutional network (GCN) is adapted to model the relations between words and non-sequential features. Finally, we employ a label-aware attention mechanism to aggregate corpus-aware non-sequential features and sequential ones for sequence labeling. The experimental results on four sequence labeling tasks (named entity recognition, chunking, multilingual sequence labeling, and target-based sentiment analysis) show that our model achieves state-of-the-art performance.

Text classification is an important task in natural language processing. Contextual information is essential for text classification, and different words usually need different sizes of contextual information. However, most existing methods learn contextual features with predefined fixed sizes, which cannot extract the different sizes of contextual features for different words. To this end, we propose a new model named Deformable Self-Attention (DSA) to flexibly learn word-specific contextual features, rather than extracting features of fixed context sizes. Our model is mainly composed of a Deformable Local Attention Weight Generation (DLAWG) module and a Multi-Range Feature Integration (MRFI) module. The DLAWG module can adaptively determine different context sizes for different words within a particular range and then learn word-specific contextual features for each word. DLAWG then employs multiple ranges to capture context dependencies of different ranges. After that, the MRFI module integrates features from different ranges by considering the interactions with features of different ranges, which can delete irrelevant features while enhancing discriminative ones. Experiments on extensive benchmark datasets and visualizations illustrate the effectiveness of our model.

The central problem of sentence classification is to extract multi-scale n-gram features for understanding the semantic meaning of sentences.Most existing models tackle this problem by stacking CNN and RNN models, which easily leads to feature redundancy and overfitting because of relatively limited datasets.In this paper, we propose a simple yet effective model called Multi-scale Orthogonal inDependEnt LSTM (MODE-LSTM), which not only has effective parameters and good generalization ability, but also considers multiscale n-gram features.We disentangle the hidden state of the LSTM into several independently updated small hidden states and apply an orthogonal constraint on their recurrent matrices.We then equip this structure with sliding windows of different sizes for extracting multi-scale n-gram features.Extensive experiments demonstrate that our model achieves better or competitive performance against state-of-the-art baselines on eight benchmark datasets.We also combine our model with BERT to further boost the generalization performance.

Text classification is a central field of inquiry in natural language processing (NLP). Although some models learn local semantic features and global long-term dependencies simultaneously, they simply combine them through concatenation either in a cascade way or in parallel while mutual effects between them are ignored. In this paper, we propose the Global-Local Mutual Attention (GLMA) model for text classification problems, which introduces a mutual attention mechanism for mutual learning between local semantic features and global long-term dependencies. The mutual attention mechanism consists of a Local-Guided Global-Attention (LGGA) and a Global-Guided Local-Attention (GGLA). The LGGA allows to assign weights and combine global long-term dependencies of word positions that are semantic related. It captures combined semantics and alleviates the gradient vanishing problem. The GGLA automatically assigns more weights to relevant local semantic features, which captures key local semantic information and filters both noises and irrelevant words/phrases. Furthermore, a weighted-over-time pooling operation is developed to aggregate the most informative and discriminative features for classification. Extensive experiments demonstrate that our model obtains the state-of-the-art performance on seven benchmark datasets and sixteen Amazon product reviews datasets. Both the result analysis and the mutual attention weights visualization further demonstrate the effectiveness of the proposed model.