Haibin Chen, Qianli Ma, Zhenxi Lin, Jiangyue Yan. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.

Metaphors are ubiquitous in natural language, and detecting them requires contextual reasoning about whether a semantic incongruence actually exists.Most existing work addresses this problem using pre-trained contextualized models.Despite their success, these models require a large amount of labeled data and are not linguistically-based.In this paper, we proposed a ContrAstive pre-Trained modEl (CATE) for metaphor detection with semi-supervised learning.Our model first uses a pre-trained model to obtain a contextual representation of target words and employs a contrastive objective to promote an increased distance between target words' literal and metaphorical senses based on linguistic theories.Furthermore, we propose a simple strategy to collect large-scale candidate instances from the general corpus and generalize the model via self-training.Extensive experiments show that CATE achieves better performance against state-of-the-art baselines on several benchmark datasets.

Current state-of-the-art sequence labeling models are typically based on sequential architecture such as Bi-directional LSTM (BiLSTM). However, the structure of processing a word at a time based on the sequential order restricts the full utilization of non-sequential features, including syntactic relationships, word co-occurrence relations, and document topics. They can be regarded as the corpus-level features and critical for sequence labeling. In this paper, we propose a Corpus-Aware Graph Aggregation Network. Specifically, we build three types of graphs, i.e., a word-topic graph, a word co-occurrence graph, and a word syntactic dependency graph, to express different kinds of corpus-level non-sequential features. After that, a graph convolutional network (GCN) is adapted to model the relations between words and non-sequential features. Finally, we employ a label-aware attention mechanism to aggregate corpus-aware non-sequential features and sequential ones for sequence labeling. The experimental results on four sequence labeling tasks (named entity recognition, chunking, multilingual sequence labeling, and target-based sentiment analysis) show that our model achieves state-of-the-art performance.

Text classification is an important task in natural language processing. Contextual information is essential for text classification, and different words usually need different sizes of contextual information. However, most existing methods learn contextual features with predefined fixed sizes, which cannot extract the different sizes of contextual features for different words. To this end, we propose a new model named Deformable Self-Attention (DSA) to flexibly learn word-specific contextual features, rather than extracting features of fixed context sizes. Our model is mainly composed of a Deformable Local Attention Weight Generation (DLAWG) module and a Multi-Range Feature Integration (MRFI) module. The DLAWG module can adaptively determine different context sizes for different words within a particular range and then learn word-specific contextual features for each word. DLAWG then employs multiple ranges to capture context dependencies of different ranges. After that, the MRFI module integrates features from different ranges by considering the interactions with features of different ranges, which can delete irrelevant features while enhancing discriminative ones. Experiments on extensive benchmark datasets and visualizations illustrate the effectiveness of our model.

As efficient recurrent neural network (RNN) models, echo state networks (ESNs) have attracted widespread attention and been applied in many application domains in the last decade. Although they have achieved great success in modeling time series, a single ESN may have difficulty in capturing the multitimescale structures that naturally exist in temporal data. In this paper, we propose the convolutional multitimescale ESN (ConvMESN), which is a novel training-efficient model for capturing multitimescale structures and multiscale temporal dependencies of temporal data. In particular, a multitimescale memory encoder is constructed with a multireservoir structure, in which different reservoirs have recurrent connections with different skip lengths (or time spans). By collecting all past echo states in each reservoir, this multireservoir structure encodes the history of a time series as nonlinear multitimescale echo state representations (MESRs). Our visualization analysis verifies that the MESRs provide better discriminative features for time series. Finally, multiscale temporal dependencies of MESRs are learned by a convolutional layer. By leveraging the multitimescale reservoirs followed by a convolutional learner, the ConvMESN has not only efficient memory encoding ability for temporal data with multitimescale structures but also strong learning ability for complex temporal dependencies. Furthermore, the training-free reservoirs and the single convolutional layer provide high-computational efficiency for the ConvMESN to model complex temporal data. Extensive experiments on 18 multivariate time series (MTS) benchmark datasets and 3 skeleton-based action recognition datasets demonstrate that the ConvMESN captures multitimescale dynamics and outperforms existing methods.

The central problem of sentence classification is to extract multi-scale n-gram features for understanding the semantic meaning of sentences.Most existing models tackle this problem by stacking CNN and RNN models, which easily leads to feature redundancy and overfitting because of relatively limited datasets.In this paper, we propose a simple yet effective model called Multi-scale Orthogonal inDependEnt LSTM (MODE-LSTM), which not only has effective parameters and good generalization ability, but also considers multiscale n-gram features.We disentangle the hidden state of the LSTM into several independently updated small hidden states and apply an orthogonal constraint on their recurrent matrices.We then equip this structure with sliding windows of different sizes for extracting multi-scale n-gram features.Extensive experiments demonstrate that our model achieves better or competitive performance against state-of-the-art baselines on eight benchmark datasets.We also combine our model with BERT to further boost the generalization performance.