Neural networks are among the state-ofthe-art techniques for language modeling.Existing neural language models typically map discrete words to distributed, dense vector representations.After information processing of the preceding context words by hidden layers, an output layer estimates the probability of the next word.Such approaches are time-and memory-intensive because of the large numbers of parameters for word embeddings and the output layer.In this paper, we propose to compress neural language models by sparse word representations.In the experiments, the number of parameters in our model increases very slowly with the growth of the vocabulary size, which is almost imperceptible.Moreover, our approach not only reduces the parameter space to a large extent, but also improves the performance in terms of the perplexity measure. 1

Distilling knowledge from a well-trained cumbersome network to a small one has recently become a new research topic, as lightweight neural networks with high performance are particularly in need in various resource-restricted systems. This paper addresses the problem of distilling word embeddings for NLP tasks. We propose an encoding approach to distill task-specific knowledge from a set of high-dimensional embeddings, so that we can reduce model complexity by a large margin as well as retain high accuracy, achieving a good compromise between efficiency and performance. Experiments reveal the phenomenon that distilling knowledge from cumbersome embeddings is better than directly training neural networks with small embeddings.

Nowadays, neural networks play an important role in the task of relation classification. By designing different neural architectures, researchers have improved the performance to a large extent in comparison with traditional methods. However, existing neural networks for relation classification are usually of shallow architectures (e.g., one-layer convolutional neural networks or recurrent networks). They may fail to explore the potential representation space in different abstraction levels. In this paper, we propose deep recurrent neural networks (DRNNs) for relation classification to tackle this challenge. Further, we propose a data augmentation method by leveraging the directionality of relations. We evaluated our DRNNs on the SemEval-2010 Task 8, and achieve an F1-score of 86.1%, outperforming previous state-of-the-art recorded results.

Transfer learning is aimed to make use of valuable knowledge in a source domain to help model performance in a target domain.It is particularly important to neural networks, which are very likely to be overfitting.In some fields like image processing, many studies have shown the effectiveness of neural network-based transfer learning.For neural NLP, however, existing studies have only casually applied transfer learning, and conclusions are inconsistent.In this paper, we conduct systematic case studies and provide an illuminating picture on the transferability of neural networks in NLP. 1

This paper proposes a tree-based convolutional neural network (TBCNN) for discriminative sentence modeling.Our model leverages either constituency trees or dependency trees of sentences.The tree-based convolution process extracts sentences structural features, which are then aggregated by max pooling.Such architecture allows short propagation paths between the output layer and underlying feature detectors, enabling effective structural feature learning and extraction.We evaluate our models on two tasks: sentiment analysis and question classification.In both experiments, TBCNN outperforms previous state-of-the-art results, including existing neural networks and dedicated feature/rule engineering.We also make efforts to visualize the tree-based convolution process, shedding light on how our models work.

Relation classification is an important research arena in the field of natural language processing (NLP).In this paper, we present SDP-LSTM, a novel neural network to classify the relation of two entities in a sentence.Our neural architecture leverages the shortest dependency path (SDP) between two entities; multichannel recurrent neural networks, with long short term memory (LSTM) units, pick up heterogeneous information along the SDP.Our proposed model has several distinct features: (1) The shortest dependency paths retain most relevant information (to relation classification), while eliminating irrelevant words in the sentence.(2) The multichannel LSTM networks allow effective information integration from heterogeneous sources over the dependency paths.(3) A customized dropout strategy regularizes the neural network to alleviate overfitting.We test our model on the SemEval 2010 relation classification task, and achieve an F 1 -score of 83.7%, higher than competing methods in the literature.

Learning non-taxonomic relations becomes an important research topic in ontology extension. Most of the existing learning approaches are mainly based on expert crafted corpora. These approaches are normally domain-specific and the corpora acquisition is laborious and costly. On the other hand, based on the static corpora, it is not able to meet personalized needs of semantic relations discovery for various taxonomies. In this paper, we propose a novel approach for learning non-taxonomic relations on demand. For any supplied taxonomy, it can focus on the segment of the taxonomy and collect information dynamically about the taxonomic concepts by using Wikipedia as a learning source. Based on the newly generated corpus, non-taxonomic relations are acquired through three steps: a) semantic relatedness detection; b) relations extraction between concepts; and c) relations generalization within a hierarchy. The proposed approach is evaluated on three different predefined taxonomies and the experimental results show that it is effective in capturing non-taxonomic relations as needed and has good potential for the ontology extension on demand.