Aspect-level sentiment classification (ASC) is a significant problem in fine-grained sentiment analysis, which automatically predicts the sentiment polarity of a given aspect in a sentence. Dependency tree-based graph convolutional networks have been widely studied for their ability to effectively capture the dependencies of aspect words with other words. However, constructing more accurate syntactic trees by introducing external knowledge has limited improvement on ungrammatical informal texts and has led to over-parameterization of the model. To alleviate this problem, we propose a sentiment and syntactic-aware graph convolutional network (SaS-GCN) that combines syntactic and sentiment relations. We use an attention mechanism and the Sparsemax activation function to construct a sparse sentiment-dependent graph. Compared with existing methods that use LSTM or CNN to obtain semantics from text directly, this graph, combined with a GCN, contains more semantic features. Moreover, we redesign the network structure of GCN, calling it EN-GCN, to make it sensitive to node dimensional features and hence to have a strong feature mining ability. The experimental results indicate that our model outperforms state-of-the-art methods. In particular, when evaluated on the Rest15 and Rest16 datasets, the F1 scores of the proposed lightweight model are 4.15% and 3.77% better than BERT respectively.

Students who have been taught new ideas need to develop their skills by carrying out further work in their own time. This often consists of a series of exercises which must be completed. While students can choose exercises themselves from online sources, they will learn more quickly and easily if the exercises are specifically tailored to their needs. A good teacher will always aim to do this, but with the large groups of students who typically take advantage of open online courses, it may not be possible. Exercise prediction, working with large-scale matrix data, is a better way to address this challenge, and a key stage within such prediction is to calculate the probability that a student will answer a given question correctly. Therefore, this paper presents a novel approach called Weighting-based Student Exercise Matrix Factorization (Wse-MF) which combines student learning ability and exercise difficulty as prior weights. In order to learn how to complete the matrix, we apply an iterative optimization method that makes the approach practical for large-scale educational deployment. Compared with eight models in cognitive diagnosis and matrix factorization, our research results suggest that Wse-MF significantly outperforms the state-of-the-art on a range of real-world datasets in both prediction quality and time complexity. Moreover, we find that there is an optimal value of the latent factor K (the inner dimension of the factorization) for each dataset, which is related to the relationship between skills and exercises in that dataset. Similarly, the optimal value of hyperparameter c0 is linked to the ratio between exercises and students. Taken as a whole, we demonstrate improvements to matrix factorization within the context of educational data.

The field of Drug-drug interaction (DDI) aims to detect descriptions of interactions between drugs from biomedical texts. Currently, researchers have extracted DDIs using pre-trained language models such as BERT, which often misclassify two kinds of DDI types, "Effect" and "Int", on the DDIExtraction 2013 corpus because of highly similar expressions. The use of knowledge graphs can alleviate this problem by incorporating different relationships for each, thus allowing them to be distinguished. Thus, we propose a novel framework to integrate the neural network with a knowledge graph, where the features from these components are complementary. Specifically, we take text features at different levels into account in the neural network part. This is done by firstly obtaining a word-level position feature using PubMedBERT together with a convolution neural network, secondly, getting a phrase-level key path feature using a dependency parsing tree, thirdly, using PubMedBERT with an attention mechanism to obtain a sentence-level language feature, and finally, fusing these three kinds of representation into a synthesized feature. We also extract a knowledge feature from a drug knowledge graph which takes just a few minutes to construct, then concatenate the synthesized feature with the knowledge feature, feed the result into a multi-layer perceptron and obtain the result by a softmax classifier. In order to achieve a good integration of the synthesized feature and the knowledge feature, we train the model using a novel multifocal loss function, KGE-MFL, which is based on a knowledge graph embedding. Finally we attain state-of-the-art results on the DDIExtraction 2013 dataset (micro F-score 86.24%) and on the ChemProt dataset (micro F-score 77.75%), which proves our framework to be effective for biomedical relation extraction tasks. In particular, we fill the performance gap (more than 5.57%) between methods that rely on and do not rely on knowledge graph embedding on the DDIExtraction 2013 corpus, when predicting the "Int" type. The implementation code is available at https://github.com/NWU-IPMI/DDIE-KGE-MFL.

Large-scale pretrained language models have led to significant improvements in Natural Language Processing. Unfortunately, they come at the cost of high computational and storage requirements that complicate their deployment on low-resource devices. This issue can be addressed by distilling knowledge from larger models to smaller ones through pseudo-labels on task-specific datasets. However, this can be difficult for tasks with very limited data. To overcome this challenge, we present a novel approach where knowledge can be distilled from a teacher model to a student model through the generation of synthetic data. For this to be done, we first fine-tune the teacher and student models, as well as a Natural Language Generation (NLG) model, on the target task dataset. We then let both student and teacher work together to condition the NLG model to generate examples that can enhance the performance of the student. We tested our approach on two data generation methods: a) Targeted generation using the Monte Carlo Tree Search (MCTS) algorithm, and b) A Non-Targeted Text Generation (NTTG) method. We evaluate the effectiveness of our approaches against a baseline that uses the BERT model for data augmentation through random word replacement. By testing this approach on the SST-2, MRPC, YELP-2, DBpedia, and TREC-6 datasets, we consistently witnessed considerable improvements over the word-replacement baseline.

Abstract In recent years, Deep Neural Networks (DNNs) have achieved excellent performance on many tasks, but it is very difficult to train good models from imbalanced datasets. Creating balanced batches either by majority data down-sampling or by minority data up-sampling can solve the problem in certain cases. However, it may lead to learning process instability and overfitting. In this paper, we propose the Batch Balance Wrapper (BBW), a novel framework which can adapt a general DNN to be well trained from extremely imbalanced datasets with few minority samples. In BBW, two extra network layers are added to the start of a DNN. The layers prevent overfitting of minority samples and improve the expressiveness of the sample distribution of minority samples. Furthermore, Batch Balance (BB), a class-based sampling algorithm, is proposed to make sure the samples in each batch are always balanced during the learning process. We test BBW on three well-known extremely imbalanced datasets with few minority samples. The maximum imbalance ratio reaches 1167:1 with only 16 positive samples. Compared with existing approaches, BBW achieves better classification performance. In addition, BBW-wrapped DNNs are 16.39 times faster, relative to unwrapped DNNs. Moreover, BBW does not require data preprocessing or additional hyper-parameter tuning, operations that may require additional processing time. The experiments prove that BBW can be applied to common applications of extremely imbalanced data with few minority samples, such as the classification of EEG signals, medical images and so on.

Abstract Sentence matching is widely used in various natural language tasks, such as natural language inference, paraphrase identification and question answering. For these tasks, we need to understand the logical and semantic relationship between two sentences. Most current methods use all information within a sentence to build a model and hence determine its relationship to another sentence. However, the information contained in some sentences may cause redundancy or introduce noise, impeding the performance of the model. Therefore, we propose a sentence matching method based on multi keyword-pair matching (MKPM), which uses keyword pairs in two sentences to represent the semantic relationship between them, avoiding the interference of redundancy and noise. Specifically, we first propose a sentence-pair-based attention mechanismsp-attentionto select the most important word pair from the two sentences as a keyword pair, and then propose a Bi-task architecture to model the semantic information of these keyword pairs. The Bi-task architecture is as follows: 1. In order to understand the semantic relationship at the word level between two sentences, we design a word-pair task (WP-Task), which uses these keyword pairs to complete sentence matching independently. 2. We design a sentence-pair task (SP-Task) to understand the sentence level semantic relationship between the two sentences by sentence denoising. Through the integration of the two tasks, our model can understand sentences more accurately from the two granularities of word and sentence. Experimental results show that our model can achieve state-of-the-art performance in several tasks. Our source code is publicly available1.

The extraction of drug-drug interactions (DDIs) is an important task in the field of biomedical research, which can reduce unexpected health risks during patient treatment. Previous work indicates that methods using external drug information have a much higher performance than those methods not using it. However, the use of external drug information is time-consuming and resource-costly. In this work, we propose a novel method for extracting DDIs which does not use external drug information, but still achieves comparable performance. First, we no longer convert the drug name to standard tokens such as DRUG0, the method commonly used in previous research. Instead, full drug names with drug entity marking are input to BioBERT, allowing us to enhance the selected drug entity pair. Second, we adopt the Key Semantic Sentence approach to emphasize the words closely related to the DDI relation of the selected drug pair. After the above steps, the misclassification of similar instances which are created from the same sentence but corresponding to different pairs of drug entities can be significantly reduced. Then, we employ the Gradient Harmonizing Mechanism (GHM) loss to reduce the weight of mislabeled instances and easy-to-classify instances, both of which can lead to poor performance in DDI extraction. Overall, we demonstrate in this work that it is better not to use drug blinding with BioBERT, and show that GHM performs better than Cross-Entropy loss if the proportion of label noise is less than 30%. The proposed model achieves state-of-the-art results with an F1-score of 84.13% on the DDIExtraction 2013 corpus (a standard English DDI corpus), which fills the performance gap (4%) between methods that rely on and do not rely on external drug information.

Knowledge tracing is an important research topic in student modeling. The aim is to model a student's knowledge state by mining a large number of exercise records. The dynamic key-value memory network (DKVMN) proposed for processing knowledge tracing tasks is considered to be superior to other methods. However, through our research, we have noticed that the DKVMN model ignores both the students' behavior features collected by the intelligent tutoring system (ITS) and their learning abilities, which, together, can be used to help model a student's knowledge state. We believe that a student's learning ability always changes over time. Therefore, this article proposes a new exercise record representation method, which integrates the features of students' behavior with those of the learning ability, thereby improving the performance of knowledge tracing. Our experiments show that the proposed method can improve the prediction results of DKVMN.

Convolutional Neural Networks (CNNs) have achieved remarkable performance breakthroughs in a variety of tasks. Recently, CNN-based methods that are fed with hand-extracted EEG features have steadily improved their performance on the emotion recognition task. In this paper, we propose a novel convolutional layer, called the Scaling Layer, which can adaptively extract effective data-driven spectrogram-like features from raw EEG signals. Furthermore, it exploits convolutional kernels scaled from one data-driven pattern to exposed a frequency-like dimension to address the shortcomings of prior methods requiring hand-extracted features or their approximations. ScalingNet, the proposed neural network architecture based on the Scaling Layer, has achieved state-of-the-art results across the established DEAP and AMIGOS benchmark datasets.

Sentiment classification is a fundamental task in many natural language processing applications. Neural networks have achieved great successes on the sentiment classification task in recent years, since recurrent neural networks and long-short-term memory networks have the ability to deal with sequences of different lengths and to capture contextual semantic information. However, the effectiveness of these methods is limited when used to extract contextual information from relatively long texts. Therefore, in our model, we apply bidirectional gated recurrent units to capture contextual information as far as possible when learning word representations, which may effectively reduce the noise compared to other methods. We also propose a novel loss function namely drop loss (DL) which makes the model focus on the hard examples - examples which are easily classified incorrectly - in order to improve the accuracy of the model. We experiment on four commonly used datasets, and the results show that the proposed method has a good performance on four datasets, and needs fewer parameters compared with recent benchmarks, such as CoVe, ULMFiT, embeddings from language models, and bidirectional encoder representations from transformers. Furthermore, we demonstrate that the classification performance of existing shallow network models can be significantly improved by using DL. In particular, the accuracy of the CNN+LSTM model improves 9% on the IMDB-10 dataset.

In this paper we propose a novel data augmentation approach where guided outputs of a language generation model, e.g.GPT-2, when labeled, can improve the performance of text classifiers through an active learning process.We transform the data generation task into an optimization problem which maximizes the usefulness of the generated output, using Monte Carlo Tree Search (MCTS) as the optimization strategy and incorporating entropy as one of the optimization criteria.We test our approach against a Non-Guided Data Generation (NGDG) process that does not optimize for a reward function.Starting with a small set of data, our results show an increased performance with MCTS of 26% on the TREC-6 Questions dataset, and 10% on the Stanford Sentiment Treebank SST-2 dataset.Compared with NGDG, we are able to achieve increases of 3% and 5% on TREC-6 and SST-2.

In a previous question answering study, we identified nine semantic-relationship types, including synonyms, hypernyms, word chains, and holonyms, that exist between terms in Text Retrieval Conference queries and those in their supporting sentences in the Advanced Question Answering for Intelligence (Graff, 2002) corpus. The most frequently occurring relationship type was the hypernym (e.g., Katherine Hepburn is an actress). The aim of the present work, therefore, was to develop a method for determining a person's occupation from syntactic data in a text corpus. First, in the P-System, we compared predicate–argument data involving a proper name for different occupations using Okapi's BM25 weighting algorithm. When classifying actors and using sufficiently frequent names, an accuracy of 0.955 was attained. For evaluation purposes, we also implemented a standard apposition-based classifier (A-System). This performs well, but only if a particular name happens to appear in apposition with the corresponding occupation. Last, we created a hybrid (H-System) which combines the strengths of P with those of A. Using data with a minimum of 100 predicate–argument pairs, H performed best with an overall lenient accuracy of 0.750 while A and P scored 0.615 and 0.656, respectively. We therefore conclude that a hybrid approach combining information from different sources is the best way to predict occupations.

Abstract The Decision Tree Forest (DTF) is an architecture for information retrieval that uses a separate decision tree for each document in a collection. Experiments were conducted in which DTFs working with the incremental tree induction (ITI) algorithm of Utgoff, Berkman, and Clouse (1997) were trained and evaluated in the medical and word processing domains using the Cystic Fibrosis and SIFT collections. Performance was compared with that of a conventional inverted index system (IIS) using a BM25‐derived probabilistic matching function. Initial results using DTF were poor compared to those obtained with IIS. We then simulated scenarios in which large quantities of training data were available, by using only those parts of the document collection that were well covered by the data sets. Consequently the retrieval effectiveness of DTF improved substantially. In one particular experiment precision and recall for DTF were 0.65 and 0.67 respectively, values that compared favorably with values of 0.49 and 0.56 for IIS.