One key task of fine-grained sentiment analysis on reviews is to extract aspects or features that users have expressed opinions on. Generally, fine-tuning BERT with sophisticated task-specific layers can achieve better performance than only extend one extra task-specific layer (e.g., a fully-connected + softmax layer) since not all tasks can easily be represented by Transformer encoder architecture and special task-specific layer can capture task-specific features. However, BERT fine-tuning may be unstable on a small-scale dataset. Besides, in our experiments, directly fine-tuning BERT on extending sophisticated task-specific layers did not take advantage of the features of task-specific layers and even restrict the performance of BERT module. To address the above consideration, this paper combines Fine-tuning with a feature-based approach to extract aspect. To the best of our knowledge, this is the first paper to combine fine-tuning with a feature-based approach for aspect extraction.

With the prevalent of short texts, discovering the topics within them has become an important task. Biterm Topic Model (BTM) is more suitable to discover topics on short texts than traditional topic models. However, there are still some challenges that dealing short texts with BTM will always ignor e the document-topic semantic information and lack the true intentions of users. In addition, it is a static method and can not manage streaming short texts when a new one arrives immediately. In order to keep document-topic information and get the topic distribution of a new short text at once, we propose a joint model based on online algorithms of Latent Dirichlet Allocation (LDA) and BTM, which combines the merits of both models. Not only does it alleviate the sparsity when addressing short texts with the online algorithm of BTM, namely Incremental Biterm Topic Model (IBTM), but also keeps document-topic information with extended LDA. And considering the differences between English and Chinese text in writing, we use combined words in short texts as key words to extend the length of short texts and keep the true intensions of users. As shown in the experiment results on two real world datasets, our method is better than other baseline methods. In the end, we explain an application of our method in the task of discovering user interest tags.

Emotion classification is an important task for identifying users' emotional expressions in text. Though a variety of neural models have been proposed nowadays, these models mainly focus on modeling the content of words or characters without fully employing the emotional features in lexical features, especially the features of part-of -speech (POS). In this paper, we reveal that the information of POS as well as that of words is important for identifying the type of emotion in a given text. We propose two simple models to fully learn the emotional features of the POS of words. Every model consists of the long short-term memory (LSTM) network as input encoders and the component of attention mechanism. One model is to concatenate the POS tags of vectors into the hidden states of representations generated by LSTM as raw feature representations and put them into the component of attention mechanism to generate the text representation toward a special emotion. The other is to use both LSTM and attention mechanism to model the context representation of words and those of POS tags respectively and concatenate these context representations as the text representation toward a special emotion. We conduct some experiments on datasets for evaluation and demonstrate the effectiveness of our model, where the datasets consist of the open-source dataset from NLPCC& 2014 and the dataset of manual annotation. Experimental results show that our models can achieve outstanding performance for emotion classification in Chinese Weibo texts and outperform classical baselines.

Short texts are prevalent on today’s Web, especially with the emergence of social media. However, how to discover the topics of streaming short texts has become an important task for many content analysis applications. Conventional topic models such as Probabilistic Latent Semantic Analysis (PLSA) and Latent Dirichlet Allocation (LDA) will suffer from sparsity problem when we infer the latent topics from short texts with them. The reason is that they derive topics from document-level word co-occurrence by modeling each document as a mixture of topics. Different from the above idea, Biterm Topic Model (BTM) discovers topics in short texts by directly modeling the generation of word co-occurrence patterns in the whole corpus. But semantic information is lacking for short texts. In this paper, in order to alleviate the sparsity problem, keep the semantic information of documents and get the latent topic information of streaming short texts immediately, we propose a joint topic model for Chinese streaming short texts (CTM) based on the online algorithms of LDA and BTM. Experiments on short texts from Sina Weibo show that our joint topic model can discover more precise topics and carry out more applications. In addition, considering the preprocessing in Chinese text is different from English and errors in extracting key phrases, we use a combined word method to extend the length of short texts and reduce errors in extracting key phrases.

With the popularity of social networks, such as mi-croblogs and Twitter, topic inference for short text is increasingly significant and essential for many content analysis tasks. Biterm topic model (BTM) is superior to conventional topic models in uncovering latent semantic relevance for short text. However, Gibbs sampling employed by BTM is very time consuming when inferring topics, especially for large-scale datasets. It requires O{K) operations per sample for K topics, where K denotes the number of topics in the corpus. In this paper, we propose an acceleration algorithm of BTM, FastBTM, using an efficient sampling method for BTM which only requires O(1) amortized time while the traditional ones scale linearly with the number of topics. FastBTM is based on Metropolis-Hastings and alias method, both of which have been widely adopted in latent Dirichlet allocation (LDA) model and achieved outstanding speedup. We carry out a number of experiments on Tweets2011 Collection dataset and Enron dataset, indicating that our method is robust enough for both short texts and normal documents. Our work can be approximately 9 times faster than traditional Gibbs sampling method per iteration, when setting K = 1000. The source code of FastBTM can be obtained from https://github.com/paperstudy/FastBTM.

Answer selection is a very important task in domain question answering. However, because of the word variety between questions and answers, there exists the lexical gap between questions and answers, which is the major challenge in question answer matching. In this work, in order to overcome the lexical gap, we propose an attention based bidirectional gated convolution with neural tensor network (ABiRCNN+NTN), which can improve the representations for both questions and answers and model their interactions with a neural tensor network. We carry out large-scale experiments on answer selection dataset, InsuranceQA and achieve new state-of-the-art results on InsuranceQA dataset. The experimental results demonstrate that our model can effectively capture the complex semantic relations between questions and answers and encode them in a more effective way. The source code of our work can be obtained from https://github.com/paperstudy/AnswerSelection.

Recently, rectified linear units (ReLUs) have been used to solve the vanishing gradient problem. Their use has led to state-of-the-art results in various problems such as image classification. In this paper, we propose the hyperbolic linear units (HLUs) which not only speed up learning process in deep convolutional neural networks but also obtain better performance in image classification tasks. Unlike ReLUs, HLUs have inheriently negative values which could make mean unit outputs closer to zero. Mean unit outputs close to zero means we can speed up the learning process because they bring the normal gradient close to the natural gradient. Indeed, the difference called bias shift between natural gradient and the normal gradient is related to the mean activation of input units. Experiments with three popular CNN architectures, LeNet, Inception network and ResNet on various benchmarks including MNIST, CIFAR-10 and CIFAR-100 demonstrate that our proposed HLUs achieve significant improvement compared to other commonly used activation functions1.

The hash symbol, called a hashtag, is used to mark the keyword or topic in a tweet. It was created organically by users as a way to categorize messages. Hashtags also provide valuable information for many research applications such as sentiment classification and topic analysis. However, only a small number of tweets are manually annotated. Therefore, an automatic hashtag recommendation method is needed to help users tag their new tweets. Previous methods mostly use conventional machine learning classifiers such as SVM or utilize collaborative filtering technique. A bottleneck of these approaches is that they all use the TF-IDF scheme to represent tweets and ignore the semantic information in tweets. In this paper, we also regard hashtag recommendation as a classification task but propose a novel recurrent neural network model to learn vector-based tweet representations to recommend hashtags. More precisely, we use a skip-gram model to generate distributed word representations and then apply a convolutional neural network to learn semantic sentence vectors. Afterwards, we make use of the sentence vectors to train a long short-term memory recurrent neural network (LSTM-RNN). We directly use the produced tweet vectors as features to classify hashtags without any feature engineering. Experiments on real world data from Twitter to recommend hashtags show that our proposed LSTM-RNN model outperforms state-of-the-art methods and LSTM unit also obtains the best performance compared to standard RNN and gated recurrent unit (GRU).

Microblog post has been a hot research source for emotion classification in recent years. However, due to bloggers' free narrative style and topics' timeliness, the data from microblog post is usually implicit and imbalanced. In this paper, the problems of emotion classification in Chinese microblog posts are solved in a hierarchical way using a knowledge-based topic model and Support Vector Machine(SVM). Based on topic model, an implicit feature detection algorithm is proposed to identify the latent emotions underlying the microblog posts. Meanwhile, a hierarchical emotion structure is employed to classify emotions into 19 classes of four levels by SVM. This structure can meet different research requirements at three granularities. The experiment results validate that our model can achieve better performance in terms of precision, recall and F-scores.

This paper deals with the adaptive variance scaling issue incontinuous Estimation of Distribution Algorithms. A phenomenon is discovered that current adaptive variance scaling method in EDA suffers from imprecise structure learning. A new type of adaptation method is proposed to overcome this defect. The method tries to measure the difference between the obtained population and the prediction of the probabilistic model, then calculate the scaling factor by minimizing the cross entropy between these two distributions. This approach calculates the scaling factor immediately rather than adapts it incrementally. Experiments show that this approach extended the class of problems that can be solved, and improve the search efficiency in some cases. Moreover, the proposed approach features in that each decomposed subspace can be assigned an individual scaling factor, which helps to solve problems with special dimension property.

This paper extends the Boltzmann Selection, a method in EDA with theoretical importance, from discrete domain to the continuous one. The difficulty of estimating the exact Boltzmann distribution in continuous state space is circumvented by adopting the multivariate Gaussian model, which is popular in continuous EDA, to approximate only the final sampling distribution. With the minimum Kullback-Leibeler divergence principle, both the mean vector and the covariance matrix of the Gaussian model can be calibrated to preserve the features of Boltzmann selection reflecting desired selection pressure. A method is proposed to adapt the selection pressure based on measuring the successfulness of the past evolution process. These works established a formal basis that helps to build probabilistic models in continuous EDA algorithms with adaptive parameters. The framework is incorporated in both the continuous UMDA and the EMNA algorithm, and tested in several benchmark problems. The experiment results are compared with some existing EDA versions and the benefit of the proposed approach is discussed.