Transfer learning and Transformer-based language models play important roles in modern natural language processing research community. In this paper, we propose Transformer model's fine-tuning and data augmentation (TMFTDA) techniques for conversational texts and noisy user-generated content. We use two NTCIR-15 tasks, namely the first Dialogue Evaluation (DialEval-1) task and the second Numeral Attachment in Financial Tweets (FinNum-2) task, to evaluate the efficacy of TMFTDA. Experimental results show that TMFTDA substantially outperforms the baselines model of Bidirectional Long Short-Term Memory (Bi-LSTM) in multi-turn dialogue system evaluation at DialEval-1's Dialogue Quality (DQ) and Nugget Detection (ND) subtasks. Moreover, TMFTDA performs to a satisfactory level at FinNum-2 with a model of Cross-lingual Language Models using a Robustly Optimized BERT Pretraining Approach (XLM-RoBERTa). The research contribution of this paper is that, we help shed some light on the usefulness of TMFTDA, for conversational texts and noisy user-generated content in social media text analytics.

It is hard to evaluate the quality of the generated text by a generative dialogue system. Currently, dialogue evaluation relies on human judges to label the quality of the generated text. It is not a reusable mechanism that can give consistent evaluation for system developers. We believe that it is easier to get consistent results on comparing two generated dialogue by two systems and it is hard to give a consistent quality score on only one system at a time. In this paper, we propose a machine learning approach to reduce the effort of human evaluation by learning the human judgment on comparing two dialogue systems. Training from the human labeling result, the evaluation model learns which generative models is better in each dialog context. Thus, it can be used for system developers to compare the fine-tuned models over and over again without the human labor. In our experiment we find the agreement between the learned model and human judge is 70%. The experiment is conducted on comparing two attention based GRU-RNN generative models.

Before purchasing a product online, customers often read the reviews posted by people who also brought the product. Customer reviews provide opinions and relevant information such as comparisons among similar products or usage experiences about the product. Previous studies addressed on the prediction of the helpfulness of customer reviews to predict the helpfulness voting results. However, the voting result of an online review is not a constant over time; predicting the voting result based on the analysis of text is not practical. Therefore, we collect the voting results of the same online customer review over time, and observe whether the number of votes will increase or not. We construct a dataset with 10,195 online reviews in six different product categories (Computer Hardware, Drink, Makeup, Pen, Shoes, and Toys) from Amazon.cn with the voting result on the helpfulness of the reviews, and monitor the helpfulness voting in six weeks. Experiments are conducted on the dataset to predict whether the helpfulness voting result of each review will increase or not. We propose a classification system that can classify the online reviews into more helpful ones, based on a set of syntactic features and neural features trained via CNN. The results show that integrating the syntactic features with the neural features can get better result.

Customer reviews provide opinions and relevant information that will affect the purchase behavior of other customers. Many studies focused on the prediction of the helpfulness of customer reviews to find the helpful reviews, which are traditionally determined by the helpful voting results. In our study, we find that the voting result of an online review is not a constant over time. Therefore, predicting the voting result based on the analysis of text is not enough; the temporal issue must be considered. We propose a system that can rank the reviews based on a set of linguistic features with a linear regression model. To evaluate our system, we collect Chinese custom reviews in eight product categories (books, digital cameras, tablet PC, backpacks, movies, men shoes, toys and cell phones) from Amazon.cn with the voting result on the helpfulness of the reviews. Since the voting result may be affected by voting time and total voting number, we define a new evaluation index and compare the regression results. The results show that the system has less prediction error when it takes the time information into the prediction model.

This special issue contains four articles based on and expanded from systems presented at the SIGHAN-7 Chinese Spelling Check Bakeoff. We provide an overview of the approaches and designs for Chinese spelling checkers presented in these articles. We conclude this introductory article with a summary of possible future directions.

Information about students’ mistakes opens a window to an understanding of their learning processes, and helps us design effective course work to help students avoid replication of the same errors. Learning from mistakes is important not just in human learning activities; it is also a crucial ingredient in techniques for the developments of student models. In this article, we report findings of our study on 4,100 erroneous Chinese words. Seventy-six percent of these errors were related to the phonological similarity between the correct and the incorrect characters, 46% were due to visual similarity, and 29% involved both factors. We propose a computing algorithm that aims at replication of incorrect Chinese words. The algorithm extends the principles of decomposing Chinese characters with the Cangjie codes to judge the visual similarity between Chinese characters. The algorithm also employs empirical rules to determine the degree of similarity between Chinese phonemes. To show its effectiveness, we ran the algorithm to select and rank a list of about 100 candidate characters, from more than 5,100 characters, for the incorrectly written character in each of the 4,100 errors. We inspected whether the incorrect character was indeed included in the candidate list and analyzed whether the incorrect character was ranked at the top of the candidate list. Experimental results show that our algorithm captured 97% of incorrect characters for the 4,100 errors, when the average length of the candidate lists was 104. Further analyses showed that the incorrect characters ranked among the top 10 candidates in 89% of the phonologically similar errors and in 80% of the visually similar errors.

BACKGROUND: Text mining in the biomedical domain is receiving increasing attention. A key component of this process is named entity recognition (NER). Generally speaking, two annotated corpora, GENIA and GENETAG, are most frequently used for training and testing biomedical named entity recognition (Bio-NER) systems. JNLPBA and BioCreAtIvE are two major Bio-NER tasks using these corpora. Both tasks take different approaches to corpus annotation and use different matching criteria to evaluate system performance. This paper details these differences and describes alternative criteria. We then examine the impact of different criteria and annotation schemes on system performance by retesting systems participated in the above two tasks. RESULTS: To analyze the difference between JNLPBA's and BioCreAtIvE's evaluation, we conduct Experiment 1 to evaluate the top four JNLPBA systems using BioCreAtIvE's classification scheme. We then compare them with the top four BioCreAtIvE systems. Among them, three systems participated in both tasks, and each has an F-score lower on JNLPBA than on BioCreAtIvE. In Experiment 2, we apply hypothesis testing and correlation coefficient to find alternatives to BioCreAtIvE's evaluation scheme. It shows that right-match and left-match criteria have no significant difference with BioCreAtIvE. In Experiment 3, we propose a customized relaxed-match criterion that uses right match and merges JNLPBA's five NE classes into two, which achieves an F-score of 81.5%. In Experiment 4, we evaluate a range of five matching criteria from loose to strict on the top JNLPBA system and examine the percentage of false negatives. Our experiment gives the relative change in precision, recall and F-score as matching criteria are relaxed. CONCLUSION: In many applications, biomedical NEs could have several acceptable tags, which might just differ in their left or right boundaries. However, most corpora annotate only one of them. In our experiment, we found that right match and left match can be appropriate alternatives to JNLPBA and BioCreAtIvE's matching criteria. In addition, our relaxed-match criterion demonstrates that users can define their own relaxed criteria that correspond more realistically to their application requirements.

In this paper, a diagnosis system for problem posing is developed. The domain of this system is restricted to area word problems. The student poses the problems through the system with phrase combination approach. Then the system diagnoses the problem and provides the information for students to revise the posed problem, if needed. The system uses an ontology-based knowledge engineering tool, InfoMap, to represent the concept knowledge for diagnosing area word problems posed by students.

This paper presents a new model to simulate procedural knowledge. This method divides procedural knowledge into two parts: process control and action performer. By adopting this method, we intend to help teacher construct curriculum and teaching strategies by capturing students' problem-solving process. Using the concept of procedural knowledge in intelligent tutoring systems, we can accumulate and duplicate the knowledge of teacher/curriculum manager and student model. Teacher/curriculum manager can help the teacher create good learning maps for students. The student model can help the teacher collect students' error types and design more appropriate teaching strategies. The implementation of our system is near completion. We will design a user-friendly interface for the system and have students play with this software to collect feedback.

We present an ontological approach to the design of the student model for a tutorial agent system (TAS). Our model emphasizes the classification and detection of error types. If the student has any systematic and predictable misconceptions, the system attempts to determine the underlying reasons for such errors. We adopt the "identification, simulation, interaction, and mapping" (ISIM) strategy to achieve this goal. The tutorial agent system first identifies which problem solving method a student is using. It then simulates the procedure in a step-by-step fashion. If there is any ambiguity in the diagnosis of error types during the simulation, the system will interact with the student to resolve it. Finally, the interaction will lead to appropriate error types. The related knowledge is constructed in an ontological framework, InfoMap. In this paper, we focus on how to construct the knowledge and how the simulation works.

We present a knowledge representation scheme, INFOMAP, together with a mechanism that matches the event of a natural language sentence with part of the domain ontology in the INFOMAP The design of this scheme is to facilitate both human browsing and computer processing of the domain ontology. INFOMAP is also a knowledge framework designed to facilitate knowledge sharing by different application systems. We constructed a question answering, system to demonstrate the power of INFOMAP. When the QA-system receives a user's query, it will extract the corresponding events or scripts based on the ontology in INFOMAP The understanding of a question involves extracting such information as the question type, the question subject, the question condition and the question context A dialogue on the question is triggered at the same time to guide the user to retrieve more relevant information.

Although extracting information from tables is essential for Internet information agents, most tables are designed for human eyes and their layout and semantic meanings are not well defined. In practice, encoding the layout of each information source is impossible. This work presents a novel semantic search approach capable of extracting information from general tables. Semantic ontology allows our agents to read tables in the same knowledge domain with different layouts. In addition, a system of layout syntax and a set of transformation rules are defined to transform tables into databases without losing their semantic meanings.

We present a game theoretic coordination mechanism in a multi agent community. We assume that all the agents are rational and have the ability to communicate with each other. In our approach, agents are treated as players in a noncooperative game defined in conventional game theory. In order to make the agents behave coordinately and to avoid an undesirable state, such as Prisoners' dilemma, we introduce a trusted third party into the conventional two-player game theory. The mechanism changes the equilibrium states by altering the payoff of the game. We show how agents are able to recognize undesirable states by reasoning on a 2 by 2 payoff matrix and find a way out by communicating with a trusted third party. A communication protocol among agents and the trusted third party is constructed to achieve a negotiation for coordination.