This paper addresses the problem of dynamically assigning tasks to a crowd consisting of AI and human workers. Currently, crowdsourcing the creation of AI programs is a common practice. To apply such kinds of AI programs to the set of tasks, we often take the ``all-or-nothing'' approach that waits for the AI to be good enough. However, this approach may prevent us from exploiting the answers provided by the AI until the process is completed, and also prevents the exploration of different AI candidates. Therefore, integrating the created AI, both with other AIs and human computation, to obtain a more efficient human-AI team is not trivial. In this paper, we propose a method that addresses these issues by adopting a ``divide-and-conquer'' strategy for AI worker evaluation. Here, the assignment is optimal when the number of task assignments to humans is minimal, as long as the final results satisfy a given quality requirement. This paper presents some theoretical analyses of the proposed method and an extensive set of experiments conducted with open benchmarks and real-world datasets. The results show that the algorithm can assign many more tasks than the baselines to AI when it is difficult for AIs to satisfy the quality requirement for the whole set of tasks. They also show that it can flexibly change the number of tasks assigned to multiple AI workers in accordance with the performance of the available AI workers.

Automatic extraction of technical terms is an important task for organizing a set of documents. While the sequence labeling formulation is the major approach, we explore a method that takes examples of terms as input and outputs phrases in the same category as the given terms to avoid the heavy cost for building a training dataset. The existing methods in this direction are template-based, which the user cannot give any feedback to the system even if some of the extracted terms are not intended. This paper proposes a framework for extracting technical terms that considers the user’s feedback by adopting active learning approach. The proposed method can extract terms consisting of multiple words by dynamically accessing an inverted index created in advance. We empirically show the effectiveness of the proposed method in comparison to the straightforward application of active learning to an existing method.

Phrase extraction is an information extraction task that extracts words or phrases in a specific category from text data, which is used in various downstream NLP technologies, including named entity recognition (NER), terminology extraction, question answering, dialogue systems, and information integration. While we need a large amount of annotated corpus for training a phrase extractor using machine learning, building such a corpus requires a lot of manual annotation work by domain experts. This research aims to reduce the annotation cost by developing a method that trains a phrase extractor using natural language explanations from experts. The proposed method transforms the natural language explanations into labeling functions, which allows us to make pseudo annotated corpus from a set of raw sentences. We empirically show the effectiveness of the model through experimental results.

Crowdsourcing is widely used in making annotated examples for supervised learning, but these labels from workers are often noisy. A practical solution for this problem is to collect multiple labels per examples and aggregate them. Although gathering repeated labels improves data quality, concerning the fact that there’s a limit on the budget we can use for crowdsourcing, it can also lead to decreasing the amount of training data. This paper empirically examines to what extent the repeated labeling contribute to the accuracy of machine learning models for image classification and text information extraction in various conditions varying budgets and worker qualities. Experiments are designed to test 4 hypotheses we made each determining the effect of budgets, worker quality, task difficulty, and redundancy on crowdsourcing. The experimental results on image classification and named entity recognition supported all 4 hypotheses and surprisingly suggests, that the repeated labeling almost always brings a negative impact on machine learning accuracy.

How to develop AI systems that can explain how they made decisions is one of the important and hot topics today. Inspired by the dual-process theory in psychology, this paper proposes a human-in-the-loop approach to develop System-2 AI that makes an inference logically and outputs interpretable explanation. Our proposed method first asks crowd workers to raise understandable features of objects of multiple classes and collect training data from the Internet to generate classifiers for the features. Logical decision rules with the set of generated classifiers can explain why each object is of a particular class. In our preliminary experiment, we applied our method to an image classification of Asian national flags and examined the effectiveness and issues of our method. In our future studies, we plan to combine the System-2 AI with System-1 AI (e.g., neural networks) to efficiently output decisions.

In a data labeling process for building machine learning, the choice of labeling data instances is known to have a significant impact on the performance of classifiers. So far, the study of active learning has addressed the issue of how to choose the subset by prioritizing the data instances based on the state of the current classifier. However, the active learning approach has two drawbacks that (i) require a training loop to update the priorities of labeling tasks and (ii) require us to choose a specific active learner while we do not know the optimal classification model. In this paper, we propose a new framework of priority-aware labeling system that allows a parallel task assignment to crowd workers without assuming a particular classifier, which is based on novel methods called “batch prioritization” and “label expansion”. We conducted experiments with multiple datasets to examine the effectiveness of the approach and found that the proposed method improves the performance of the final classifiers more quickly than the active learning approach despite that the labeling tasks can be processed in a fully parallel manner.

Named entity recognition (NER) is a fundamental technique that brings basic semantic awareness to natural language processing applications and services. Since we need a large amount of training data to train a custom NER model, distant supervision that leverages named entity dictionaries is expected to be a promising approach to train NER models quickly. However, dictionary matching causes a considerable number of errors that deteriorates both precision and recall of the final NER models, and we need to mitigate its effect. In this study, we particularly aim at improving precision of NER models by accounting for dictionary matching errors. Experimental results show that the proposed method can achieve an improvement of precisions especially under poor dictionary performance conditions.

Semantic content generalization is a method for text summarization that reduces the difficulty of training of neural networks by replacing some phrases such as named entities with generalized terms. The semantic content generalization has achieved remarkable results in enhancing the performance of the sequence to sequence attention model. Besides that, the pointer generator network could ease the training of the summarization based on a mechanism that copies words from the original text, which shares a similar idea with semantic content generalization. The purpose of this work is to test and verify the effect of semantic content generalization on the pointer generator network. Therefore, we use the preprocessing of semantic content generalization and then combine it with the pointer generator network. We examine the performance through an experiment using CNN/DailyMail dataset. From the experiment, we found that the semantic content generalization can improve the performance of the pointer generator network.

We are conducting a research to train second language(L2) learners's second language ability by utilizing chat system. The main problem of existing chat systems is that it is not possible to chat with learners to adapt their second language level. In this research, in order to add a function to an existing chat system we need to measure the learner's second language level. So, to extract learners' second language capability, we propose a method to predict the language examination score of learners from chat context. This research investigates, first whether the number of utterances, number of sentences, word tokens and word types per utterance of chat context are correlated with second language examination score. Second, we build a predicting model to see the relationship between the chat context and second language examination score. As feature values of regression model for predicting the language examination score, we use variables chat time, sentence time, word token and word type. Also the unnatural sentence structure as a variable. For evaluation we use the root mean square error to check the results of prediction model, we use this model with Japanese and English chat and compare the results. We show how this chat context data is affecting the second language examination score and discuss strategies for future enhancements.

News extraction from Twitter data is a hot topic. But can we extract much more than just news? The purpose of this research is to find, either news is the only information which can be extracted from Twitter data or it contains much more insights about real life events. So, we introduce a technique for analysis of Twitter's raw content. After pre-processing of tweets data, we apply hashtag pooling and extract topics using available topic modeling algorithm Latent Dirichlet Allocation (LDA) without modifying its core machinery. In the second part, estimated number of tweets per day and correlated top hashtags for each topic are calculated using day-hashtag pooling. Finally, the continues time series graph is constructed for topic analysis. Our findings show interesting results of bursty news detection, topic popularity, people's way to perceiving an event, real-life event's transition over time and before & after affects of a specific event.

Named entity recognition (NER) research has been spreading into specialty domains. A specialty domain corpus is smaller than a general domain corpus. Moreover, annotating a specialty domain corpus is more expensive than annotating a general corpus. Therefore, in this paper, we introduce a model that uses point-wise prediction and active learning to achieve a high extraction performance even in a small annotation corpus. We demonstrate the effectiveness of our approach through a simulation of active learning.

Extracting topics and their popularities in microblogs is a promising approach to discover popular topics in the world. To challenge this task, some methods that estimate popularity of topics based on Latent Dirichlet Allocation (LDA) has been proposed. However, LDA fails to extract favorable topics on a collection of short text documents such as microblogs because the word co-occurrence information in an individual document is sparse. Therefore, in order to extract topics from microblogs, we should use a model specialized for short text documents. In this paper, we propose a topic popularity estimation method using Biterm TopicModel (BTM), which can alleviate the problem caused by document level word co-occurrence sparsity. We extract topics from the microblog documents with BTM for each time period and estimate the frequency of each topic occurrence. The proposed method can analyze the popularity of topics in a real time because we apply anefficient inference algorithm for BTMonsmall batches of tweets. Experiments on tweets collection show that some of the topics extracted by the proposed method correspond to the real world events and a topic burstiness gets higher when the event occurs.

In this paper, we propose a new approach to unsupervised phrase detection that is based on a sentence segmentation. Unlike the existing approach that examines only word-based statistics, the proposed method detects phrases by considering the most likely segmentation for each sentence. We develop a Bayesian model that estimates phrase boundaries and the grammatical roles of each phrase at the same time, which can be trained in an unsupervised manner by using Gibbs sampling. The experimental results show that the phrase detection by using the proposed model can recognize about 30 times more phrases than the existing popular method in the same precision because of the successful detection of infrequent phrases.

Twitter reflects events and trends in users' real lives because many of them post tweets related to their experiences. Many studies have succeeded in detecting events such as earthquakes and influenza epidemics, along with real-life information from a large amount of tweets, by assuming users as social sensors. On the other hand, inactive users who don't engage in posting activity, are increasing according as time progresses. To collect a large amount of tweets based on specific users for successful Twitter studies, we have to know the characteristics of users who are active over long periods of time. In this paper, we clarify the characteristics of growth users over a long time to strategically collect a large amount of specific users' tweets. We explore the status of users who were active in 2012, and classify users into three statuses of Dead, Lock, and Alive. Based on the differences between the numbers of tweets in 2012 and 2016, we further classify alive users into three types of Eraser, Slumber, and Growth. We analyze the characteristic feature values observed in each user behavior and provide interesting findings with each status/type based on GMM clustering and point-wise mutual information. Finally, we propose a growth user prediction method by a simple formula consisting of feature values and evaluate the effectiveness. We found that active users more easily dropped out than inactive users, and users who engaged in reciprocal communications by replies and retweets often became Growth type.

Many Twitter users post tweets that are related to their particular interests. Users can also collect information by following other users. One approach clarifies user interests by tagging labels based on the users. A user tagging method is important to discover candidate users with similar interests. Typical approaches estimate user interests with terms in tweets and by applying graph theory such as following networks. In contrast, we propose a new user tagging method using the posting time series data of the number of tweets and developed the following hypothesis: Since users have interests, they will post more tweets at the time occurring the events compared with general times. Based on this hypothesis, we extract interests as burst levels from the user and hashtag time series data with Kleinberg's burst enumerating algorithm. We manage the burst levels of users as the term frequency in documents and calculate the hashtag scores for each user by three typical score calculation methods: cosine similarity, Naive Bayes, and TF-IDF. Thus, the proposed method needs no linguistic analysis which requires heavy computational resources. With our sophisticated experimental evaluations with actually active users, we demonstrate the high efficiency of our tagging methods, evaluate them using such information retrieval system evaluation metrics as expected reciprocal rank (ERR) and Q-measure, and clarify the strengths and limitations of each one. Naive Bayes and cosine similarity are especially suitable for user tagging and tag score calculation tasks.