Abstract Large-scale crowdsourcing with high-quality results, such as online surveys and data labeling tasks, is in wide demand. Existing techniques on task assignment result quality optimization, however, have addressed only a part of optimization space that does not necessarily reflect the real-world problem. For example, in reality, the proper worker set for task assignment and the workers’ submissions to the task are non-obvious in advance, which is not considered by existing techniques. Hence, this paper discusses a task assignment algorithm that dynamically probes the proper worker set(s) and worker submission models for the task to optimize task assignments in terms of quality, time, and expense. Specifically, we introduce CrowdBwO (Crowd Bandit with Optimization), a novel multi-armed bandit algorithm that is based on batched bandits and bandits with Knapsack and incorporates worker submission models. CrowdBwO dynamically determines and utilizes proper worker set(s) and worker submission models for each task under uncertainty to achieve high-performance crowdsourcing. We conducted extensive experiments with synthetic workers and real workers to evaluate CrowdBwO in two specific problem settings. Our extensive experimental results demonstrate that CrowdBwO is significant for real-world crowdsourcing and has a high performance.

Predicting how social networks change in the future is important in many applications. Results in social network research have shown that the change in the network can be explained by a small number of concepts, such as "homophily" and "transitivity". However, existing prediction methods require many latent features that are not connected to such concepts, making the methods' black boxes and their prediction results difficult to interpret, making them harder to derive scientific knowledge about social networks. In this study, we propose NetEvolve a novel multi-agent reinforcement learning-based method that predicts changes in a given social network. Given a sequence of changes as training data, NetEvolve learns the characteristics of the nodes with interpretable features, such as how the node feels rewards for connecting with similar people and the cost of the connection itself. Based on the learned feature, NetEvolve makes a forecast based on multi-agent simulation. The method achieves comparable or better accuracy than existing methods in predicting network changes in real-world social networks while keeping the prediction results interpretable.

Nowadays, it is a common practice for crowd workers to develop ML models that classify data items. We envision the Human+AI crowd where crowd programmers develop "AI workers," which are black-box software agents that work among other human workers. The problem here is evaluating such AI workers is different from evaluating human workers in that they may not be spam workers, although they have low accuracy at the beginning of their learning process or for a particular label. Therefore, existing work evaluates the output from AI workers every time they output the task results. Obviously, such a naive evaluation does not scale because there are a tremendous number of task results to be evaluated. This paper addresses the problem of how to efficiently evaluate AI worker outputs by skipping the AI evaluation when the AI is unlikely to satisfy the expected accuracy. We conducted an experiment to compare two strategies and found that both reduce the number of evaluations by orders of magnitude while keeping the number of task assignments to AI workers.

Many existing optimization approaches deal with task assignments on one single crowdsourcing platform. This paper addresses the difficulties of the optimal platform selection for qualification tasks on one single platform. We proposed a novel approach about assigning qualification tasks to workers iteratively on multiple platforms to maximize the total number of collected qualified workers on a limited budget. We applied Multi-Armed Bandit (MAB) algorithms to create strategies for the platform selections to achieve this goal. The conducted experiments revealed that (1) the optimal platform is not always trivial, and (2) the strategies created by MAB algorithms can achieve high-quality assignments under different settings which also satisfied different requesters’ needs.

If a person changes their interests and opinions, how will that change and affect their friendships? Conversely, when a person changes who they are friends with, how will that affect their interests and opinions. Clearly, the person's relationships and interests are related. For example, if several of a person's friends are smokers, but they themselves are not, they are likely to either start smoking as well or to reduce their relationships with the smokers and make new, non-smoking friends. We propose DualCast, a method for predicting the evolution of friendship edges between nodes, as well as the attribute values (which represent opinions and preferences), of nodes in an attributed network. One of the main contributions of the present study is the ability to assume and estimate two scores for each node: the influence (its power to influence neighbors) and its susceptibility (how easily it can be influenced). Our DualCast has the following novel benefits: (A) Expressive: it can capture when links between nodes are dropped, as well as polarization that occurs with changes in interests, (B) Scalable: its performance is linear with input size, (C) Accurate: it is up to 8% more accurate in forecasting links between nodes, and up to 20% more accurate for attribute-values, when tested on publicly available, real datasets with 100K nodes.

When selecting workers in microtask crowdsourcing platforms, requesters select qualified workers by looking at the evaluation results for the tasks in the past or by conducting qualifying tests for the tasks. As a result, they choose workers whose skill levels are above some threshold. This sometimes limits the number of workers who perform the tasks, which has a negative effect for both of requesters and workers. In this paper, we explore an approach to increasing the work opportunities for many workers, by finding task assignment based on the estimated skill level of workers and the difficulty level of tasks. We show the result of a preliminary experiment to discuss the potential and limitation of this approach.

Entity matching is an issue of interest in information integration and data cleaning. Since the representations of the same entity vary, it is often impossible to fully automate the entity matching and require human inputs. However, to guarantee high-quality entity matching, how to integrate human resources into the entity matching while minimizing the cost of human resources? In this paper, we propose BUBBLE, a novel human-in-the-loop entity matching framework hybridizing Bayesian inference and crowdsourcing. To guarantee entity matching quality, Bayesian inference is conducted to determine whether the matching requires crowdsourcing. We show that we can define Bayesian error rate for this problem. For optimization, we use metric learning to select the candidate matching pairs by nearest-neighbor search in the learned embedding space, and we construct a k-nearest neighbor graph to avoid the redundant matching. We applied BUBBLE to a bibliographic data matching problem on the National Diet Library. The experimental results show that BUBBLE can assign tasks to humans with higher quality results compared to those of the same number of task assignments to humans. The result also shows that our optimization scheme is effective without sacrificing the quality.

This paper proposes a scheme of detecting topic evolutions in bibliographic databases. There have been a lot of scientific bibliographies, such as DBLP, CiteSeerX, MEDLINE/PubMed, ADS, arXiv, etc., and hence it has been extremely important to extract useful information from these databases. It should be noticed that, in such databases, citations play crucial role to represent relationships among different publications. To make the best use of citation information as well as textual features for extracting topic evolutions in a bibliographic database, we propose a scheme based on non-negative matrix factorization (NMF). More precisely, we first partition the set of publications in a database according to their publication years, and apply NMF to extract clusters of publications. Notice that we take into account citation information to perform NMF for better clustering. Having obtained sets of publications for each time span, we associate similar clusters in consecutive time spans according to their similarity. Thus we can obtain time evolution of topics and clusters of publications. In the experiments we demonstrate the proposed scheme can successfully extract topic evolutions in real bibliographic databases, CiteSeerX and arXiv.