The coronavirus disease 2019 (COVID-19) pandemic has spread across the globe from the beginning of 2020 and people worldwide have been receiving news about the same from government offices, press conferences and various other media outlets. The COVID-19 Information Watcher Project started in 2020 to collect and organize reliable information sources worldwide. However, it is difficult to automatically identify reliable information sources in foreign countries for several reasons. First, what kind of information sources are reliable heavily depend on each county situation. In some countries people trust their government’s official information but in other countries they do not. Secondly, such reliable information sources often provide information in their local languages. Reliable information sources are not necessarily top-ranked by search engines. Crowdsourcing is a promising way to deal with such a case. However, crowd-sourcing platforms do not cover crowds in all countries. In this study, we report some results of our attempt to collect local information regarding COVID-19 from several countries through multi-hop crowdsourcing, in which we allow crowd workers on a crowdsourcing platform to use other platforms in other countries. We show two case studies, Russia and Afghanistan. Our results show that the multi-hop crowdsourcing is a promising way to collect COVID-19 information from different countries.

This paper addresses the problem of dynamically assigning tasks to a crowd consisting of AI and human workers. Currently, crowdsourcing the creation of AI programs is a common practice. To apply such kinds of AI programs to the set of tasks, we often take the ``all-or-nothing'' approach that waits for the AI to be good enough. However, this approach may prevent us from exploiting the answers provided by the AI until the process is completed, and also prevents the exploration of different AI candidates. Therefore, integrating the created AI, both with other AIs and human computation, to obtain a more efficient human-AI team is not trivial. In this paper, we propose a method that addresses these issues by adopting a ``divide-and-conquer'' strategy for AI worker evaluation. Here, the assignment is optimal when the number of task assignments to humans is minimal, as long as the final results satisfy a given quality requirement. This paper presents some theoretical analyses of the proposed method and an extensive set of experiments conducted with open benchmarks and real-world datasets. The results show that the algorithm can assign many more tasks than the baselines to AI when it is difficult for AIs to satisfy the quality requirement for the whole set of tasks. They also show that it can flexibly change the number of tasks assigned to multiple AI workers in accordance with the performance of the available AI workers.

At disaster response, it is essential to grab whole picture of damage situation quickly and early after disaster occurrence in order to make disaster response effective and efficient. However, it takes much time to understand damage situation because there is not enough information about it. Against this issue, we proposed implementation of CyborgCrowd for situation awareness in disaster response. In order to validate its possibility, we planned the first international disaster drill in October, 2019. In this drill, we simulated to detect flooded area by West Japan Flood occurred in 2018 from aerial photos by collaboration between crowdsourcing and AIs following Human-in-the-Loop process. Especially, in this drill, AIs were also crowdsourced. In this research, we validated the transition of the efforts from crowdsourcing and AIs to detecting flooded area, and verified the accuracy of result by comparing with the actual flooded area published by Geospatial Information Authority of Japan. Furthermore, we found some suggestion about features of detection results by humans and AIs. For example, some humans detected flooded area roughly, however AIs detected it much closely. Based on those features, we proposed the way to decrease the difference between results by humans and AIs. This was essential for local responders to understand the whole picture of damage situation after disaster occurrence urgently. In this paper, we introduced the framework of international disaster drill, clarified the result of validation, and mentioned the possibility of effective collaboration between crowdsourcing and AIs for quick situation awareness in disaster response.

The most typical approach today to data processing which does not have proven algorithms is to first request humans to provide labels to a small set of data and then develop artificial intelligences (AIs) with the data to perform all the remaining tasks. This development is sometimes crowdsourced through platforms such as Kaggle. The approach, however, is not always effective; if the AI does not meet the quality requirement, we may have to give up the development and all the data items have to be done manually. In order to avoid this all-or-nothing situation, “selective” AI programs that perform tasks which they are confident to do will be effective. This study addresses the problem of designing an incentive structure for crowdsourcing the development of such selective AI programs. This paper shows the results of our real-world experiment with a stair-step incentive structure and the behavior of a worker who developed the AI agent under the incentive. This paper also discusses the limitations of the proposed incentive design.

This paper reports the result of a preliminary experiment on active learning strategies for the hierarchical labeling microtasks. A typical example of hierarchical labeling microtask consists of a set of labeling tasks for partitions of a large image; starting from the whole image, the workers choose to give a label or divide it into smaller ones. This paper shows the result of an experiment to compare several strategies for active learning in the setting. The result suggests that the difference in the strategies affects the performance in the early stage.

Can people learn from machines behavior in microtask based crowdsourcing? Can we train the machines as our mentor even without domain expertise? In this paper, we investigate how the task results improve concerning quality during and after presenting machine prediction as a reference answer in self-correction. Four reference types were examined in the experiment; Correct, Random, Machine prediction trained by correct answers, and that trained by human answers. Learning effects were observed only in presenting machine prediction, although those accuracy rates were far from correct (100%). Moreover, there were no learning effects in "Correct" and "Random". This suggests the following hypothesis: Since machine learners make some "models" for the problem, it is easier for humans to interpret the outputs of machine learners than the results without via them; it is more difficult to interpret not only random answers but also the correct answers in a case where the perfect interpretation of the problem is difficult. Furthermore, some workers answered with higher accuracy rate than machines in the post-test. Therefore, this strategy can be expected to be useful for bootstrapping solutions in the situation where unknown problems occur without expertise or at a low cost.

Self-correction for crowdsourced tasks is a two-stage setting that allows a crowd worker to review the task results of other workers; the worker is then given a chance to update his/her results according to the review.Self-correction was proposed as an approach complementary to statistical algorithms in which workers independently perform the same task. It can provide higher-quality results with few additional costs. However, thus far, the effects have only been demonstrated in simulations, and empirical evaluations are needed. In addition, as self-correction gives feedback to workers, an interesting question arises: whether perceptual learning is observed in self-correction tasks. This paper reports our experimental results on self-corrections with a real-world crowdsourcing service.The empirical results show the following: (1) Self-correction is effective for making workers reconsider their judgments. (2) Self-correction is more effective if workers are shown task results produced by higher-quality workers during the second stage. (3) Perceptual learning effect is observed in some cases. Self-correction can give feedback that shows workers how to provide high-quality answers in future tasks.The findings imply that we can construct a positive loop to improve the quality of workers effectively.We also analyze in which cases perceptual learning can be observed with self-correction in crowdsourced microtasks.