Despite an abundance of educational resources on the Web, there exists a gap between teachers and the efficient utilization of these resources. A fundamental component of teaching is the preparation of a lesson plan—an organized sequence of educational content—and for the most part, the task of generating lesson plans today is manual and laborious. To address this gap, we present CollectiveTeach, a platform that enables educators to generate lesson plans. CollectiveTeach has two main facets: (i) an information retrieval engine that gathers relevant documents pertaining to a topic, and (ii) a framework to sequence the retrieved documents into coherent lesson plans. We present a novel architecture that leverages information retrieval algorithms, data mining techniques, and user feedback to generate automated lesson plans. We built and deployed CollectiveTeach for 3 popular undergraduate Computer Science subjects: Algorithms, Operating Systems, and Machine Learning, on a corpus of ∼ 100,000 web pages. Further, we evaluated the platform in 3 phases: (1) computing the precision of the documents retrieved, (2) a user study with 10 participants who assessed lesson plans returned by CollectiveTeach based on appropriateness, quality, and coverage and (3) benchmarking our sequencing approach against the Beam-Search approach. Our results show that CollectiveTeach achieves high precision in retrieving content relevant to a user’s query, users are satisfied with the appropriateness, coverage, and reliability of the generated lesson plans and that our sequencing approach is effective. These results indicate that CollectiveTeach is a promising platform that could enrich the lesson plan generation process and encourage collaboration amongst the community of educators and learners.

We study the problem of identifying unreliable and adversarial workers in crowdsourcing systems where workers (or users) provide labels for tasks (or items). Most existing studies assume that worker responses follow specific probabilistic models; however, recent evidence shows the presence of workers adopting non-random or even malicious strategies. To account for such workers, we suppose that workers comprise a mixture of honest and adversarial workers. Honest workers may be reliable or unreliable, and they provide labels according to an unknown but explicit probabilistic model. Adversaries adopt labeling strategies different from those of honest workers, whether probabilistic or not. We propose two reputation algorithms to identify unreliable honest workers and adversarial workers from only their responses. Our algorithms assume that honest workers are in the majority, and they classify workers with outlier label patterns as adversaries. Theoretically, we show that our algorithms successfully identify unreliable honest workers, workers adopting deterministic strategies, and worst- case sophisticated adversaries who can adopt arbitrary labeling strategies to degrade the accuracy of the inferred task labels. Empirically, we show that filtering out outliers using our algorithms can significantly improve the accuracy of several state-of-the-art label aggregation algorithms in real-world crowdsourcing datasets.

Many socio-economic indicators are sensitive to real-world events. Proper characterization of the events can help to identify the relevant events that drive fluctuations in these indicators. In this paper, we propose a novel generative model of real-world events and employ it to extract events from a large corpus of news articles. We introduce the notion of an event class, which is an abstract grouping of similarly themed events. These event classes are manifested in news articles in the form of event triggers which are specific words that describe the actions or incidents reported in any article. We use the extracted events to predict fluctuations in different socio-economic indicators. Specifically, we focus on food prices and predict the price of 12 different crops based on real-world events that potentially influence food price volatility, such as transport strikes, festivals etc. Our experiments demonstrate that incorporating event information in the prediction tasks reduces the root mean square error (RMSE) of prediction by 22% compared to the standard ARIMA model. We also predict sudden increases in the food prices (i.e. spikes) using events as features, and achieve an average 5-10% increase in accuracy compared to baseline models, including an LDA topic-model based predictive model.