Crowdsourcing is widely proposed as a method to solve a large variety of judgment tasks, such as classifying website content, peer grading in online courses, or collecting real-world data. As the data reported by workers cannot be verified, there is a tendency to report random data without actually solving the task. This can be countered by making the reward for an answer depend on its consistency with answers given by other workers, an approach called peer consistency . However, it is obvious that the best strategy in such schemes is for all workers to report the same answer without solving the task. Dasgupta and Ghosh [2013] show that, in some cases, exerting high effort can be encouraged in the highest-paying equilibrium. In this article, we present a general mechanism that implements this idea and is applicable to most crowdsourcing settings. Furthermore, we experimentally test the novel mechanism, and validate its theoretical properties.

In online labor platforms such as Amazon Mechanical Turk, a good strategy to obtain quality answers is to take aggregate answers submitted by multiple workers, exploiting the wisdom of crowds. However, human computation issusceptible to systematic biases which cannot be corrected by using multiple workers.We investigate a game-theoretic bonus scheme, called peer truth serum (PTS), to overcome this problem. We report on the design and outcomes of a set of experiments to validate this scheme. Results show peer truth serum can indeed correct the biases and increase the answer accuracy by up to 80%.

Sensing and monitoring of our natural environment are important for sustainability. As sensor systems grow to a large scale, it will become infeasible to place all sensors under centralized control. We investigate community sensing, where sensors are controlled by self-interested agents that report their measurements to a center. The center can control the agents only through incentives that motivate them to provide the most accurate and useful reports. We consider different game-theoretic mechanisms that provide such incentives and analyze their properties. As an example, we consider an application of community sensing for monitoring air pollution.

Online reviews have become increasingly popular as a way to judge the quality of various products and services. However, recent work demonstrates that the absence of reporting incentives leads to a biased set of reviews that may not reflect the true quality. In this paper, we investigate underlying factors that influence users when reporting feedback. In particular, we study both reporting incentives and reporting biases observed in a widely used review forum, the Tripadvisor Web site. We consider three sources of information: first, the numerical ratings left by the user for different aspects of quality; second, the textual comment accompanying a review; third, the patterns in the time sequence of reports. We first show that groups of users who discuss a certain feature at length are more likely to agree in their ratings. Second, we show that users are more motivated to give feedback when they perceive a greater risk involved in a transaction. Third, a user's rating partly reflects the difference between true quality and prior expectation of quality, as inferred from previous reviews. We finally observe that because of these biases, when averaging review scores there are strong differences between the mean and the median. We speculate that the median may be a better way to summarize the ratings.

Recommender systems based on user feedback rank items by aggregating users' ratings in order to select those that are ranked highest. Ratings are usually aggregated using a weighted arithmetic mean. However, the mean is quite sensitive to outliers and biases, and thus may not be the most informative aggregate. We compare the accuracy and robustness of three different aggregators: the mean, median and mode. The results show that the median may often be a better choice than the mean, and can significantly improve recommendation accuracy and robustness in collaborative filtering systems.

We consider schemes for obtaining truthful reports on a common but hidden signal from large groups of rational, self-interested agents. One example are online feedback mechanisms, where users provide observations about the quality of a product or service so that other users can have an accurate idea of what quality they can expect. However, (i) providing such feedback is costly, and (ii) there are many motivations for providing incorrect feedback. Both problems can be addressed by reward schemes which (i) cover the cost of obtaining and reporting feedback, and (ii) maximize the expected reward of a rational agent who reports truthfully. We address the design of such incentive-compatible rewards for feedback generated in environments with pure adverse selection. Here, the correlation between the true knowledge of an agent and her beliefs regarding the likelihoods of reports of other agents can be exploited to make honest reporting a Nash equilibrium. In this paper we extend existing methods for designing incentive-compatible rewards by also considering collusion. We analyze different scenarios, where, for example, some or all of the agents collude. For each scenario we investigate whether a collusion-resistant, incentive-compatible reward scheme exists, and use automated mechanism design to specify an algorithm for deriving an efficient reward mechanism.

Prediction markets efficiently extract and aggregate the private information held by individuals about events and facts that can be publicly verified. However, facts such as the effects of raising or lowering interest rates can never be publicly verified, since only one option will be implemented. Online opinion polls can still be used to extract and aggregate private information about such questions. This paper addresses incentives for truthful reporting in online opinion polls. The challenge lies in designing reward schemes that do not require a-priori knowledge of the participants' beliefs. We survey existing solutions, analyze their practicality and propose a new mechanism that extracts accurate information from rational participants.

Online reputation mechanisms need honest feedback to function effectively. Self-interested agents report the truth only when explicit rewards offset the potential gains obtained from lying. Feedback payment schemes (monetary rewardsfor submitted feedback) can make truth-telling rational based on the correlation between the reports of different buyers. In this paper we investigate incentive-compatible payment mechanisms that are also resistant to collusion: groups of agents cannot collude on a lying strategy without suffering monetary losses. We analyze several scenarios, where, for example, some or all of the agents collude. For each scenario we investigate both existential and implementation problems. Throughout the paper we use automated mechanism design to compute the best possible mechanism for a given setting.

Online reviews have become increasingly popular as a way to judge the quality of various products and services. Previous work has demonstrated that contradictory reporting and underlying user biases make judging the true worth of a service difficult. In this paper, we investigate underlying factors that influence user behavior when reporting feedback. We look at two sources of information besides numerical ratings: linguistic evidence from the textual comment accompanying a review, and patterns in the time sequence of reports. We first show that groups of users who amply discuss a certain feature are more likely to agree on a common rating for that feature. Second, we show that a user's rating partly reflects the difference between true quality and prior expectation of quality as inferred from previous reviews. Both give us a less noisy way to produce rating estimates and reveal the reasons behind user bias.Our hypotheses were validated by statistical evidence from hotel reviews on the TripAdvisor website.

Service-level agreements (SLAs) establish a contract between service providersand clients concerning Quality of Service (QoS) parameters. Without properpenalties, service providers have strong incentives to deviate from theadvertised QoS, causing losses to the clients. Reliable QoS monitoring (andproper penalties computed on the basis of delivered QoS) are thereforeessential for the trustworthiness of a service-oriented environment. In thispaper, we present a novel QoS monitoring mechanism based on quality ratings from theclients. A reputation mechanism collects the ratings and computes theactual quality delivered to the clients. The mechanism provides incentives forthe clients to report honestly, and pays special attention to minimizing costand overhead1.

Reputation mechanisms offer an effective alternative to verification authorities for building trust in electronic markets with moral hazard. Future clients guide their business decisions by considering the feedback from past transactions; if truthfully exposed, cheating behavior is sanctioned and thus becomes irrational. It therefore becomes important to ensure that rational clients have the right incentives to report honestly. As an alternative to side-payment schemes that explicitly reward truthful reports, we show that honesty can emerge as a rational behavior when clients have a repeated presence in the market. To this end we describe a mechanism that supports an equilibrium where truthful feedback is obtained. Then we characterize the set of pareto-optimal equilibria of the mechanism, and derive an upper bound on the percentage of false reports that can be recorded by the mechanism. An important role in the existence of this bound is played by the fact that rational clients can establish a reputation for reporting honestly.

Online reputation mechanisms need honest feedback to function effectively. Self interested agents report the truth only when explicit rewards offset the cost of reporting and the potential gains that can be obtained from lying. Side-payment schemes (monetary rewards for submitted feedback) can make truth-telling rational based on the correlation between the reports of different buyers.In this paper we use the idea of automated mechanism design to construct the payments that minimize the budget required by an incentive-compatible reputation mechanism. Such payment schemes are defined by a linear optimization problem that can be solved efficiently in realistic settings. Furthermore, we investigate two directions for further lowering the cost of incentive-compatibility: using several reference reports to construct the side-payments, and filtering out reports that are probably false.

Reputation mechanisms offer an efficient way of building the necessary level of trust in electronic markets. Feedback about an agent's past behavior can be aggregated into a measure of reputation, and used by other agents for taking trust decisions. Unfortunately, true feedback cannot be automatically assumed. In the absence of Trusted Third Parties, the mechanism has to make it rational for agents to truthfully share reputation information. In this paper we describe two mechanisms that can be used in decentralized environments for eliciting true feedback. The mechanisms are accompanied by examples inspired by real scenarios.