Crowdsourcing has been part of the IR toolbox as a cheap and fast mechanism to obtain labels for system development and evaluation. Successful deployment of crowdsourcing at scale involves adjusting many variables, a very important one being the number of workers needed per human intelligence task (HIT). We consider the crowdsourcing task of learning the answer to simple multiple-choice HITs, which are representative of many relevance experiments. In order to provide statistically significant results, one often needs to ask multiple workers to answer the same HIT. A stopping rule is an algorithm that, given a HIT, decides for any given set of worker answers to stop and output an answer or iterate and ask one more worker. In contrast to other solutions that try to estimate worker performance and answer at the same time, our approach assumes the historical performance of a worker is known and tries to estimate the HIT difficulty and answer at the same time. The difficulty of the HIT decides how much weight to give to each worker's answer. In this paper we investigate how to devise better stopping rules given workers' performance quality scores. We suggest adaptive exploration as a promising approach for scalable and automatic creation of ground truth. We conduct a data analysis on an industrial crowdsourcing platform, and use the observations from this analysis to design new stopping rules that use the workers' quality scores in a non-trivial manner. We then perform a number of experiments using real-world datasets and simulated data, showing that our algorithm performs better than other approaches.

We consider the crowdsourcing task of learning the answer to simple multiple-choice microtasks. In order to provide statistically significant results, one often needs to ask multiple workers to answer the same microtask. A stopping rule is an algorithm that for a given microtask decides for any given set of worker answers if the system should stop and output an answer or iterate and ask one more worker. A quality score for a worker is a score that reflects the historic performance of that worker. In this paper we investigate how to devise better stopping rules given such quality scores. We conduct a data analysis on a large-scale industrial crowdsourcing platform, and use the observations from this analysis to design new stopping rules that use the workers’ quality scores in a non-trivial manner. We then conduct a simulation based on a real-world workload, showing that our algorithm performs better than the more naive approaches.

We have proposed the use of dynamically priced network services to provide QoS guarantees within a network. End-to-end QoS can be achieved by using several of these services, perhaps from different ISPs. In this paper we consider the problem of determining the bandwidth to allocate each service in order to maximize revenue, assuming that a single ISP can estimate the demand curves for each of its services. We develop and analyze two heuristics which provide time versus revenue tradeoffs. To determine the optimality of our solutions, we map the optimal allocation problem into a multiple choice multidimensional knapsack problem that approaches optimality as we increase the number of bandwidth allocation choices for each service. Our first heuristic, IterLP, achieves revenue close to 99% of the optimal solution, achieving this result in a very short time. The second heuristic, IterGreedy, achieves approximately 93% optimality, but executes more quickly than IterLP.

We have previously proposed the use of dynamically priced network services to provide QoS guarantees within a network. End-to-end QoS can be achieved by concatenating several of these services from different ISPs. In this paper we consider the problem of a single ISP determining the optimal paths on which to route each service within its network, as well as the optimal bandwidth to allocate to each service, in order for the ISP to maximize its revenue. We assume that the ISP can estimate the demand functions for each service. We define three heuristics: service grouping, iterative bottleneck avoidance, and iterative bottleneck avoidance with tabu. We demonstrate that iterative bottleneck avoidance with tabu achieves approximately 98% of an optimal solution.

Most research in network pricing focuses on improving QoS for a single network. These approaches are not easily extensible to a set of interconnected networks, as the inter-ISP negotiations would be unwieldy. Our approach is to provide a general QoS-aware networking service. We first introduce the concept of QoS-transit services: transit services with QoS guarantees, where dynamic pricing is used to regulate demand on the associated links. After reviewing the structure of the Internet, and the locations of bottlenecks, we describe an architecture for delivering QoS across multiple networks using QoS-transit services. We describe the charging and routing frameworks, and detail how billing, metering, and policing can be achieved. Furthermore, we address security considerations, and discuss compatibilities with current network protocols. Finally, since we cannot expect all ISPs to offer QoS-transit services, we describe the concept of overlay ISPs: ISPs who provide QoS-transit services by controlling an overlay network

We have previously proposed the use of dynamically priced network services to provide QoS guarantees within a network. End-to-end QoS can be achieved by concatenating several of these services, perhaps from different ISPs. In this paper we consider the problem of a single ISP determining the bandwidth to allocate to each service, and on which path, in order to maximize revenue while guaranteeing end-to-end QoS. No knowledge of demand functions is assumed. Optimal allocation of bandwidth to services is first considered, where services are assumed to be routed on predetermined paths. We define the Iterative Allocation Adjustment heuristic, based on the concepts of tatonnement, which, through simulation, is shown to achieve over 95% of the optimal revenue for an ISP. We also examine how to value the links in the network to identify rerouting possibilities, or possible routes for new services, in order to improve the revenue of an ISP.