This article discusses the transition from annotated data to a gold standard, that is, a subset that is sufficiently noise-free with high confidence. Unless appropriately reinterpreted, agreement coefficients do not indicate the quality of the data set as a benchmarking resource: High overall agreement is neither sufficient nor necessary to distill some amount of highly reliable data from the annotated material. A mathematical framework is developed that allows estimation of the noise level of the agreed subset of annotated data, which helps promote cautious benchmarking.

We study a sequential auction for sharing a wireless resource (bandwidth or power) among competing transmitters. The resource is assumed to be managed by a spectrum broker (auctioneer), who collects bids and allocates discrete units of the resource via a sequential second-price auction. It is well known that a second price auction for a single indivisible good has an efficient dominant strategy equilibrium; this is no longer the case when multiple units of a homogeneous good are sold in repeated iterations. For two users with full information, we show that such an auction has a unique equilibrium allocation. The worst-case efficiency of this allocation is characterized under the following cases: (i) both bidders have a concave valuation for the spectrum resource, and (ii) one bidder has a concave valuation and the other bidder has a convex valuation (e.g., for the other useriquests power). Although the worst-case efficiency loss can be significant, numerical results are presented, which show that for randomly placed transmitter-receiver pairs with rate utility functions, the sequential second-price auction typically achieves the efficient allocation. For more than two users it is shown that this mechanism always has a pure strategy equilibrium, but in general there may be multiple equilibria. We give a constructive procedure for finding one equilibrium; numerical results show that when all users have concave valuations the efficiency loss decreases with an increase in the number of users.

A sequence of prices and demands are rationalizable if there exists a concave, continuous and monotone utility function such that the demands are the maximizers of the utility function over the budget set corresponding to the price. Afriat [1] presented necessary and sufficient conditions for a finite sequence to be rationalizable. Varian [20] and later Blundell et al. [3, 4] continued this line of work studying nonparametric methods to forecasts demand. Their results essentially characterize learnability of degenerate classes of demand functions and therefore fall short of giving a general degree of confidence in the forecast. The present paper complements this line of research by introducing a statistical model and a measure of complexity through which we are able to study the learnability of classes of demand functions and derive a degree of confidence in the forecasts.Our results show that the class of all demand functions has unbounded complexity and therefore is not learnable, but that there exist interesting and potentially useful classes that are learnable from finite samples. We also present a learning algorithm that is an adaptation of a new proof of Afriat's theorem due to Teo and Vohra [17].