Failure to accurately measure the outcomes of an experiment can lead to bias and incorrect conclusions. Online controlled experiments (aka AB tests) are increasingly being used to make decisions to improve websites as well as mobile and desktop applications. We argue that loss of telemetry data (during upload or post-processing) can skew the results of experiments, leading to loss of statistical power and inaccurate or erroneous conclusions. By systematically investigating the causes of telemetry loss, we argue that it is not practical to entirely eliminate it. Consequently, experimentation systems need to be robust to its effects. Furthermore, we note that it is nontrivial to measure the absolute level of telemetry loss in an experimentation system. In this paper, we take a top-down approach towards solving this problem. We motivate the impact of loss qualitatively using experiments in real applications deployed at scale, and formalize the problem by presenting a theoretical breakdown of the bias introduced by loss. Based on this foundation, we present a general framework for quantitatively evaluating the impact of telemetry loss, and present two solutions to measure the absolute levels of loss. This framework is used by well-known applications at Microsoft, with millions of users and billions of sessions. These general principles can be adopted by any application to improve the overall trustworthiness of experimentation and data-driven decision making.

User surveys for Quality of Experience (QoE) are a critical source of information for application developers. In addition to the common “star rating” used to estimate Mean Opinion Score (MOS), more detailed survey questions (problem tokens) about specific areas provide valuable insight into the factors impacting QoE. This paper explores two aspects of problem token questionnaire design. First, we study the bias introduced by fixed question order, and second, we provide a methodology to manage the size of the survey while keeping it informative. Based on 900,000 calls gathered using a randomized controlled experiment from Skype, we find that token selections can be strongly biased due to token positions and display design. This selection bias can be significantly reduced by randomizing the display order of tokens. It is worth noting that users respond to the randomized-order variant at levels that are comparable to the fixed-order variant. The effective selection of a subset of tokens is achieved by extracting tokens that provide the highest information gain over user ratings. This selection is known to be in the class of NP-hard problems. We apply a well-known greedy submodular maximization method on our dataset to capture 94% of the information using just 30% of thequestions.

User-perceived quality-of-experience (QoE) in internet telephony systems is commonly evaluated using subjective ratings computed as a Mean Opinion Score (MOS). In such systems, while user MOS can be tracked on an ongoing basis, it does not give insight into which factors of a call induced any perceived degradation in QoE - it does not tell us what caused a user to have a sub-optimal experience. For effective planning of product improvements, we are interested in understanding the impact of each of these degrading factors, allowing the estimation of the return (i.e., the improvement in user QoE) for a given investment. To obtain such insights, we advocate the use of an end-of-call “problem token questionnaire” (PTQ) which probes the user about common call quality issues (e.g., distorted audio or frozen video) which they may have experienced. In this paper, we show the efficacy of this questionnaire using data gathered from over 700,000 end-of-call surveys gathered from Skype (a large commercial VoIP application). We present a method to rank call quality and reliability issues and address the challenge of isolating independent factors impacting the QoE. Finally, we present representative examples of how these problem tokens have proven to be useful in practice.

The packet loss characteristics of Internet paths that include residential broadband links are not well understood, and there are no good models for their behaviour. This complicates the design of real-time video applications targeting home users, since it is difficult to choose appropriate error correction and concealment algorithms without a good model for the types of loss observed. Using measurements of residential broadband networks in the UK and Finland, we show that existing models for packet loss, such as the Gilbert model and simple hidden Markov models, do not effectively model the loss patterns seen in this environment. We present a new two-level Markov model for packet loss that can more accurately describe the characteristics of these links, and quantify the effectiveness of this model. We demonstrate that our new packet loss model allows for improved application design, by using it to model the performance of forward error correction on such links.

Packet loss is a major problem for real-time Internet applications. Markov models of packet loss are often used to develop and evaluate the performance of these applications. Despite their wide use, these models have not been validated in terms of how well they capture the loss conditions experienced by residential Internet users. We evaluate the accuracy of common packet loss models using traces of IPTV-like traffic measured on residential ADSL and Cable links, and find that these models are insufficient to capture the observed packet loss patterns. We introduce a new type of model, incorporating packet delay information, and show improved accuracy over previous models.

Spotify is a popular music-streaming service which has seen widespread use across Europe. While Spotify's server- side behaviour has previously been studied, little is known about the client-side behaviour. In this paper, we describe an experimental study where we collect packet headers for Spotify traffic over multiple 24-hour time frames at a client host. Two distinct types of behaviour are observed, when tracks are being downloaded, and when the client is only serving requests from other peers. We also note wide variation in connection lifetimes, as seen in other studies of peer-to-peer systems. These findings are relevant for improving Spotify itself, and for the designers of other hybrid peer-to-peer and server-based distribution architectures.

Little performance data currently exists for streaming high-quality Internet video to residential users. Data on streaming performance will provide valuable input to the design of new protocols and applications, such as congestion control and error-correction schemes, and sizing playout buffers in video receivers. This paper presents measurements of streaming real-time UDP traffic to a number of residential users, and discusses the basic characteristics of the data.

Packet loss is one of the principal threats to quality of experience for IPTV systems. However, the packet loss characteristics of the residential access networks which carry IPTV are not widely understood. We present packet level measurements of streaming IPTV-like traffic over four residential access links, and describe the extent and nature of packet loss we encountered. We discuss the likely impact of these losses for IPTV traffic, and outline steps which can ameliorate this.