As Machine Learning-based applications become increasingly pervasive, a growing concern is how to balance the need for large, representative data sets with the need to respect user data privacy. The increased compute and connectivity capabilities of edge devices (e.g. phones, PCs) presents us with new avenues for achieving this balance, including a promising approach known as federated learning with local privacy. However, today we have gaps in practical knowledge about applicability, trade-offs, and benefits for large-scale realworld implementation. In this paper, using large-scale data from a real-world Windows Update ML-driven application (as well as the publicly available CIFAR-10 data set to enhance reproducibility), we report empirical evaluations of four practical considerations: heterogeneity in device availability that may cause bias, resiliency of federated learning with local differential privacy, benefits of time-varying adaptive configurations, and data transmission/storage savings based on the Pareto principle. We discuss the implications of these findings for practitioners and researchers.

Randomized controlled trials (i.e. A/B testing) is the gold standard for evaluating improvements and accelerating innovations in the digital space. Prior work has shown that qualitative user feedback is an effective and important tool in the analysis of A/B tests. However, manual inductive qualitative analysis of feedback-best practices today-is expensive and unscalable, which may lead to the omission of important insights about quality and user experiences. Prior work has shown various automated NLP algorithms to be effective in extracting insights from feedback in the digital domain. But we lack understanding of the differences in insights gained from a manual inductive qualitative analysis versus automated NLP algorithms, for analyzing digital randomized controlled trials (where a key objective is understanding differences between control and treatment conditions). In this paper, we compare insights from manual inductive qualitative analyses and from six automated NLP algorithms, using data from large-scale real-world digital randomized controlled trials. We find that collocation algorithms (notable, trigrams) are promising, providing similar insights as manual analyses with substantially lower cost; however, issues remain and improvements are needed. We discuss implications for future research and for operationalization.

Good software engineers are essential to the creation of good software. However, most of what we know about software-engineering expertise are vague stereotypes, such as 'excellent communicators' and 'great teammates'. The lack of specificity in our understanding hinders researchers from reasoning about them, employers from identifying them, and young engineers from becoming them. Our understanding also lacks breadth: what are all the distinguishing attributes of great engineers (technical expertise and beyond)? We took a first step in addressing these gaps by interviewing 59 experienced engineers across 13 divisions at Microsoft, uncovering 53 attributes of great engineers. We explain the attributes and examine how the most salient of these impact projects and teams. We discuss implications of this knowledge on research and the hiring and training of engineers.

Many software producers utilize beta programs to predict post-release quality and to ensure that their products meet quality expectations of users. Prior work indicates that software producers need to adjust predictions to account for usage environments and usage scenarios differences between beta populations and post-release populations. However, little is known about how usage characteristics relate to field quality and how usage characteristics differ between beta and post-release. In this study, we examine application crash, application hang, system crash, and usage information from millions of Windows® users to 1) examine the effects of usage characteristics differences on field quality (e.g. which usage characteristics impact quality), 2) examine usage characteristics differences between beta and post-release (e.g. do impactful usage characteristics differ), and 3) report experiences adjusting field quality predictions for Windows. Among the 18 usage characteristics that we examined, the five most important were: the number of application executed, whether the machines was pre-installed by the original equipment manufacturer, two sub-populations (two language/geographic locales), and whether Windows was 64-bit (not 32-bit). We found each of these usage characteristics to differ between beta and post-release, and by adjusting for the differences, accuracy of field quality predictions for Windows improved by ~59%.

Assessing the reliability of mass-market software (MMS), such as the Windowsregoperating system, presents many challenges. In this paper, we share insights gained from the Windows Vistaregand Windows VistaregSP1 operating systems. First, we find that the automated reliability monitoring approach, which periodically reports reliability status, provides higher quality data and requires less effort compared to other approaches available today. We describe one instance in detail: the Windows reliability analysis component, and illustrate its advantages using data from Windows Vista. Second, we show the need to account for usage scenarios during reliability assessments. For pre-release versions of Windows Vista and Vista SP1, usage scenarios differ by 2-4X for Microsoft internal and external samples; corresponding reliability assessments differ by 2-3X. Our results help motivate and guide further research in reliability assessment.

Software producers can better manage the quality of their deployed software products using estimates of quality. Current best practices for making estimates are to use software reliability growth modeling (SRGM), which assumes that testing environments approximate deployment environments. This important assumption does not hold for widely used software products, which are operated in a wide variety of configurations under many different usage scenarios. However, the literature contains little empirical data on the impact of this violation of assumptions on the accuracy and the usefulness of predictions. In this paper, we report results and experiences using SRGM on an IBM federated database project. We examine defect data from 3 releases spanning approximately 9 years. We find SRGM to be of limited use to the project: absolute relative errors are at least 34%, and predictions are, at times, implausible. We discuss alternative approaches for estimating quality of widely used software products.

Quantitatively-based risk management can reduce the risks associated with field defects for both software producers and software consumers. In this paper, we report experiences and results from initiating risk-management activities at a large systems development organization. The initiated activities aim to improve product testing (system/integration testing), to improve maintenance resource allocation, and to plan for future process improvements. The experiences we report address practical issues not commonly addressed in research studies: how to select an appropriate modeling method for product testing prioritization and process improvement planning, how to evaluate accuracy of predictions across multiple releases in time, and how to conduct analysis with incomplete information. In addition, we report initial empirical results for two systems with 13 and 15 releases. We present prioritization of configurations to guide product testing, field defect predictions within the first year of deployment to aid maintenance resource allocation, and important predictors across both systems to guide process improvement planning. Our results and experiences are steps towards quantitatively-based risk management.

Predicting software quality as perceived by a customer may allow an organization to adjust deployment to meet the quality expectations of its customers, to allocate the appropriate amount of maintenance resources, and to direct quality improvement efforts to maximize the return on investment. However, customer perceived quality may be affected not simply by the software content and the development process, but also by a number of other factors including deployment issues, amount of usage, software platform, and hardware configurations. We predict customer perceived quality as measured by various service interactions, including software defect reports, requests for assistance, and field technician dispatches using the afore mentioned and other factors for a large telecommunications software system. We employ the non-intrusive data gathering technique of using existing data captured in automated project monitoring and tracking systems as well as customer support and tracking systems. We find that the effects of deployment schedule, hardware configurations, and software platform can increase the probability of observing a software failure by more than 20 times. Furthermore, we find that the factors affect all quality measures in a similar fashion. Our approach can be applied at other organizations, and we suggest methods to independently validate and replicate our results.

Open source software systems are critical infrastructure for many applications; however, little has been precisely measured about their quality. Forecasting the field defect-occurrence rate over the entire lifespan of a release before deployment for open source software systems may enable informed decision-making. In this paper, we present an empirical case study often releases of OpenBSD. We use the novel approach of predicting model parameters of software reliability growth models (SRGMs) using metrics-based modeling methods. We consider three SRGMs, seven metrics-based prediction methods, and two different sets of predictors. Our results show that accurate field defect-occurrence rate forecasts are possible for OpenBSD, as measured by the Theil forecasting statistic. We identify the SRGM that produces the most accurate forecasts and subjectively determine the preferred metrics-based prediction method and set of predictors. Our findings are steps towards managing the risks associated with field defects

Defect-occurrence projection is necessary for the development of methods to mitigate the risks of software defect occurrences. In this paper, we examine user-reported software defect-occurrence patterns across twenty-two releases of four widely-deployed, business-critical, production, software systems: a commercial operating system, a commercial middleware system, an open source operating system (OpenBSD), and an open source middleware system (Tomcat). We evaluate the suitability of common defect-occurrence models by first assessing the match between characteristics of widely-deployed production software systems and model structures. We then evaluate how well the models fit real world data. We find that the Weibull model is flexible enough to capture defectoccurrence behavior across a wide range of systems. It provides the best model fit in 16 out of the 22 releases. We then evaluate the ability of the moving averages and the exponential smoothing methods to extrapolate Weibull model parameters using fitted model parameters from historical releases. Our results show that in 50% of our forecasting experiments, these two naïve parameter-extrapolation methods produce projections that are worse than the projection from using the same model parameters as the most recent release. These findings establish the need for further research on parameter-extrapolation methods that take into account variations in characteristics of widely-deployed, production, software systems across multiple releases.