Canary testing is an emerging technique that offers to minimize the risk of deploying a new version of software. It does so by slowly transferring load from the current to the new ("canary") version. As this ramp-up progresses, a human compares the performance and correctness of the two versions, and assesses whether to abort the canary version. For canary testing to be effective, a plethora of metrics must be analyzed, including CPU utilization and logged errors, across hundreds to thousands of machines. Performing this analysis manually is both time consuming and error prone. In this paper, we present CanaryAdvisor, a tool for automatic canary testing of cloud-based applications. CanaryAdvisor continuously monitors the deployed versions of an application and detects degradations in correctness, performance, and/or scalability. We describe our design and implementation of the CanaryAdvisor and outline open challenges.

The behavior of multithreaded programs is often difficult to understand and predict. Synchronization operations and limited computational resources combine to produce complex non-linear dependencies between a program's configuration parameters and its performance. Performance models are used to understand these dependencies. Such models are complex, and constructing them requires a solid understanding of the program's behavior. As a result, building models of complex applications manually is extremely time-consuming and error-prone. In this paper we demonstrate that such models can be built automatically.

We present an approach for building performance models of multithreaded programs automatically. We use a combination of static and a dynamic analyses of a single representative run of the program to build its model. The model can predict performance of the program under a variety of configurations. This paper outlines how we construct the model and demonstrates how the resultant models accurately predict the performance %and resource utilization of complex multithreaded programs.

Code defects introduced during the development of the software system can result in failures after its release. Such post-release failures are costly to fix and have negative impact on the reputation of the released software. In this paper we propose a methodology for early detection of faulty code changes. We describe code changes with metrics and then use a statistical model that discriminates between faulty and non-faulty changes. The predictions are done not at a file or binary level but at the change level thereby assessing the impact of each change. We also study the impact of code branches on collecting code metrics and on the accuracy of the model. The model has shown high accuracy and was developed into a tool called CheckinMentor. CheckinMentor was deployed to predict risk for the Windows Phone software. However, our methodology is versatile and can be used to predict risk in a variety of large complex software systems.

We consider the problem of understanding locking behavior in large Java programs using a combination of data collection, data analysis, and visualization. Our technique starts by collecting partial information about all locks used in the program. It then analyzes this information to determine sets of locks with common behaviors and to determine, for each set of locks, how that lock is used, e.g. if it is used as a mutex, semaphore, read-write lock, etc. The result of the analysis is then presented to the user who can select specific locks for full analysis during a subsequent run. Visualizing locking information is particularly difficult since the time scale of a lock can be ten or more orders of magnitude different from the time scale of the overall run and locks can be used millions of times. We provide different visualizations and visualization techniques for this purpose. First, we analyze either the partial or full traces and identify patterns of how each lock is used and display just those patterns along with their frequency. Second, we provide a thread-centric view of locking that supports fish-eye views at the microsecond level as well as time compression. Third, we provide a lock-centric view that is based on the specific type of lock to show its particular behavior.

Code Bubbles is an integrated development environment that concentrates on the user experience. The environment is very visual and includes a number of different visualizations, both static and dynamic. We will demonstrate the environment and the various visualizations on a realistic scenario based on our current work.

Predicting the performance of a computer program facilitates its efficient design, deployment, and problem detection. However, predicting performance of multithreaded programs is complicated by complex locking behavior and concurrent usage of computational resources. Existing performance models either require running the program in many different configurations or impose restrictions on the types of programs that can be modeled. This paper presents our approach towards building performance models that do not require vast amounts of training data. Our models are built using a combination of queuing networks and probabilistic call graphs. All necessary information is collected using static and dynamic analyses of a single run of the program. In our experiments these models were able to accurately predict performance of different types of multithreaded programs and detected those configurations that result in the programs' high performance.

The high cost of software maintenance necessitates methods to improve the efficiency of the maintenance process. Such methods typically need a vast amount of knowledge about a system, which is often mined from software repositories. Collecting this data becomes a challenge if the system was developed using multiple code branches. In this paper we present an integration resolution algorithm that facilitates data collection across multiple code branches. The algorithm tracks code integrations across different branches and associates code changes in the main development branch with corresponding changes in other branches. We provide evidence for the practical relevance of this algorithm during the development of the Windows Vista Service Pack 2.

Building large software systems is difficult. Maintaining large systems is equally hard. Making post-release changes requires not only thorough understanding of the architecture of a software component about to be changed but also its dependencies and interactions with other components in the system. Testing such changes in reasonable time and at a reasonable cost is a difficult problem as infinitely many test cases can be executed for any modification. It is important to obtain a risk assessment of impact of such post-release change fixes. Further, testing of such changes is complicated by the fact that they are applicable to hundreds of millions of users, even the smallest mistakes can translate to a very costly failure and re-work. There has been significant amount of research in the software engineering community on failure prediction, change analysis and test prioritization. Unfortunately, there is little evidence on the use of these techniques in day-to-day software development in industry. In this paper, we present our experiences with CRANE: a failure prediction, change risk analysis and test prioritization system at Microsoft Corporation that leverages existing research for the development and maintenance of Windows Vista. We describe the design of CRANE, validation of its useful-ness and effectiveness in practice and our learnings to help enable other organizations to implement similar tools and practices in their environment.

Incorrect changes made to the stable parts of a software system can cause failures - software regressions. Early detection of faulty code changes can be beneficial for the quality of a software system when these errors can be fixed before the system is released. In this paper, a statistical model for predicting software regressions is proposed. The model predicts risk of regression for a code change by using software metrics: type and size of the change, number of affected components, dependency metrics, developerpsilas experience and code metrics of the affected components. Prediction results could be used to prioritize testing of changes: the higher is the risk of regression for the change, the more thorough testing it should receive.