The number of mobile apps and the number of mobile devices have increased considerably in the past few years. To succeed in the competitive market of mobile apps, such as Google Play Store, developers should improve the user-perceived quality of their apps. In this paper, we investigate the relationship between mobile device attributes and the user-perceived quality of Android apps. We observe that the user-perceived quality of apps varies across devices. Device attributes, such as the CPU and the screen resolution, share a significant relationship with the user-perceived quality. However, having a better characteristic of an attribute, such as a higher display resolution, does not necessarily share a positive relationship with the user-perceived quality. App developers should not only consider the app attributes but also consider the device attributes of the available devices to deliver high-quality apps. The original paper is published in the Empirical Software Engineering journal communicated by Lin Tan.

Performance issues are often the cause of failures in today's large-scale software systems. These issues make performance testing essential during software maintenance. However, performance testing is faced with many challenges. One challenge is determining how long a performance test must run. Although performance tests often run for hours or days to uncover performance issues (e.g., memory leaks), much of the data that is generated during a performance test is repetitive. Performance analysts can stop their performance tests (to reduce the time to market and the costs of performance testing) if they know that continuing the test will not provide any new information about the system's performance. To assist performance analysts in deciding when to stop a performance test, we propose an automated approach that measures how much of the data that is generated during a performance test is repetitive. Our approach then provides a recommendation to stop the test when the data becomes highly repetitive and the repetitiveness has stabilized (i.e., little new information about the systems' performance is generated). We performed a case study on three open source systems (i.e., CloudStore, PetClinic and Dell DVD Store). Our case study shows that our approach reduces the duration of 24-hour performance tests by 75% while preserving more than 91.9% of the information about the system's performance. In addition, our approach recommends a stopping time that is close to the most cost-effective stopping time (i.e., the stopping time that minimize the duration of the test and maximizes the amount of information about the system's performance provided by performance testing).

A good understanding of the factors impacting defects in software systems is essential for software practitioners, because it helps them prioritize quality improvement efforts (e.g., testing and code reviews). Defect prediction models are typically built using classification or regression analysis on product and/or process metrics collected at a single point in time (e.g., a release date). However, current defect prediction models only predict if a defect will occur, but not when, which makes the prioritization of software quality improvements efforts difficult. To address this problem, Koru et al. applied survival analysis techniques to a large number of software systems to study how size (i.e., lines of code) influences the probability that a source code module (e.g., class or file) will experience a defect at any given time. Given that 1) the work of Koru et al. has been instrumental to our understanding of the size-defect relationship, 2) the use of survival analysis in the context of defect modelling has not been well studied and 3) replication studies are an important component of balanced scholarly debate, we present a replication study of the work by Koru et al. In particular, we present the details necessary to use survival analysis in the context of defect modelling (such details were missing from the original paper by Koru et al.). We also explore how differences between the traditional domains of survival analysis (i.e., medicine and epidemiology) and defect modelling impact our understanding of the size-defect relationship. Practitioners and researchers considering the use of survival analysis should be aware of the implications of our findings.

The rise of large-scale software systems poses new challenges for the software performance engineering field. Failures in these systems are typically associated with performance issues, rather than with feature bugs. Therefore, performance testing has become essential to ensuring the problem-free operation of these systems. However, the performance testing process is faced with a major challenge: evolving field workloads and deployment configurations, often lead to "outdated" tests that are not reflective of the field. Hence performance analysts must maintain their field-representative tests by evolving these tests. Therefore, we propose new approaches for maintaining and evolving field-representative performance tests. We believe that field-representative tests can be autonomically maintained and evolved, i.e., the configurable parameters of a performance test can be optimized, such that the resulting test exercises the system in a similar manner as the field. Such approaches should help performance analysts to ensure the "realism" of their performance testing efforts, resulting in higher quality systems.

Ultra-Large-Scale (ULS) systems face continuously evolving field workloads in terms of activated/disabled feature sets, varying usage patterns and changing deployment configurations. These evolving workloads often have a large impact, on the performance of a ULS system. Hence, continuous load testing is critical to ensuring the error-free operation of such systems. A common challenge facing performance analysts is to validate if a load test closely resembles the current field workloads. Such validation may be performed by comparing execution logs from the load test and the field. However, the size and unstructured nature of execution logs makes such a comparison unfeasible without automated support. In this paper, we propose an automated approach to validate whether a load test resembles the field workload and, if not, determines how they differ by compare execution logs from a load test and the field. Performance analysts can then update their load test cases to eliminate such differences, hence creating more realistic load test cases. We perform three case studies on two large systems: one open-source system and one enterprise system. Our approach identifies differences between load tests and the field with a precision of >75% compared to only >16% for the state-of-the-practice.

Load tests ensure that software systems are able to perform under the expected workloads. The current state of load test analysis requires significant manual review of performance counters and execution logs, and a high degree of system-specific expertise. In particular, memory-related issues (e.g., memory leaks or spikes), which may degrade performance and cause crashes, are difficult to diagnose. Performance analysts must correlate hundreds of megabytes or gigabytes of performance counters (to understand resource usage) with execution logs (to understand system behaviour). However, little work has been done to combine these two types of information to assist performance analysts in their diagnosis. We propose an automated approach that combines performance counters and execution logs to diagnose memory-related issues in load tests. We perform three case studies on two systems: one open-source system and one large-scale enterprise system. Our approach flags ≤ 0.1% of the execution logs with a precision ≥ 80%.

Large-scale software systems handle increasingly larger workloads by implementing highly concurrent and distributed design patterns. The thread pool pattern uses pools of pre-existing and reusable threads to limit thread lifecycle over-head (thread creation and destruction) and resource thrashing (thread proliferation). However, these advantages are weighed against performance issues caused by concurrency risks, like synchronization errors or deadlock, and thread pool-specific risks, like poorly tuned pool size or thread leakage. Detecting these performance issues during load testing requires a thorough understanding of how thread pools behave, yet most performance analysts have limited knowledge of the system and are flooded with terabytes of data from load tests. We propose a methodology to identify threads with performance deviations in thread pools. Our methodology ranks threads based on the dissimilarity of their resource usage metrics. A case study on a large-scale industrial software system shows that our methodology can identify threads with performance deviations with an average precision of 100% and an average recall of 76.61%. Our methodology performs very well when ranking long-lived deviations, such as memory leaks, but more work is needed to rank short-lived deviations, such as CPU spikes.

Large-scale software systems achieve concurrency on enormous scales using a number of different design patterns. Many of these design patterns are based on pools of pre-existing and reusable threads that facilitate incoming service requests. Thread pools limit thread lifecycle overhead (thread creation and destruction) and resource thrashing (thread proliferation). Despite their potential for scalability, thread pools are hard to configure and test because of concurrency risks like synchronization errors and dead lock, and thread pool-specific risks like resource thrashing and thread leakage. Addressing these challenges requires a thorough understanding of the behaviour of the threads in the thread pool. We argue for a methodology to automatically identify and rank deviations in the behaviour of threads based on resource usage.

The recent meteoric rise in the use of smart phones and other mobile devices has led to a new class of applications, i.e., micro-apps, that are designed to run on devices with limited processing, memory, storage and display resources. Given the rapid succession of mobile technologies and the fierce competition, micro-app vendors need to release new features at break-neck speed, without sacrificing product quality. To understand how different mobile platforms enable such a rapid turnaround-time, this paper compares three pairs of feature-equivalent Android and Blackberry micro-apps. We do this by analyzing the micro-apps along the dimensions of source code, code dependencies and code churn. BlackBerry micro-apps are much larger and rely more on third party libraries. However, they are less susceptible to platform changes since they rely less on the underlying platform. On the other hand, Android micro-apps tend to concentrate code into fewer files and rely heavily on the Android platform. On both platforms, code churn of micro-apps is very high.