Understanding and continually improving the user experience is critical to the success of web applications, especially those that are business-driven.There exist a multitude of tools to monitor user activities on a website, which then provide metrics to help developers and company leadership understand where their users experience pain-points.However, all tools suffer from their own limitations and ultimately it is important for companies to have as much control (as possible) over all data collected by third-party tools, and be able to make decisions related to its storage, retention, processing, etc., in an agile manner.At Dottid, we use a third-party tool to understand and enhance the user experience on our web application, recently however, we ran into a problem regarding failed logins that required us to architect a solution ourselves, since the tool cannot address this issue for us.The solution leverages log collection and big data analytics and its architecture has paved the way for us to build more actionable insights than if we were using the tool as-is.We transparently share our experiences, and the details of our solution and rationale, with the goal of benefiting others, and promoting further industry-academic collaboration, in this space.

Automating web application testing is a very tricky process due to inherent complexity, dynamic behavior(s) in web pages, differences in the way browsers render the same content (especially on different form factors), and so on. Yet manual testing is not a practical option, and such automation is a must in the interests of effectiveness and efficiency, because of the large number of browsers/devices that users can choose from, and also given the rapid software development cycles of today. This paper discusses our efforts at Varidesk to automate web tests against our main website - which offers many features and content, but is also a true eCommerce site where users from around the globe can purchase a very broad variety of active workspace solutions that we offer. Our solution was developed in-house, from the ground up, and leveraged and extended freely available automation and test libraries such as Selenium WebDriver and NUnit respectively. We talk about the challenges we faced and how we overcame them, as well as provide technical insights on real-world concerns such as managing test brittleness, and integrating the web tests into an existing Continuous Integration and Continuous Deployment (CI/CD) pipeline. Part of the novelty of this paper is that we are also transparent on the rationale behind our decision to build versus buy, and how we managed resources, especially in terms of cost. We also present lessons learned, and encouraged by the success that we have observed, hope that the results will be beneficial to academia and practitioners alike.

Continuous Integration (CI) and Continuous Delivery (CD) are widely considered to be best practices in software development. Studies have shown however, that adopting these practices can be challenging and there are many barriers that engineers may face, such as – overly long build times, lack of support for desired workflows, issues with configuration, etc. At Varidesk, we recently began shifting our primary web application (from a monolithic) to a micro-services-based architecture and also adapted our software development practices to aim for more effective CI/CD. In doing so, we also ran into some of the same afore-mentioned barriers. In this paper we focus on two specific challenges that we faced – long wait times for builds/releases to be queued and completed, and the lack of support for tooling, especially from a cross-cloud perspective. We then present the solutions that we came up with, which involved re-thinking DevOps as it applied to us, and re-building our own CI/CD pipelines based on DevOps-supporting approaches such as containerization, infrastructure-as-code, and orchestration. Our re-designed pipelines have led us to see speed increases, in terms of total build/release time, in the range of 330x-1110x and have enabled us to seamlessly move from a single-cloud to a multi- cloud environment, with no architectural changes to any apps.

Effective debugging is crucial to producing reliable software. Manual debugging is becoming prohibitively expensive, especially due to the growing size and complexity of programs. Given that fault localization is one of the most expensive activities in program debugging, there has been a great demand for fault localization techniques that can help guide programmers to the locations of faults. In this paper, a technique named DStar (D*) is proposed which can suggest suspicious locations for fault localization automatically without requiring any prior information on program structure or semantics. D* is evaluated across 24 programs, and is compared to 38 different fault localization techniques. Both single-fault and multi-fault programs are used. Results indicate that D* is more effective at locating faults than all the other techniques it is compared to. An empirical evaluation is also conducted to illustrate how the effectiveness of D* increases as the exponent * grows, and then levels off when the exponent * exceeds a critical value. Discussions are presented to support such observations.

SUMMARY A vast number of software fault localization techniques have been proposed recently with the growing realization that manual debugging is time‐consuming, tedious, and error‐prone, and fault localization is one of the most expensive debugging activities. Although some techniques perform better than others on a large number of data sets, they do not do so on all data sets and therefore, the actual quality of fault localization can vary considerably by using just one technique. This paper proposes the use of a consensus‐based strategy that combines the results of multiple fault localization techniques to consistently provide high quality performance, irrespective of data set. Empirical evidence based on case studies conducted on six sets of programs (seven programs of the Siemens suite, and the gzip, grep, make, space, and Ant programs) and three different fault localization techniques (Tarantula, Ochiai, and H3) suggests that the consensus‐based strategy holds merit and generally provides close to the best, if not the best, results. Empirically, we show that this is true of both single‐fault and multifault programs. Additionally, the consensus‐based strategy makes use of techniques that all operate on the same set of input data, minimizing the overhead. It is also simple to include or exclude techniques from consensus, making it an easily extensible or tractable strategy. Copyright © 2011 John Wiley & Sons, Ltd.

We propose the application of a modified radial basis function neural network in the context of software fault localization, to assist programmers in locating bugs effectively. This neural network is trained to learn the relationship between the statement coverage information of a test case and its corresponding execution result, success or failure. The trained network is then given as input a set of virtual test cases, each covering a single statement. The output of the network, for each virtual test case, is considered to be the suspiciousness of the corresponding covered statement. A statement with a higher suspiciousness has a higher likelihood of containing a bug, and thus statements can be ranked in descending order of their suspiciousness. The ranking can then be examined one by one, starting from the top, until a bug is located. Case studies on 15 different programs were conducted, and the results clearly show that our proposed technique is more effective than several other popular, state of the art fault localization techniques. Further studies investigate the robustness of the proposed technique, and illustrate how it can easily be applied to programs with multiple bugs as well.

It is becoming prohibitively expensive and time consuming, as well as tedious and error-prone, to perform debugging manually. Among the debugging activities, fault localization has been one of the most expensive, and therefore, a large number of fault-localization techniques have been proposed over the recent years. This paper presents a crosstab-based statistical technique that makes use of the coverage information of each executable statement and the execution result (success or failure) with respect to each test case to localize faults in an effective and efficient manner. A crosstab is constructed for each executable statement, and a statistic is computed to determine the suspiciousness of the corresponding statement. Statements with a higher suspiciousness are more likely to contain bugs and should be examined before those with a lower suspiciousness. Case studies are performed on both small- (the Siemens and Unix suites) and large-sized programs (space, grep, gzip, and make), and results suggest that the crosstab-based technique (CBT) is more effective (in terms of a smaller percentage of executable statements that have to be examined until the first statement containing the fault is reached) than other techniques, such as Tarantula. Further studies using the Siemens suite reveal that the proposed technique is also more effective at locating faults than other statistically oriented techniques, such as SOBER and Liblit05. Additional experiments evaluate the CBT from other perspectives, such as its efficiency in terms of time taken, its applicability to object-oriented languages (on a very large Java program: Ant), and its sensitivity to test suite size, and demonstrate its superior performance.

According to a study commissioned by the National Institute of Standards and Technology in 2002, software bugs cost the U.S. economy an estimated $59.5 billion annually, or about 0.6 percent of the nation's gross domestic product (GDP). The same study also found that more than one-third of these costs, or an estimated $22.2 billion, could be eliminated by an improved testing infrastructure. These numbers would be significantly higher if the study were conducted today.

Software fault localization techniques typically rank program components, such as statements or predicates, in descending order of their suspiciousness (likelihood of being faulty). During debugging, programmers may examine these components, starting from the top of the ranking, in order to locate faults. However, the assigned suspiciousness to each component may not always be unique, and thus some of them may be tied for the same position in the ranking. In such a scenario, the total number of components that a programmer needs to examine in order to find the faults may vary considerably. The greater the variability, the harder it is for a programmer to decide which component to examine first, and the harder it is to accurately compute the expected effectiveness of a fault localization technique. In this paper, we first conduct a case study, based on three fault localization techniques across four sets of programs, which reveals that the phenomenon of assigning the same suspiciousness to multiple components is not limited to any technique or program in particular. Thus, to reduce variability and alleviate this problem, four tie-breaking strategies are discussed and evaluated empirically in our second case study. Results indicate that the strategies can not only reduce the number of ties in the rankings, but also maintain the effectiveness of the fault localization techniques. We also propose a new metric for evaluating fault localization techniques called CScore, which takes the notion of ties into account. Finally, an additional slicing-based approach to breaking ties is discussed briefly, which aims to provide further insights into tie-breaking and stimulate further research in the area.

Poorly designed software systems are one of main causes of accidents in safety-critical systems, and thus, the importance of safety analysis for software has greatly increased over the recent years. Software safety can be improved by analyzing both its desired and undesired behaviors, and this in turn requires expressive power such that both can be modeled. However, there is a considerable gap between modeling methods for desired and undesired behaviors. Therefore, we propose a method to bridge the gap between fault trees (for undesired behavior) and UML state machine diagrams (for desired behavior). More specifically, we present rules and algorithms that facilitate the transformation of a hazard (in the context of fault trees) to a UML state machine diagram. We illustrate our proposed approach via an example on a microwave-oven system. Our proposed transformation can help engineers identify how the hazards may occur, thereby allowing them to prevent the hazard from occurring.

This paper shares the experiences and lessons learned from conducting an NSF-sponsored eightweek summer research program for ten undergraduate students from multiple universities. The focus of the research was on ¿Verification and Validation for Software Safety¿, and emphasized a strong fundamental knowledge of software safety while maintaining a close collaboration with industry. The program included lectures and special tutorials from guest speakers, field trips, review sessions, as well as posters and final presentations by the students.

This paper proposes a strategy for automatically fixing faults in a program by combining the processes of mutation and fault localization. Statements that are ranked in order of their suspiciousness of containing faults can then be mutated in the same order to produce possible fixes for the faulty program. The proposed strategy is evaluated against the seven benchmark programs of the Siemens suite and the Ant program. Results indicate that the strategy is effective at automatically suggesting fixes for faults without any human intervention.

Multiple faults in a program may interact with each other in a variety of ways. A test case that fails due to a fault may not fail when another fault is added, because the second fault may mask the failure-causing effect of the first fault. Multiple faults may also collectively cause failure on a test case that does not fail due to any single fault alone. Many studies try to perform fault localization on multi-fault programs and several of them seek to match a failed test to its causative fault. It is therefore, important to better understand the interference between faults in a multi-fault program, as an improper assumption about test case failure may lead to an incorrect matching of failed test to fault, which may in turn result in poor fault localization. This paper investigates such interference and examines if one form of interference holds more often than another, and uniformly across all conditions. Empirical studies on the Siemens suite suggest that no one form of interference holds unconditionally and that observation of failure masking is a more frequent event than observation of a new test case failure.