Context: Selenium is a widely used tool for end-to-end (E2E) web testing. However, it is often criticized for brittleness, slowness, and flakiness. In parallel, newer frameworks and Artificial Intelligence (AI) are reshaping the test automation landscape. Objectives: This study aims to investigate current practices, challenges, and emerging trends in Selenium-based test automation. Methods: We designed and executed a large-scale online survey targeting software professionals who use Selenium. The questionnaire covered technical practices, tooling, AI usage, perceived challenges, and competing tools. Results: A total of 88 complete responses were analyzed using descriptive statistics and thematic coding. The results show that Selenium remains the dominant tool for regression and functional testing, primarily using the Page Object Model (POM) pattern. The most reported challenges are related to assertability, asynchrony, and brittleness. AI tools like ChatGPT are gaining traction for test generation. Playwright is the most prominent alternative. Conclusion: While Selenium is recognized as a cornerstone in many automation workflows, its limited native test-specific features present a significant drawback. The findings indicate an increasing demand for testing-focused improvements within the Selenium ecosystem, as well as for enhanced integration with AI-driven development tools.

Web applications are essential in modern society and require thorough testing to ensure reliability and dependability. End-to-End (E2E) testing is key to evaluating both front-end and back-end components of complex web applications. Recent research has focused on improving E2E test suites by reducing maintenance costs, mitigating flakiness, and enhancing test resilience. Yet, a major challenge remains the lack of a publicly available benchmark for comparing techniques. Developing test suites for research is complex and prone to bias, highlighting the need for a shared benchmark to facilitate comparisons and experimental validation. We address this gap by providing a set of test suites designed as a benchmark for E2E testing studies. They come with scripts and automated installers for seamless deployment of the application under test in Docker containers, enhancing their usability. Our benchmark consists of 36 Java Selenium WebDriver-based E2E test suites for 8 different web applications. Each application includes multiple test suites with varying characteristics, such as the use of the Page Object pattern and advanced waiting mechanisms. Additionally, for 4 out of the 8 web applications, we provide test suites for two different versions, enabling studies on test suite evolution. Our benchmark includes 1166 test scripts, featuring 492 Page Objects and 8284 locators. To manage asynchronous behavior and reduce flakiness, the test suites incorporate 718 thread sleeps and 815 explicit waits. With a total of 51838 lines of code (LOC) this dataset can potentially be used as a resource, for example, to evaluate test automation strategies, study test suite evolution, or explore flakiness mitigation techniques.

End-to-end (E2E) testing is an approach in which an application is automatically tested through scripts that simulate the actions a user would perform. Properly managing asynchronous interactions is crucial in this approach to avoid test failures and flakiness. In the Selenium WebDriver framework, this is typically addressed by using thread sleeps (which pause the test for a fixed time) or explicit waits (function calls that pause the test execution until a specified condition is met). Explicit waits require the selection of both a condition to wait for (e.g., element visibility, element clickability) and an element on which that condition applies. Since thread sleeps are unreliable and replacing them with appropriate explicit waits is a time consuming task, in this work, we leverage a Large Language Model (LLM) to assist testers in selecting the most appropriate explicit waits. We defined a structured procedure (a series of prompts) for engaging with the LLM and validated this approach empirically on three test suites affected by asynchronous waiting issues, as well as on 12 synthetic examples. Additionally, we compared our approach with SleepReplacer, the current state-of-the-art tool for replacing thread sleeps with explicit waits in E2E web test suites. The results show that the LLM-based approach can automatically replace the majority of thread sleeps in a test suite on the first attempt, outperforming SleepReplacer.

Context: While test automation offers numerous benefits, it also introduces significant challenges. Two challenges that developers and testers face on a daily basis, particularly when using Selenium WebDriver to test web applications, are driver management (involving tasks such as version identification, download, installation, and maintenance) and management of test lifecycle phases (using specific test libraries, as for example JUnit, and inserting annotations into the code). These manual tasks make test suite development particularly tedious, error-prone, and expensive. Recently, to ease the burden on developers and testers, some Java libraries have been proposed, called WebDriverManager and Selenium-Jupiter, capable of automatically carrying out the driver management process for Selenium WebDriver and simplifying the development of test suites. These libraries appear to be very promising but until now no one has experimentally evaluated their effectiveness. Objective: To investigate the effectiveness of WebDriverManager and Selenium-Jupiter in reducing driver management times and boilerplate code. Method: We designed and conducted a family of experiments (three for WebDriverManager and two for Selenium-Jupiter) with 104 master student participants from the University of Genoa, Italy (across academic years 2021/2022 and 2022/2023) and nine professional participants. Results: Results indicate that the adoption of Selenium WebDriver with WebDriverManager significantly reduces setup time for multi-browser test suites from 33% to 50% (depending on the tester experience). Additionally, Selenium-Jupiter reduces test suite development time significantly (20% on average). Although it also decreases total code length, the reduction is relatively small compared to overall code length. Conclusion: WebDriverManager and Selenium-Jupiter can be seen as valuable solutions for enhancing testers’ productivity by shortening the time needed to develop test suites and minimizing the amount of code to write.

Web applications quality is commonly assessed by executing End-to-End (E2E) test scripts interacting with those systems as a human tester would. To avoid setting up the web application state for each test script, testers usually create test scripts that may depend on others previously executed. However, the presence of dependencies prevents parallelization, a fundamental technique for speedup the execution of large test suites. In this paper, we present Stile , a tool for parallelizing the execution of E2E web test scripts that generates and executes a set of test schedules satisfying two important constraints: (1) every schedule respects existing test dependencies, and (2) all test scripts in the test suite are executed at least once. Moreover, Stile optimizes the execution by running only once the test scripts that are shared among the schedules. We empirically evaluated Stile on eight E2E test suites by comparing the execution time of Stile both with the sequential execution and with the parallel execution based on Selenium Grid. Our results show that Stile can reduce the execution time up to 80% w.r.t. the sequential execution and up to 50% w.r.t. Grid. Moreover, Stile provides a reduction in the CPUs usage (i.e., overall CPU-time) up to 75%.

Automated testing is vital for ensuring the reliability of web applications. This paper presents a preliminary study on leveraging artificial intelligence (AI) models, specifically ChatGPT and Github Copilot, to generate test scripts for web end-to-end testing. Through experimentation, we evaluated the feasibility and effectiveness of AI language models in generating test scripts based on natural language descriptions of user interactions with web applications.

Abstract A new advancement in test automation is the use of natural language processing (NLP) to generate test cases (or test scripts) from natural language text. NLP is innovative in this context and promises of reducing test cases creation time and simplifying understanding for “non‐developer” software testers as well. Recently, many vendors have launched on the market many proposals of NLP‐based tools and testing frameworks but their superiority has never been empirically validated. This paper investigates the adoption of NLP‐based test automation in the web context with a series of case studies conducted to compare the costs of the NLP testing approach—measured in terms of test cases development and test cases evolution—with respect to more consolidated approaches, that is, programmable (or script‐based) testing and capture&replay testing. The results of our study show that NLP‐based test automation appears to be competitive for small‐ to medium‐sized test suites such as those considered in our empirical study. It minimizes the total cumulative cost (development and evolution) and does not require software testers with programming skills.

Abstract Mutation testing is an important technique able to evaluate the bug-detection effectiveness of existing software test suites. Mutation testing tools exist for several languages, e.g., Java and JavaScript, but no solutions are available for managing the mutation testing process for entire web applications, in the context of end-to-end (E2E) web testing. In this paper, we propose Mutta, a novel tool able to automate the entire mutation testing process. Mutta mutates the various server source files of the target web application, runs the E2E test suite against the mutated web applications, and finally collects the test outcomes. To evaluate Mutta, we designed a case study using the mutated versions of the target web application with the aim of comparing the effectiveness of two different approaches to E2E web testing: (1) test cases based on classical assertions and (2) test cases relying on differential testing. In detail, Mutta has been executed on two web applications, each equipped with different test suites to compare assertions with differential testing. In this scenario, Mutta generated a large number of mutants (more than 15k overall), took into account the coverage information to consider only the mutants actually executed, deployed the mutated web app, ran the entire E2E test suites (about 87k tests runs overall), and finally, it correctly saved the test suite results. Thus, results of the case study show that Mutta can be successfully employed to automate the entire mutation testing process of E2E web test suites and, therefore, can be used in practice to evaluate the effectiveness of different test suites (e.g., based on different techniques, E2E frameworks, or composed by a different number of test scripts).

Modern web applications are complex and used for tasks of primary importance, so their quality must be guaranteed at the highest levels. For this reason, testing techniques (e.g., end-to-end) are required to validate the overall behavior of web applications. One of the most popular tools for testing web applications is Selenium WebDriver. Selenium WebDriver automates the browser to mimic real user actions on the web.While Selenium has made testing easier for many Teams worldwide, it still has its share of challenges. To better understand the challenges and the corresponding solutions adopted we decided to undertake a personal opinion survey from the industry (in total with 78 highly skilled participants) with a focus on the Selenium ecosystem.The results allow understanding which challenges are consid-ered more relevant by professionals in their daily practice and which are the techniques, approaches, and tools they adopt to face them. Therefore, this study is useful to (1) practitioners interested in understanding how to solve the problems they face every day and (2) researchers interested in proposing innovative solutions to problems having a solid industrial impact.

In software engineering, observability is the ability to determine the current state of a software system based on its external outputs or signals such as metrics, logs, or traces. Web engineers rely on the web browser console as the primary tool to monitor the client-side of web applications during end-to-end tests. However, this is a manual and time-consuming task due to the different browsers available. This paper presents BrowserWatcher, an open-source browser extension providing cross-browser capabilities to observe web applications and automatically gather browser console logs in different browsers (e.g., Chrome, Firefox, or Edge). We have leveraged this extension to conduct an empirical study analyzing the browser console of the top-50 public websites manually and automatically. The results show that BrowserWatcher gathers all the well-known log categories such as console or error traces. It also reveals that each web browser additionally includes other types of logs, which differ among browsers, thus providing distinct pieces of information for the same website.

Abstract Tests, when not correctly implemented, can pass on incorrect system implementations rather than fail. In this case, they are named silent horrors or false‐negative tests. They make releasing low‐quality (buggy) versions of the software system more probable. Furthermore, faithfully implementing test specifications is crucial when they play the role of documentation, like when documenting components or services or driving legacy systems' re‐engineering. This paper presents TestWizard, a novel approach and tool for automatically assessing individual tests' quality from the point of view of their coherence to specifications. TestWizard automatically assesses the quality of each individual test case w.r.t. its specification, providing detailed reports on why a single test is a false negative, hence helping testers fix them. Thus, TestWizard can help to automate the test code review process, which is still mainly manual today. The analysis of 1012 test implementations, developed by 123 students in three experiments, shows that TestWizard is (1) by far more accurate than code review performed by multiple students, (2) slightly better than code review performed by three senior experts, and (3) always able to detect a significant percentage of false‐negative test methods (up to 21.22%).

Non-robust (fragile) test execution is a commonly reported challenge in GUI-based test automation, despite much research and several proposed solutions. A test script needs to be resilient to (minor) changes in the tested application but, at the same time, fail when detecting potential issues that require investigation. Test script fragility is a multi-faceted problem. However, one crucial challenge is how to reliably identify and locate the correct target web elements when the website evolves between releases or otherwise fail and report an issue. This article proposes and evaluates a novel approach called similarity-based web element localization (Similo), which leverages information from multiple web element locator parameters to identify a target element using a weighted similarity score. This experimental study compares Similo to a baseline approach for web element localization. To get an extensive empirical basis, we target 48 of the most popular websites on the Internet in our evaluation. Robustness is considered by counting the number of web elements found in a recent website version compared to how many of these existed in an older version. Results of the experiment show that Similo outperforms the baseline; it failed to locate the correct target web element in 91 out of 801 considered cases (i.e., 11%) compared to 214 failed cases (i.e., 27%) for the baseline approach. The time efficiency of Similo was also considered, where the average time to locate a web element was determined to be 4 milliseconds. However, since the cost of web interactions (e.g., a click) is typically on the order of hundreds of milliseconds, the additional computational demands of Similo can be considered negligible. This study presents evidence that quantifying the similarity between multiple attributes of web elements when trying to locate them, as in our proposed Similo approach, is beneficial. With acceptable efficiency, Similo gives significantly higher effectiveness (i.e., robustness) than the baseline web element localization approach.

End-to-end automated test scripts are a great way to ensure the quality of web applications, but are often perceived as expensive both during their initial development and subsequent maintenance activities. However, maintenance costs can be re-duced when test scripts adopt the Page Object (PO) pattern, a sort of web page facade exposing methods to the test scripts. In this work, we proposed ASSESSOR, a novel tool capable of reducing the effort needed for building PO-based Selenium WebDriver test suites. ASSESSOR allows to simply record the test cases, with only a few additional steps compared to Selenium IDE, and then to automatically generate PO-based WebDriver test suites. The in-depth evaluation performed with four web applications shows that ASSESSOR's adoption allows to reduce the development effort of PO-based web test scripts compared to the classic manual approach: 59% time reduction overall, corresponding to a 2.44 increment in productivity.

Abstract In the last few years, Internet of Things (IoT) systems have drastically increased their relevance in many fundamental sectors. For this reason, assuring their quality is of paramount importance, especially in safety-critical contexts. Unfortunately, few quality assurance proposals for assuring the quality of these complex systems are present in the literature. In this paper, we extended and improved our previous approach for semi-automated model-based generation of executable test scripts. Our proposal is oriented to system-level acceptance testing of IoT systems. We have implemented a prototype tool taking in input a UML model of the system under test and some additional artefacts, and producing in output a test suite that checks if the system’s behaviour is compliant with such a model. We empirically evaluated our tool employing two IoT systems: a mobile health IoT system for diabetic patients and a smart park management system part of a smart city project. Both systems involve sensors or actuators, smartphones, and a remote cloud server. Results show that the test suites generated with our tool have been able to kill 91% of the overall 260 generated mutants (i.e. artificial bugged versions of the two considered systems). Moreover, the optimisation introduced in this novel version of our prototype, based on a minimisation post-processing step, allowed to reduce the time required for executing the entire test suites (about -20/25%) with no adverse effect on the bug-detection capability.

Abstract Assuring quality of web applications is fundamental, given their relevance in the today’s world. A possible way to reach this goal is through end-to-end (E2E) testing, an approach in which a web application is automatically tested by performing the actions that a user would do. With modern web applications (for example, single-page applications), it is of great importance to properly handle asynchronous calls in the test suite. In E2E Selenium WebDriver test suites, asynchronous calls are usually managed in two ways: using thread sleeps or explicit waits. The first is easier to use, but is inefficient and can lead to instability (also called flakiness, a problem often present in test suites that makes us lose confidence in the testing phase), while the second is usually more efficient but harder to use because, if the correct kind of wait is not carefully selected, it can introduce flakiness too. To help Testers, who often opt for the first strategy, we present in this work a tool-based approach to automatically replace thread sleeps with explicit waits in an E2E Selenium WebDriver test suite without introducing new flakiness. We empirically validated our tool named SleepReplacer on four different test suites, and we found that it can correctly replace in an automatic way from 81 to 100% of thread sleeps, leading to a significant reduction of the total execution time of the test suite (i.e., from 13 to 71%).

Automated end-to-end (E2E) Web testing relying on frameworks such as Selenium Web Driver is commonly used to assess the quality of web applications. However, the resulting test scripts may require long execution times, due to their interaction with the browser GUI and backend services. To avoid repeated and costly setup of the Web application state, testers tend to build test suites whose test scripts depend on each other (i.e., one test case sets up the application state expected by another test case). In this paper we present Stile, a tool for the parallel execution of Web test scripts that ensures the compliance of all execution schedules with the dependencies among the involved test scripts, while at the same time minimizing the execution time and the computation time required for such parallel execution. Experimental results show that execution times can be approximately halved thanks to Stile.

Abstract In this paper, we present Precise SOM (Precise Service Oriented Modelling) —a novel lightweight method for integrated domain and business process modelling—which follows the service‐oriented paradigm, uses a UML profile as notation and provides detailed workflows to guide the production of the models. In our method, the UML models are precisely defined by means of a metamodel and of a set of constraints, and by restricting UML to the essential language constructs, to help modellers to avoid common mistakes and to guarantee, by construction, a good quality. Precise SOM has been validated by detailing how it can be used in various modelling tasks, some of them illustrated by a (industrial) case study.

Summary By ensuring adequate functional coverage, End‐to‐End (E2E) testing is a key enabling factor of continuous integration. This is even more true for web applications, where automated E2E testing is the only way to exercise the full stack used to create a modern application. The test code used for web testing usually relies on DOM locators, often expressed as XPath expressions, to identify the web elements and to extract the data checked in assertions. When applications evolve, the most dominant cost for the evolution of test code is due to broken locators, which fail to locate the target element in the novel versions and must be repaired. In this paper, we formulate the robust XPath locator generation problem as a graph exploration problem, instead of relying on ad‐hoc heuristics as the one implemented by the state of the art tool robula+. Our approach is based on a statistical adaptive algorithm implemented by the tool sidereal, which outperforms robula+'s heuristics in terms of robustness by learning the potential fragility of HTML properties from previous versions of the application under test. sidereal was applied to six applications and to a total of 611 locators and was compared against two baseline algorithms, robula+ and Montoto. The adoption of sidereal results in a significant reduction of the number of broken locators (respectively ‐55% and ‐70%). The time for generating such robust locators was deemed acceptable being in the order of hundredths of second.

Web crawlers can perform long running in-depth explorations of a web application, achieving high coverage of the navigational structure. However, a crawling trace cannot be easily turned into a minimal test suite that achieves the same coverage. In fact, when the crawling trace is segmented into test cases, two problems arise: (1) test cases are dependent on each other, therefore they may raise errors when executed in isolation, and (2) test cases are redundant, since the same targets are covered multiple times by different test cases. In this paper, we propose DANTE, a novel web test generator that computes the test dependencies associated with the test cases obtained from a crawling session, and uses them to eliminate redundant tests and produce executable test schedules. DANTE can effectively turn a web crawler into a test case generator that produces minimal test suites, composed only of feasible tests that contribute to achieve the final coverage. Experimental results show that DANTE, on average, (1) reduces the error rate of the test cases obtained by crawling traces from 85% to zero, (2) produces minimized test suites that are 84% smaller than the initial ones, and (3) outperforms two competing crawling-based and model-based techniques in terms of coverage and breakage rate.

Automated web testing is an appealing option, especially when continuous testing practices are adopted. However, web test cases are known to be fragile and to break easily when a web application evolves. The Page Object (PO) design pattern addresses such problem by providing a layer of indirection that decouples test cases from the internals of the web page, where web page elements are located and triggered by the web tests. However, PO development could potentially introduce an additional burden to the already strictly constrained testing activities. This paper reports an empirical investigation of costs and benefits due to the introduction of the PO pattern in web test suite development. In particular, we conducted a family of controlled experiments in which test cases were developed with and without the PO pattern. While the benefits of POs did not compensate for the extra development effort they require in the limited experimental setting of our study, results indicate that when the test suite to be developed is at least 10× larger, test development becomes more efficient with than without POs.

Assuring the quality of Internet of Things (IoT) systems is of paramount importance, and guaranteeing their reliability and compliance with the requirements is mandatory, but few attempts have been made so far. In previous works, we proposed two approaches for acceptance testing and runtime verification of IoT systems. Both works rely on a UML state machine to specify the system expected behaviour. In the acceptance testing approach, the interesting paths to exercise are identified and translated into executable test scripts. In the runtime verification approach, the relevant events during the system execution are monitored and compared against a formal specification derived from the UML state machine. In this paper, we compare the effectiveness of our two approaches, by applying them to a mobile health IoT system for the management of diabetic patients, employing over 100 mutated versions of the original system and analysing more than 1000 different executions. Results show that both approaches are effective in different ways in detecting bugs. While the acceptance testing approach is more effective to detect the bugs affecting the user interface, the runtime verification approach tracks better the subtle deviations from the system expected behaviour, in particular those concerning network issues.

E2E web test suites are prone to test dependencies due to the heterogeneous multi-tiered nature of modern web apps, which makes it difficult for developers to create isolated program states for each test case. In this paper, we present the first approach for detecting and validating test dependencies present in E2E web test suites. Our approach employs string analysis to extract an approximated set of dependencies from the test code. It then filters potential false dependencies through natural language processing of test names. Finally, it validates all dependencies, and uses a novel recovery algorithm to ensure no true dependencies are missed in the final test dependency graph. Our approach is implemented in a tool called TEDD and evaluated on the test suites of six open-source web apps. Our results show that TEDD can correctly detect and validate test dependencies up to 72% faster than the baseline with the original test ordering in which the graph contains all possible dependencies. The test dependency graphs produced by TEDD enable test execution parallelization, with a speed-up factor of up to 7×.

Existing web test generators derive test paths from a navigational model of the web application, completed with either manually or randomly generated input values. However, manual test data selection is costly, while random generation often results in infeasible input sequences, which are rejected by the application under test. Random and search-based generation can achieve the desired level of model coverage only after a large number of test execution at- tempts, each slowed down by the need to interact with the browser during test execution. In this work, we present a novel web test generation algorithm that pre-selects the most promising candidate test cases based on their diversity from previously generated tests. As such, only the test cases that explore diverse behaviours of the application are considered for in-browser execution. We have implemented our approach in a tool called DIG. Our empirical evaluation on six real-world web applications shows that DIG achieves higher coverage and fault detection rates significantly earlier than crawling-based and search-based web test generators.

Internet of things (IoT) systems are becoming ubiquitous and assuring their quality is fundamental. Unfortunately, a few proposals for testing these complex, and often safety‐critical, systems are present in the literature. The authors propose an approach for acceptance testing of IoT systems adopting graphical user interfaces as a principal way of interaction. Acceptance testing is a type of black box testing based on test scenarios, i.e. sequences of steps/actions performed by the user or the system. In their approach, test scenarios are derived from a state machine that expresses the behaviour of the system under test, and test cases are derived from them by specifying the actual data and assertions and made executable by implementing the corresponding test scripts. As a case study, they selected a mobile health IoT system for diabetes management composed of local sensors/actuators, smartphones, and a remote cloud‐based system. The effectiveness of the approach has been evaluated by measuring the capability of two test suites implemented using different localisation strategies (visual and structure‐based) in detecting mutants of the original m‐health system. Results show the effectiveness of the test suites implemented by following the proposed approach since 93% of the generated mutants have been detected.

Summary Test automation tools are widely adopted for testing complex Web applications. Three generations of tools exist: first, based on screen coordinates; second, based on DOM–based commands; and third, based on visual image recognition. In our previous work, we proposed Pesto, a tool able to migrate second‐generation Selenium WebDriver test suites towards third‐generation Sikuli ones. In this work, we extend Pesto to manage Web elements having (1) complex visual interactions and (2) multiple visual appearances. Pesto relies on aspect‐oriented programming, computer vision, and code transformations. Our new improved tool has been evaluated on two Web test suites developed by an independent tester. Experimental results show that Pesto manages and transforms correctly test suites with Web elements having complex visual interactions and multistate elements. By using Pesto, the migration of existing DOM–based test suites to the visual approach requires a low manual effort, since our approach proved to be very accurate.

Automated test scripts are used with success in many web development projects, so as to automatically verify key functionalities of the web application under test, reveal possible regressions and run a large number of tests in short time. However, the adoption of automated web testing brings advantages but also novel problems, among which the test code fragility problem. During the evolution of the web application, existing test code may easily break and testers have to correct it. In the context of automated DOM-based web testing, one of the major costs for evolving the test code is the manual effort necessary to repair broken web page element locators – lines of source code identifying the web elements (e.g. form fields and buttons) to interact with. In this work, we present Robula+, a novel algorithm able to generate robust XPath-based locators – locators that are likely to work correctly on new releases of the web application. We compared Robula+ with several state of the practice/art XPath locator generator tools/algorithms. Results show that XPath locators produced by Robula+ are by far the most robust. Indeed, Robula+ reduces the locators' fragility on average by 90% w.r.t. absolute locators and by 63% w.r.t. Selenium IDE locators. Copyright © 2016 John Wiley & Sons, Ltd.

The main reason for the fragility of web test cases is the inability of web element locators to work correctly when the web page DOM evolves. Web elements locators are used in web test cases to identify all the GUI objects to operate upon and eventually to retrieve web page content that is compared against some oracle in order to decide whether the test case has passed or not. Hence, web element locators play an extremely important role in web testing and when a web element locator gets broken developers have to spend substantial time and effort to repair it. While algorithms exist to produce robust web element locators to be used in web test scripts, no algorithm is perfect and different algorithms are exposed to different fragilities when the software evolves. Based on such observation, we propose a new type of locator, named multi-locator, which selects the best locator among a candidate set of locators produced by different algorithms. Such selection is based on a voting procedure that assigns different voting weights to different locator generation algorithms. Experimental results obtained on six web applications, for which a subsequent release was available, show that the multi-locator is more robust than the single locators (about -30% of broken locators w.r.t. the most robust kind of single locator) and that the execution overhead required by the multiple queries done with different locators is negligible (2-3% at most).

UML is a large notation offering many diagrams and a large set of constructs for each of them covering any possible modelling need. As a result its specification is a huge book, its metamodel is large, and defining/understanding its static and dynamic semantics is difficult. These features have a negative impact on the perception of the UML and lead in some cases to replace it by ad-hoc lean and simple DSLs. On the other hand, people naturally tend to downsize UML considering only a part of its constructs. Thus, the following question arises: which are the most/less used UML diagrams/constructs? We would like to answer to this question by means of a survey, trying to detect which parts of the UML are the most used. In this work, we focus our attention on the usage of the Activity Diagram constructs. To see how much a construct is used we preliminarily investigate books, ourses/tutorials, and tools covering UML. As future work, we will conduct a personal opinion survey on the same topic.

Automated testing of web applications reduces the effort needed in manual testing. Old 1st generation tools, based on screen coordinates, produce quite fragile test suites, tightly coupled with the specific screen resolution, window position and size experienced during test case recording. These tools have been replaced by a 2nd generation of tools, which offer easy selection and interaction with the web elements, based on DOM-oriented commands. Recently, a new 3rd generation of tools came up based on visual image recognition, bringing the promise of wider applicability and simplicity. A tester might ask if the migration towards such new technology is worthwhile, since the manual effort to rewrite a test suite might be overwhelming. In this paper, we propose PESTO, a tool facing the problem of the automated migration of 2nd generation test suites to the 3rd generation. PESTO determines automatically the screen position of each web element located on the DOM by a 2nd generation test case. It then calculates a screenshot image centred around the web element so as to ensure unique visual matching. Then, the entire source code of the DOM-based test suite is transformed into a visual test suite, based on such automatically extracted images and using specific visual commands.

Over the last few years, the software engineering community has proposed a number of modeling methods to represent functional requirements. Among them, use cases are recognized as an easy to use and intuitive way to capture and define such requirements. Screen mockups (also called user-interface sketches or user interface-mockups) have been proposed as a complement to use cases for improving the comprehension of functional requirements. In this article, we aim at quantifying the benefits achievable by augmenting use cases with screen mockups in the comprehension of functional requirements with respect to effectiveness, effort, and efficiency. For this purpose, we conducted a family of four controlled experiments, involving 139 participants having different profiles. The experiments involved comprehension tasks performed on the requirements documents of two desktop applications. Independently from the participants' profile, we found a statistically significant large effect of the presence of screen mockups on both comprehension effectiveness and comprehension task efficiency. No significant effect was observed on the effort to complete tasks. The main pragmatic lesson is that the screen mockups addition to use cases is able to almost double the efficiency of comprehension tasks.

This paper reports the results and some challenges we discovered during the design and execution of a pilot experiment with 21 bachelor students aimed at investigating the effect of documentation accuracy during software maintenance and evolution activities. As documentation we considered: a high level system functionality description and UML documents. Preliminary results indicate a benefit of +15% in terms of efficiency (computed as number of correct tasks per minute) when a more accurate documentation is used. The discovered challenging aspects to carefully consider in future executions of the experiment are as follows: selecting "the right" documentation artefacts, maintenance tasks and documentation versions, verifying that the subjects really used the documentation during the experiment and measuring documentation-code alignment.

This poster presents an industrial case study about test automation and test suite maintenance in the context of Web applications. The Web application under test is a Learning Content Management System (eXact learning LCMS). We analysed the costs associated with the realignment of four equivalent Selenium WebDriver test suites, implemented using the page object pattern and different methods to locate web page elements, to a subsequent release of eXact learning LCMS. In our study, the two ID-based test suites required significantly less maintenance effort than the XPath-based ones.

In this poster, we sketch our research plan about a “massive” empirical evaluation of model-driven techniques following the first two already conducted steps in that respect (an exploratory survey and a series of controlled experiments concerning maintainability). We intend to experiment UML-based model-driven techniques in several contexts (e.g., desktop and Web applications), focusing on several software characteristics (e.g., maintainability and productivity) and employing empirical methods such as controlled experiments, surveys, case studies.

SUMMARY Analyzing and understanding the evolution of large object‐oriented software systems is an important but difficult task in which matching algorithms play a fundamental role. An error‐tolerant graph matching (ETGM) algorithm can identify evolving classes that maintain a stable structure of relations (associations, inheritances, and aggregations) with other classes and thus likely constitute the backbone of the system. Therefore, to study the evolution of class diagrams, we first develop a novel ETGM algorithm, which improves the performance of our previous algorithm. Second, we describe the process of building an oracle to validate the results of our approach to solve the class diagram evolution problem. Third, we report for the new algorithm the impact of its parameters on the F‐measure summarizing precision (quantifying the exactness of the solution) and recall (quantifying the completeness of the solution). Finally, with tuned parameters, we carry out and report an extensive empirical evaluation of our algorithm using small (Rhino), medium (Azureus and ArgoUML), and large systems (Mozilla and Eclipse). We thus show that this novel algorithm is scalable, stable and has better time performance than its earlier version. Copyright © 2010 John Wiley & Sons, Ltd.

Matching algorithms play a fundamental role in many important but difficult software engineering activities, especially design evolution analysis and model comparison. We present MADMatch, a fast and scalable many-to-many approximate diagram matching approach based on an error-tolerant graph matching (ETGM) formulation. Diagrams are represented as graphs, costs are assigned to possible differences between two given graphs, and the goal is to retrieve the cheapest matching. We address the resulting optimization problem with a tabu search enhanced by the novel use of lexical and structural information. Through several case studies with different types of diagrams and tasks, we show that our generic approach obtains better results than dedicated state-of-the-art algorithms, such as AURA, PLTSDiff, or UMLDiff, on the exact same datasets used to introduce (and evaluate) these algorithms.

Background: The main claimed advantage of Model-driven engineering is improvement in productivity. However, few information is available about its actual adoption during software development and maintenance in the industry. Objective: The main aim of this work is investigating the level of maturity in the adoption of software models and of Model-driven engineering in the Italian industry. The perspective is that of software engineering researchers. Method: First, we conducted an exploratory personal opinion survey with 155 Italian software professionals. The data were collected with the help of a web-based on-line questionnaire. Then, we conducted focused interviews with three software professionals to interpret doubtful results. Results: Software modelling is a very relevant phenomenon in the Italian industry. Model-Driven techniques are used in the industry, even if (i) only for a limited extent, (ii) despite a quite generalized dissatisfaction about available tools and (iii) despite a generally low experience of the IT personnel in such techniques. Limitations: Generalization of results is limited due to the sample size. Moreover, possible self-exclusion from participants not interested in modelling could have biased the results. Conclusion: Results reinforce existing evidence regarding the usage of software modelling and (partially of) Model-driven engineering in the industry but highlight several aspects of immaturity of the Italian industry

We conducted a personal opinion survey in two rounds - years 2008 and 2011 - with the aim of investigating the level of knowledge and adoption of SOA in the Italian industry. We are also interested in understanding what is the trend of SOA (positive or negative?) and what are the methods, technologies and tools really used in the industry. The main findings of this survey are the following: (1) SOA is a relevant phenomenon in Italy, (2) Web services and RESTFul services are well-known/used and (3) orchestration languages and UDDI are little known and used. These results suggest that in Italy SOA is interpreted in a more simplistic way with respect to the current/real definition (i.e., without the concepts of orchestration/choreography and registry). Currently, the adoption of SOA is medium/low with a stable/positive trend of pervasiveness.

One of the claimed advantages of Model-driven development is the improvement in maintainability. However, few studies consider this aspect from an empirical point of view. This paper reports the results of a controlled experiment with 21 bachelor students aimed at investigating the effectiveness of Model-driven development during software maintenance and evolution activities. The tool used in the experiment is UniMod, a specific implementation of executable UML. Preliminary results indicate a relevant shortening of time with no significant impact on correctness, gained through the use of UniMod instead of conventional programming (i.e., code-centric programming).

Background: In the modeling of object oriented software systems, the UML object diagrams are recognized very useful to complement class diagrams. However, up to now, there exists only one experiment [Torchiano 2004] that investigates this concern. Aim: To confirm or contradict the findings of the original experiment, we have conducted a replication and the achieved results have been presented in this paper. Both the replication and the original experiment have been conducted to investigate whether the use of object diagrams to complement class diagrams affects the comprehension of software systems. Method: The replication has been conducted with a group of 24 graduated subjects in Computer Science of the University of Basilicata. The experiment adopts a counterbalanced design, thus ensuring that each subject work on two comprehension tasks, experimenting each time class and object diagrams together or class diagrams alone. The comprehension on each task has been assessed using a questionnaire-based approach. In particular, we have measured the comprehension level of each subject using an information retrieval based approach that allowed us to get a balance between correctness and completeness of the answers. Results: The results show that the subjects significantly benefit from the use of object diagrams in the comprehension of software systems, thus confirming and strengthening the findings of the original experiment. Conclusions: It is advisable to complement the usual class diagrams with object diagrams to increase the understandability of software systems. To raise the generalizability of the results, replications of this study are necessary especially with professional software engineers

In the context of an Italian research project, this paper reports on an on-line survey, performed with 155 software professionals, with the aim of investigating about their opinions and experiences in modeling during software development and Model-driven engineering usage. The survey focused also on used modeling languages, processes and tools. A preliminary analysis of the results confirmed that Model-driven engineering, and more in general software modeling, are very relevant phenomena. Approximately 68% of the sample use models during software development. Among then, 44% generate code starting from models and 16% execute them directly. The preferred language for modeling is UML but DSLs are used as well.

Several projects rely on one or more Heroes who are the only ones who understand and know certain critical parts of a system. Often Heroes are very useful in the economy of a project but, their presence can increase the risk of project failure if they decide to leave the project. For this reason, tools for measuring the amount of spread of knowledge within a team (i.e. the Truck factor) and identifying possible Heroes are welcomed.

In this paper, we present and discuss the results of an internal replication of a controlled experiment for assessing the effectiveness of including screen mockups when adopting Use Cases. The results of the original experiment indicate a clear improvement in terms of understandability of functional requirements when screen mockups are present with no significant impact on effort. The data analysis of the replication, conducted also in this case with undergraduate students, confirms the results of the original experiment with slight differences, thus confirming that screen mockups facilitate the understanding of requirements without influencing the effort. We also sketch here some issues related to the documentation and communication between experimenters

In order to increase stakeholders' comprehension on software requirements, Use Cases can be enhanced with screen mock-ups (i.e., GUI prototypes sketched with a special conceived graphical tool). However, the effort to write Use Cases augmented with screen mockups may increase, thus not justifying their adoption in the requirements engineering process.

Fixing bugs and implementing enhancements are very relevant activities in a typical software life cycle. They require, as a pre-requisite, the location of a portion of impacted code within a possibly large codebase. This operation can be extremely difficult and time-consuming particularly for developers not much familiar with the software. With that perspective we focus on a simple research question: is it possible to support impact analysis using the information available in software repositories, in particular code comments and version control log? We devised a simple and novel approach, based on Natural Language Processing techniques, that provides support in impact analysis. On the average the proposed approach is very selective with a 99% specificity and achieves a recall of 96% and a precision of 13.6% with respect to a manually built gold standard.

In recent years, several design notations have been proposed to model domain-specific applications or reference architectures. In particular, Conallen has proposed the UML Web Application Extension (WAE): a UML extension to model Web applications. The aim of our empirical investigation is to test whether the usage of the Conallen notation supports comprehension and maintenance activities with significant benefits, and whether such benefits depend on developers ability and experience. This paper reports and discusses the results of a series of four experiments performed in different locations and with subjects possessing different experience-namely, undergraduate students, graduate students, and research associates-and different ability levels. The experiments aim at comparing performances of subjects in comprehension tasks where they have the source code complemented either by standard UML diagrams or by diagrams stereotyped using the Conallen notation. Results indicate that, although, in general, it is not possible to observe any significant benefit associated with the usage of stereotyped diagrams, the availability of stereotypes reduces the gap between subjects with low skill or experience and highly skilled or experienced subjects. Results suggest that organizations employing developers with low experience can achieve a significant performance improvement by adopting stereotyped UML diagrams for Web applications.

Source code obfuscation is a protection mechanism widely used to limit the possibility of malicious reverse engineering or attack activities on a software system. Although several code obfuscation techniques and tools are available, little knowledge is available about the capability of obfuscation to reduce attackers' efficiency, and the contexts in which such an efficiencymay vary.

Test-driven software development tackles the problem of operationally defining the features to be implemented by means of test cases. This approach was recently ported to the early development phase, when requirements are gathered and clarified. Among the existing proposals, Fit (Framework for Integrated Testing) supports the precise specification of requirements by means of so called Fit tables, which express relevant usage scenarios in a tabular format, easily understood also by the customer. Fit tables can be turned into executable test cases through the creation of pieces of glue code, called fixtures.

Executable acceptance test case—in particular Fit (Framework for Integrated Test) tables—originally intended for the development phase proved useful in maintenance activities too. Empirical evidence suggests that Fit tables are useful in improving the comprehension of change requirements and the correctness of the maintained code. Stemming from eight experiments formerly performed by the authors, this paper presents a set of lessons learned and guidelines useful for project managers on the use of Fit tables in maintenance tasks. Specifically, the paper discusses the use of Fit tables in maintenance tasks considering a set of dimensions, ranging from maintainers’ experience to the nature of application being maintained and to the kind of benefits introduced by Fit tables. Benefits of Fit tables, such as improving the code correctness and comprehension, increase with developers experience and complex requirements but decrease with Web-based applications and when programmers work in pairs.

Ajax supports the development of rich-client Web applications, by providing primitives for the execution of asynchronous requests and for the dynamic update of the page structure and content. Often, Ajax Web applications consist of a single page whose elements are updated in response to callbacks activated asynchronously by the user or by a server message. These features give rise to new kinds of faults that are hardly revealed by existing Web testing approaches. In this paper, we propose a novel state-based testing approach, specifically designed to exercise Ajax Web applications. The document object model (DOM) of the page manipulated by the Ajax code is abstracted into a state model. Callback executions triggered by asynchronous messages received from the Web server are associated with state transitions. Test cases are derived from the state model based on the notion of semantically interacting events. We evaluate the approach on a case study in terms of fault revealing capability. We also measure the amount of manual interventions involved in constructing and refining the model required by this approach.

Abstract Web applications are becoming more and more complex and difficult to maintain. To satisfy the customer's demands, they need to be updated often and quickly. In the maintenance phase, Web site understanding is a central activity. In this phase, programmers spend a lot of time and effort in the comprehension of the internal Web site structure. Such activity is often required because the available documentation is not aligned with the implementation, if not missing at all. Reverse engineering techniques have the potential to support Web site understanding, by providing views that show the organization of a site and its navigational structure. However, representing each Web page as a node in a diagram recovered from the source code of the Web site often leads to huge and unreadable graphs. Moreover, since the level of connectivity is typically high, the edges in such graphs make the overall result even less usable. In this paper, we propose an approach to Web site understanding based on clustering of client‐side HTML pages with similar content. This approach works well with content‐oriented sites rather than application‐oriented ones and uses a crawler to download the Web pages of the target Web site. The presence of common keywords is exploited to decide when it is appropriate to group pages together. An experimental work, including 17 Web sites, validates our approach and shows that the clusters produced automatically are close to those that a human would produce for a given Web site. Copyright © 2007 John Wiley & Sons, Ltd.

This short paper reports a pilot experiment conducted with master students, in which we investigated whether fit test cases were helpful to clarify change requirements in a maintenance task.

Proponents of design notations tailored for specific application domains or reference architectures, often available in the form of UML stereotypes, motivate them by improved understandability and modifiability. However, empirical studies that tested such claims report contradictory results, where the most intuitive notations are not always the best performing ones. This indicates the possible existence of relevant influencing factors, other than the design notation itself. In this work we report the results of a family of three experiments performed at different locations and with different subjects, in which we assessed the effectiveness of UML stereotypes for Web design in support to comprehension tasks. Replications with different subjects allowed us to investigate whether subjects' ability and experience play any role in the comprehension of stereotyped diagrams. We observed different behaviors of users with different degrees of ability and experience, which suggests alternative comprehension strategies of (and tool support for) different categories of users.

While most researchers agree on the need for empirical validation of theoretical results, two main issues remain unaddressed: first, few such studies are actually performed and second, coordination among different studies is very rare. This working session aim at bringing together the researchers interested in conducting empirical studies in maintenance and evolution. The goal is to define an important topic, design a family of experiments, provide the basis to conduct a set of coordinated experiments to advance the state of empirical evidence in this area.

Abstract Web application design requires the modeling of multiple, separate concerns, such as the navigational structure, the business logic and the data persistence. To this aim, several methodologies have been conceived. One of them, the Web Application Extension (WAE), extends the UML notation by means of stereotypes and tagged values intended to capture Web‐specific concepts (e.g., the navigational structure). Although the WAE methodology is nowadays quite mature and ready for industrial adoption, the question whether it is able to actually facilitate the task of developers and maintainers has still to be empirically investigated. This paper reports and discusses the results from a controlled experiment on the benefits associated with the use of the WAE notation in the execution of comprehension tasks, carried out before maintenance. The WAE notation was compared against the use of pure unified modified language. Results indicate that the use of the WAE notation significantly improves the level of comprehension, although it does not increase the time needed to perform the comprehension task in a significant way. Copyright © 2007 John Wiley & Sons, Ltd.

Multilingual Web sites are expected to provide the same content expressed in various languages, presented according to a common style, with the same interaction facilities. To this extent, most Web developers start from a source language version of the site and produce the multilingual versions by providing translations in all supported languages. Translation pages are usually generated by replicating the HTML structure and the scripting language sections of the original pages and by translating the textual sections into the target languages. This practice exposes the site to several problems during its evolution. Updates may be not properly propagated to all translations, and unwanted divergences can be introduced over time in content, presentation and interaction. In this paper, we propose a prototype toolkit, limited to Western languages, that can help restructuring an existing static Web site, and migrating its multilingual content to a unified and consistent representation. First of all, pages are classified according to the language of their content. Then, correspondences among pages in the original language and their translations are determined. Based on the computation of the edit operations necessary to make each page consistent with its translations, the site is updated to a new version where all pages are aligned. In the last phase, a unified representation of the structure and of the multilingual content of each page is inserted into a Content Management System. This ensures a consistent future evolution of the site. The prototype toolkit has been tested on 10 existing static Web sites, with texts in Italian, English, German and Spanish. For some of the above-mentioned phases, alternative solutions have been considered and their relative advantages have been evaluated against a manually constructed gold standard. We are quite confident that with some adaptation, most of the results we obtained can be extended to any pair of Western languages. Copyright © 2005 John Wiley & Sons, Ltd.

Aspect-oriented programming (AOP) provides mechanisms for the separation of crosscutting concerns - functionalities scattered through the system and tangled with the base code. Existing systems are a natural testbed for the AOP approach since they often contain several crosscutting concerns which could not be modularized using traditional programming constructs. This paper presents an automated approach to the problem of migrating systems developed according to the object-oriented programming (OOP) paradigm into aspect-oriented programming (AOP). A simple set of six refactorings has been defined to transform OOP to AOP and has been implemented in the AOP-migrator tool, an Eclipse plug-in. A set of enabling transformations from OOP to OOP complement the initial set of refactorings. The paper presents the results of four case studies, which use the approach to migrate selected crosscutting concerns from medium-sized Java programs (in the range of 10K to 40K lines of code) into equivalent programs in AspectJ. The case study results show the feasibility of the migration and indicate the importance of the enabling transformations as a preprocessing step

This paper presents a human-guided automated approach to refactoring object oriented programs to the aspect oriented paradigm. The approach is based upon the iterative application of four steps: discovery, enabling, selection, and refactoring. After discovering potentially applicable refactorings, the enabling step transforms the code to improve refactorability. During the selection phase the particular refactorings to apply are chosen. Finally, the refactoring phase transforms the code by moving the selected code to a new aspect. This paper presents the results of an evaluation in which one of the crosscutting concerns of a 40,000 LoC program (JHotDraw) is refactored.

The current situation in the development of Web applications is reminiscent of the early days of software systems, when quality was totally dependent on individual skills and lucky choices. In fact, Web applications are typically developed without following a formalized process model: requirements are not captured and design is not considered; developers quickly move to the implementation phase and deliver the application without testing it. Not differently from more traditional software system, however, the quality of Web applications is a complex, multidimensional attribute that involves several aspects, including correctness, reliability, maintainability, usability, accessibility, performance and conformance to standards. In this context, aim of this PhD thesis was to investigate, define and apply a variety of conceptual tools, analysis, testing and restructuring techniques able to support the quality of Web applications. The goal of analysis and testing is to assess the quality of Web applications during their development and evolution; restructuring aims at improving the quality by suitably changing their structure.

Abstract The World Wide Web, initially intended as a way to publish static hypertexts on the Internet, is moving toward complex applications. Static Web sites are being gradually replaced by dynamic sites, where information is stored in databases and non‐trivial computation is performed. In such a scenario, ensuring the quality of a Web application from the user's perspective is crucial. Techniques are being investigated for the analysis and testing of Web applications for such a purpose. However, a static analysis of the source code may be extremely difficult (and, in general, infeasible) because of the presence of dynamic generation of the HTML code that is part of the application under analysis. In this paper, a dynamic analysis technique is proposed for the extraction of a Web application model through its execution. Availability of statistical data about the accesses to the pages generated by the Web application is exploited for statistical testing, based on the recovered model. Test cases can be prioritized, so as to exercise the most frequently followed paths first. Moreover, statistical reproduction of the user's navigation paths allows for an estimation of the reliability of the application. Copyright © 2004 John Wiley & Sons, Ltd.

Current practice of Web site development does not address explicitly the problems related to multilingual sites. The same information, as well as the same navigation paths, page formatting and organization, are expected to be provided by the site independently from the chosen language. This is typically ensured by adopting personal conventions on the way pages are named and on their location in the file system. Updates are then performed manually and consistency depends on the ability of the programmers not to miss any impact of the change. In this paper an extension to XHTML, called MLHTML (MultiLingual XHTML), is proposed as the target representation of a restructuring process aimed at producing a maintainable and consistent multilingual Web site. MLHTML centralizes the language dependent variants of a page in a single representation, where shared parts are not duplicated Existing sites can be migrated to MLHTML by means of the algorithms described in this paper. After classifying the pages according to their language, a page alignment technique is exploited to identify corresponding pages and to eliminate inconsistencies. Transformation into MLHTML can then be achieved automatically.

The economic relevance of Web applications increases the importance of controlling and improving their quality. Moreover, the newly available technologies for their development allow the insertion of sophisticated functions, but often leave the developers responsible for their organization and evolution. As a consequence, a high demand is emerging for methodologies and tools for the quality assurance of Web-based systems. In this paper, a UML model of Web applications is proposed for their high-level representation. Such a model is the starting point for several analyses, which can help in the assessment of the static site structure. Moreover, it drives Web application testing, in that it can be exploited to define white-box testing criteria and to semi-automatically generate the associated test cases. The proposed techniques were applied to several real-world Web applications. The results suggest that automatic support for verification and validation activities can be extremely beneficial. In fact, it guarantees that all paths in the site which satisfy a selected criterion are properly exercised before delivery. The high level of automation that is achieved in test case generation and execution increases the number of tests that are conducted and simplifies the regression checks.

Program slicing revealed a useful way to limit the search of software defects during debugging and to better understand the decomposition of the application into computations. We propose to extend the extraction of slices to Web applications, in order to produce a reduced Web application which behaves as the original one with respect to some criterion, i.e., some displayed information of interest. After presenting the theoretical implications of applying slicing to Web applications, we demonstrate its usefulness with reference to an example, derived from a survey of a set of travel agency sites. Web application slicing helps to disclose relevant information and understand the internal system structure.

Web sites are becoming important assets for several companies, which need to incorporate sophisticated technologies into complex and large Web based systems. As a consequence, methodologies and tools are required for their design, implementation and maintenance. In particular the possibility for a site to evolve so as to provide updated and accessible information is a fundamental need. Web sites are considered the object of several analyses, focused on their structure and their history, with the purpose of supporting maintenance activities. Structural information may help understanding the organization of the pages in the site, while history analysis provides indications on modifications that do not correspond to the original design or that produce undesirable effects. A tool was developed to implement the analysis of Web site structure and evolution. Its application to some examples downloaded from the Web highlights several areas where the extracted information can improve the control on the maintenance phase and provide valuable support.