Web test automation techniques often rely on crawlers to infer models of web applications for automated test generation. However, current crawlers rely on state equivalence algorithms that struggle to distinguish near-duplicate pages, often leading to redundant test cases and incomplete coverage of application functionality. In this paper, we present a model-based test generation approach that employs transformer-based Siamese neural networks (SNNs) to infer web application models more accurately. By learning similarity-based representations, SNNs capture structural and textual relationships among web pages, improving near-duplicate detection during crawling and enhancing the quality of inferred models, and thus, the effectiveness of generated test suites. Our evaluation across nine web apps shows that SNNs outperform state-of-the-art techniques in near-duplicate detection, resulting in superior web app models with an average F-1 score improvement of 56%. These enhanced models enable the generation of more effective test suites that achieve higher code coverage, with improvements ranging from 6% to 21% and averaging at 12%.

Test input generators (TIGs) are widely used to assess the robustness of Deep Learning (DL) image classifiers, yet they often produce invalid inputs that fall outside the semantic domain of the task, misleading quality assessment. While several automated validators have been proposed, there is a critical mismatch between automated and human validation criteria and, thus, automated validators are merely a proxy of domain validity, as perceived by human testers. We introduce DeepNaqqal, a supervised test input validator that learns validity directly from human-annotated labels using transfer learning on deep vision models. Our empirical study on automated validation of misclassification-inducing inputs compares DeepNaqqal against six state-of-the-art validators across three image classification tasks and multiple TIG families, using independent human assessment as ground truth. Our results show that DeepNaqqal consistently achieves the highest agreement with human judgments, while generalizing to unseen TIGs and remaining effective with substantially reduced labeled data.

Abstract Large Language Models (LLMs) are increasingly used in software development to generate functions, such as attack detectors , that implement security requirements. A key challenge is ensuring the LLMs have enough knowledge to address specific security requirements, such as information about existing attacks. For this, we propose an approach integrating Retrieval Augmented Generation (RAG) and Self-Ranking into the LLM pipeline. RAG enhances the robustness of the output by incorporating external knowledge sources, while the Self-Ranking technique, inspired by the concept of Self-Consistency, generates multiple reasoning paths and creates ranks to select the most robust detector. Our extensive empirical study targets code generated by LLMs to detect two prevalent injection attacks in web security: Cross-Site Scripting (XSS) and SQL injection (SQLi). Results show a significant improvement in detection performance while employing RAG and Self-Ranking, with an increase of up to 71%pt (on average 37%pt) and up to 43%pt (on average 6%pt) in the F2-Score for XSS and SQLi detection, respectively.

The increasing integration of IoT devices into critical infrastructure has made them prime targets for cyberattacks. Many of these devices rely on outdated or legacy software, which introduces inherent vulnerabilities and complicates firmware updates, making identifying and testing these weaknesses essential. Traditional methods typically employ black-box approaches, mutating network requests generated during device operation to craft potential attack vectors. However, these methods face limitations when dealing with encrypted or proprietary protocols. Recent tools, such as Diane and IoTFuzzer, interact with IoT devices through their mobile companion apps and use fuzzing techniques to modify request content, causing crashes in IoT device software. Although these approaches can effectively trigger software crashes, they do not generate actual exploits, as they do not precisely target or exploit specific vulnerabilities. To address these limitations, we introduce MITHRAS, the first approach that uses mobile companion apps to deliver maliciously mutated requests directly to IoT devices, explicitly targeting Remote Code Execution (RCE) vulnerabilities. MITHRAS uses Deep Reinforcement Learning to efficiently navigate the communication code within companion apps, dynamically mutating request payloads before transmission. Adapting to previous attack outcomes, MITHRAS refines its strategy, mimicking human decision-making to improve the effectiveness of exploit generation.

Cross-site scripting (XSS) poses a significant threat to web application security. While Deep Learning (DL) has shown remarkable success in detecting XSS attacks, it remains vulnerable to adversarial attacks due to the discontinuous nature of the mapping between the input (i.e., the attack) and the output (i.e., the prediction of the model whether an input is classified as XSS or benign). These adversarial attacks employ mutation-based strategies for different components of XSS attack vectors, allowing adversarial agents to iteratively select mutations to evade detection. Our work replicates a state-of-the-art XSS adversarial attack, highlighting threats to validity in the reference work and extending it towards a more effective evaluation strategy. Moreover, we introduce an XSS Oracle to mitigate these threats. The experimental results show that our approach achieves an escape rate above 96% when the threats to validity of the replicated technique are addressed.

Simulation-based testing is widely used to assess the reliability of Autonomous Driving Systems (ADS), but its effectiveness is limited by the operational design domain (ODD) conditions available in such simulators. To address this limitation, in this work, we explore the integration of generative artificial intelligence techniques with physics-based simulators to enhance ADS system-level testing. Our study evaluates the effectiveness and computational overhead of three generative strategies based on diffusion models, namely instruction-editing, inpainting, and inpainting with refinement. Specifically, we assess these techniques' capabilities to produce augmented simulator-generated images of driving scenarios representing new ODDs. We employ a novel automated detector for invalid inputs based on semantic segmentation to ensure semantic preservation and realism of the neural generated images. We then performed system-level testing to evaluate the ability of the ADS to generalize to newly synthesized ODDs. Our findings show that diffusion models help to increase the coverage of ODD for system-level ADS testing. Our automated semantic validator achieved a percentage of false positives as low as 3%, retaining the correctness and quality of the images generated for testing. Our approach successfully identified new ADS system failures before real-world testing.

Reinforcement Learning (RL) is increasingly adopted to train agents that can deal with complex sequential tasks, such as driving an autonomous vehicle or controlling a humanoid robot. Correspondingly, novel approaches are needed to ensure that RL agents have been tested adequately before going to production. Among them, mutation testing is quite promising, especially under the assumption that the injected faults (mutations) mimic the real ones. In this paper, we first describe a taxonomy of real RL faults obtained by repository mining. Then, we present the mutation operators derived from such real faults and implemented in the tool$\mu \text{PRL}$. Finally, we discuss the experimental results, showing that$\mu \text{PRL}$is effective at discriminating strong from weak test generators, hence providing useful feedback to developers about the adequacy of the generated test scenarios.

Search-based test generators are effective at producing unit tests with high coverage. However, such automatically generated tests have no meaningful test and variable names, making them hard to understand and interpret by developers. On the other hand, large language models (LLMs) can generate highly readable test cases, but they are not able to match the effectiveness of search-based generators, in terms of achieved code coverage. In this paper, we propose to combine the effectiveness of search-based generators with the readability of LLM generated tests. Our approach focuses on improving test and variable names produced by search-based tools, while keeping their semantics (i.e., their coverage) unchanged. Our evaluation on nine industrial and open source LLMs show that our readability improvement transformations are overall semantically-preserving and stable across multiple repetitions. Moreover, a human study with ten professional developers, show that our LLM-improved tests are as readable as developer-written tests, regardless of the LLM employed.

Ensuring robust robotic navigation in dynamic environments is a key challenge, as traditional testing methods often struggle to cover the full spectrum of operational requirements. This paper presents the industrial adoption of Surrealist, a simulation-based test generation framework originally for UAVs, now applied to the ANYmal quadrupedal robot for industrial inspection. Our method uses a search-based algorithm to automatically generate challenging obstacle avoidance scenarios, uncovering failures often missed by manual testing. In a pilot phase, generated test suites revealed critical weaknesses in one experimental algorithm (40.3% success rate) and served as an effective benchmark to prove the superior robustness of another (71.2% success rate). The framework was then integrated into the ANYbotics workflow for a six-month industrial evaluation, where it was used to test five proprietary algorithms. A formal survey confirmed its value, showing it enhances the development process, uncovers critical failures, provides objective benchmarks, and strengthens the overall verification pipeline.

In a recent study, Reinforcement Learning (RL) used in combination with many-objective search, has been shown to outperform alternative techniques (random search and many-objective search) for online testing of Deep Neural Network-enabled systems. The empirical evaluation of these techniques was conducted on a state-of-the-art Autonomous Driving System (ADS). This work is a replication and extension of that empirical study. Our replication shows that RL does not outperform pure random test generation in a comparison conducted under the same settings of the original study, but with no confounding factor coming from the way collisions are measured. Our extension aims at eliminating some of the possible reasons for the poor performance of RL observed in our replication: (1) the presence of reward components providing contrasting feedback to the RL agent; (2) the usage of an RL algorithm (Q-learning) which requires discretization of an intrinsically continuous state space. Results show that our new RL agent is able to converge to an effective policy that outperforms random search. Results also highlight other possible improvements, which open to further investigations on how to best leverage RL for online ADS testing.

With the increased popularity of Deep Neural Networks (DNNs), increases also the need for tools to assist developers in the DNN implementation, testing and debugging process. Several approaches have been proposed that automatically analyse and localise potential faults in DNNs under test. In this work, we evaluate and compare existing state-of-the-art fault localisation techniques, which operate based on both dynamic and static analysis of the DNN. The evaluation is performed on a benchmark consisting of both real faults obtained from bug reporting platforms and faulty models produced by a mutation tool. Our findings indicate that the usage of a single, specific ground truth (e.g. the human-defined one) for the evaluation of DNN fault localisation tools results in pretty low performance (maximum average recall of 0.33 and precision of 0.21). However, such figures increase when considering alternative, equivalent patches that exist for a given faulty DNN. The results indicate that DeepFD is the most effective tool, achieving an average recall of 0.55 and a precision of 0.37 on our benchmark.

Abstract Despite the recent developments in obstacle avoidance and other safety features, autonomous Unmanned Aerial Vehicles (UAVs) continue to face safety challenges. No previous work investigated the relationship between the behavioral uncertainty of a UAV, characterized in this work by inconsistent or erratic control signal patterns, and the unsafety of its flight. By quantifying uncertainty, it is possible to develop a predictor for unsafety, which acts as a flight supervisor. We conducted a large-scale empirical investigation of safety violations using PX4-Autopilot, an open-source UAV software platform. Our dataset of over 5,000 simulated flights, created to challenge obstacle avoidance, allowed us to explore the relation between uncertain UAV decisions and safety violations: up to 89% of unsafe UAV states exhibit significant decision uncertainty, and up to 74% of uncertain decisions lead to unsafe states. Based on these findings, we implemented Superialist (Supervising Autonomous Aerial Vehicles), a runtime uncertainty detector based on autoencoders, the state-of-the-art technology for anomaly detection. Superialist achieved high performance in detecting uncertain behaviors with up to 96% precision and 93% recall. Despite the observed performance degradation when using the same approach for predicting unsafety (up to 74% precision and 87% recall), Superialist enabled early prediction of unsafe states up to 50 seconds in advance.

Web applications quality is commonly assessed by executing End-to-End (E2E) test scripts interacting with those systems as a human tester would. To avoid setting up the web application state for each test script, testers usually create test scripts that may depend on others previously executed. However, the presence of dependencies prevents parallelization, a fundamental technique for speedup the execution of large test suites. In this paper, we present Stile , a tool for parallelizing the execution of E2E web test scripts that generates and executes a set of test schedules satisfying two important constraints: (1) every schedule respects existing test dependencies, and (2) all test scripts in the test suite are executed at least once. Moreover, Stile optimizes the execution by running only once the test scripts that are shared among the schedules. We empirically evaluated Stile on eight E2E test suites by comparing the execution time of Stile both with the sequential execution and with the parallel execution based on Selenium Grid. Our results show that Stile can reduce the execution time up to 80% w.r.t. the sequential execution and up to 50% w.r.t. Grid. Moreover, Stile provides a reduction in the CPUs usage (i.e., overall CPU-time) up to 75%.

System-level testing is fundamental to ensure the reliability of software systems. However, the execution time for system tests can be quite long, sometimes prohibitively long, especially in a regimen of continuous integration and deployment. One way to speed things up is to run the tests in parallel, provided that the execution schedule respects any dependency between tests. We present two novel approaches to detect dependencies in system-level tests, namely PFAST and MEM-FAST, which are highly parallelizable and optimistically run test schedules to exclude many dependencies when there are no failures. We evaluated our approaches both asymptotically and practically, on six Web applications and their system-level test suites, as well as on MySQL system-level tests. Our results show that, in general, PFAST is significantly faster than the state-of-the-art PRADET dependency detection algorithm, while producing parallelizable schedules that achieve a significant reduction in the overall test suite execution time.

Hyperproperties are used to define correctness requirements that involve relations between multiple program executions. This allows, for instance, to model security and concurrency requirements, which cannot be expressed by means of trace properties.

Testing autonomous driving systems (ADSs) is essential to ensure their safety. Existing testing techniques manipulate the objects of the driving environment in order to trigger a misbehavior of the ADS under test. Reinforcement learning (RL) approaches have been applied to effectively modify the dynamic objects of the environment (e.g., pedestrians and other vehicles), also known as Non-Playable Characters (NPCs). However, existing RL approaches implement centralized controllers of the environment, resulting in possibly unrealistic and even invalid behaviors of the NPCs. In this paper, we propose to model NPCs as independent and fully autonomous agents to challenge the ADS under test (i.e., the ego ADS). In the first step of our approach, we train an adversarial ADS by designing a reward function as a linear combination of two components: (1) a component that encourages it to drive well in the given driving scenario, and (2) an adversarial component, that smoothly guides the agent towards a collision with the ego ADS. In our second step, we resume training of the ego ADS, to increase its robustness towards the behaviors of the adversarial ADS. Our experiments on a highway driving scenario show that the adversarial ADS is significantly more effective at generating collisions of the ego ADS than a random baseline. Moreover, adversarial retraining induces safe behaviors of the ego ADS, preventing the adversarial ADS from colliding.

The automated real-time recognition of unexpected situations plays a crucial role in the safety of autonomous vehicles, especially in unsupported and unpredictable scenarios. This paper evaluates different Bayesian uncertainty quantification methods from the deep learning domain for the anticipa-tory testing of safety-critical misbehaviours during system-level simulation-based testing. Specifically, we compute uncertainty scores as the vehicle executes, following the intuition that high uncertainty scores are indicative of unsupported runtime conditions that can be used to distinguish safe from failure-inducing driving behaviors. In our study, we conducted an evaluation of the effectiveness and computational overhead associated with two Bayesian uncertainty quantification methods, namely Me-Dropout and Deep Ensembles, for misbehaviour avoidance. Over-all, for three benchmarks from the Udacity simulator comprising both out-of-distribution and unsafe conditions introduced via mutation testing, both methods successfully detected a high number of out-of-bounds episodes providing early warnings several seconds in advance, outperforming two state-of-the-art misbehaviour prediction methods based on autoencoders and attention maps in terms of effectiveness and efficiency. Notably, Deep Ensembles detected most misbehaviours without any false alarms and did so even when employing a relatively small number of models, making them computationally feasible for real-time detection. Our findings suggest that incorporating uncertainty quantification methods is a viable approach for building fail-safe mechanisms in deep neural network-based autonomous vehicles.

The growing adoption of Deep Learning (DL) systems in all areas of life makes the exposure and repair of errors in such systems a task of paramount importance. Existing approaches for measuring test adequacy and producing automated repair patches for such systems rely heavily on the activation values of neurons. However, there is no empirical evidence that links the behaviour of a DL model in terms of its neuron activation patterns to the type of faults present in that DL model. In this work, we perform a large-scale empirical study in which we inject artificial faults into a DL model and check whether the same types of faults lead to similar activation patterns. To analyse the patterns, we propose the notion of the spectrum of a deep neural network (DNN), which is defined as the probability distribution of the activation values of the neurons of the DNN. We perform our analysis on 6 subject systems considering 24 different fault types, each associated with a specific DL mutation operator. Our results show that we can successfully identify the fault type based on the top 3 spectra similarity matches in 75% of the cases and clusters of similar spectra have low impurity (i.e., they contain mostly one or few fault types). To demonstrate a practical application of this finding, we train different classifiers to predict the error type in a DL model from the spectra of the activation values. Our results show that the prediction accuracy can be as high as 79%, when the classifier is trained on subject-specific spectra.

Abstract Simulation-based testing represents an important step to ensure the reliability of autonomous driving software. In practice, when companies rely on third-party general-purpose simulators, either for in-house or outsourced testing, the generalizability of testing results to real autonomous vehicles is at stake. In this paper, we enhance simulation-based testing by introducing the notion ofdigital siblings—a multi-simulator approach that tests a given autonomous vehicle on multiple general-purpose simulators built with different technologies, that operate collectively as an ensemble in the testing process. We exemplify our approach on a case study focused on testing the lane-keeping component of an autonomous vehicle. We use two open-source simulators as digital siblings, and we empirically compare such a multi-simulator approach against a digital twin of a physical scaled autonomous vehicle on a large set of test cases. Our approach requires generating and running test cases for each individual simulator, in the form of sequences of road points. Then, test cases are migrated between simulators, using feature maps to characterize the exercised driving conditions. Finally, the joint predicted failure probability is computed, and a failure is reported only in cases of agreement among the siblings. Our empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin. We discuss the findings of our case study and detail how our approach can help researchers interested in automated testing of autonomous driving software.

Deep Reinforcement Learning (DRL) has received a lot of attention from the research community in recent years. As the technology moves away from game playing to practical contexts, such as autonomous vehicles and robotics, it is crucial to evaluate the quality of DRL agents. In this article, we propose a search-based approach to test such agents. Our approach, implemented in a tool called Indago , trains a classifier on failure and non-failure environment (i.e., pass) configurations resulting from the DRL training process. The classifier is used at testing time as a surrogate model for the DRL agent execution in the environment, predicting the extent to which a given environment configuration induces a failure of the DRL agent under test. The failure prediction acts as a fitness function, guiding the generation towards failure environment configurations, while saving computation time by deferring the execution of the DRL agent in the environment to those configurations that are more likely to expose failures. Experimental results show that our search-based approach finds 50% more failures of the DRL agent than state-of-the-art techniques. Moreover, such failures are, on average, 78% more diverse; similarly, the behaviors of the DRL agent induced by failure configurations are 74% more diverse.

Recent decades have seen the rise of large-scale Deep Neural Networks (DNNs) to achieve human-competitive performance in a variety of AI tasks. Often consisting of hundreds of million, if not hundreds of billion, parameters, these DNNs are too large to be deployed to or efficiently run on resource-constrained devices such as mobile phones or Internet of Things microcontrollers. Systems relying on large-scale DNNs thus have to call the corresponding model over the network, leading to substantial costs for hosting and running the large-scale remote model, costs which are often charged on a per-use basis. In this article, we propose BiSupervised , a novel architecture, where, before relying on a large remote DNN, a system attempts to make a prediction on a small-scale local model. A DNN supervisor monitors said prediction process and identifies easy inputs for which the local prediction can be trusted. For these inputs, the remote model does not have to be invoked, thus saving costs while only marginally impacting the overall system accuracy. Our architecture furthermore foresees a second supervisor to monitor the remote predictions and identify inputs for which not even these can be trusted, allowing to raise an exception or run a fallback strategy instead. We evaluate the cost savings and the ability to detect incorrectly predicted inputs on four diverse case studies: IMDb movie review sentiment classification, GitHub issue triaging, ImageNet image classification, and SQuADv2 free-text question answering. In all four case studies, we find that BiSupervised allows to reduce cost by at least 30% while maintaining similar system-level prediction performance. In two case studies (IMDb and SQuADv2), we find that BiSupervised even achieves a higher system-level accuracy, at reduced cost, compared to a remote-only model. Furthermore, measurements taken on our setup indicate a large potential of BiSupervised to reduce average prediction latency.

This is the Replicated Computational Results (RCR) Report for our TOSEM paper “Adopting Two Supervisors for Efficient Use of Large-Scale Remote Deep Neural Networks”, where we propose a novel client-server architecture allowing to leverage the high accuracy of huge neural networks running on remote servers while reducing the economical and latency costs typically coming from using such models. As part of this RCR, we provide a replication package, which allows the full replication of all our results and is specifically designed to facilitate reuse.

Testing Autonomous Driving Systems (ADSs) is crucial to ensure their reliability when navigating complex environments. ADSs may exhibit unexpected behaviours when presented, during operation, with driving scenarios containing features inadequately represented in the training dataset. To address this shift from development to operation, developers must acquire new data with the newly observed features. This data can be then utilised to fine tune the ADS, so as to reach the desired level of reliability in performing driving tasks. However, the resource-intensive nature of testing ADSs requires efficient methodologies for generating targeted and diverse tests. In this work, we introduce a novel approach, DeepAtash-LR , that incorporates a surrogate model into the focused test generation process. This integration significantly improves focused testing effectiveness and applicability in resource-intensive scenarios. Experimental results show that the integration of the surrogate model is fundamental to the success of DeepAtash-LR . Our approach was able to generate an average of up to 60× more targeted, failure-inducing inputs compared to the baseline approach. Moreover, the inputs generated by DeepAtash-LR were useful to significantly improve the quality of the original ADS through fine tuning.

Metamorphic testing is a popular approach that aims to alleviate the oracle problem in software testing. At the core of this approach are Metamorphic Relations (MRs), specifying properties that hold among multiple test inputs and corresponding outputs. Deriving MRs is mostly a manual activity, since their automated generation is a challenging and largely unexplored problem. This paper presentsGenMorph, a technique to automatically generate MRs for Java methods that involve inputs and outputs that are boolean, numerical, or ordered sequences.GenMorphuses an evolutionary algorithm to search foreffectivetest oracles, i.e., oracles that trigger no false alarms and expose software faults in the method under test. The proposed search algorithm is guided by two fitness functions that measure the number of false alarms and the number of missed faults for the generated MRs. Our results show thatGenMorphgenerates effective MRs for 18 out of 23 methods (mutation score >20%). Furthermore, it can increaseRandoop’s fault detection capability in 7 out of 23 methods, andEvosuite’s in 14 out of 23 methods. When compared with AUTOMR, a state-of-the-art MR generator,GenMorphalso outperformed its fault detection capability in 9 out of 10 methods.

Recent advances in Deep Neural Networks (DNNs) and sensor technologies are enabling autonomous driving systems (ADSs) with an ever-increasing level of autonomy. However, assessing their dependability remains a critical concern. State-of-the-art ADS testing approaches modify the controllable attributes of a simulated driving environment until the ADS misbehaves. In such approaches, environment instances in which the ADS is successful are discarded, despite the possibility that they could contain hidden driving conditions in which the ADS may misbehave. In this paper, we presentGenBo(GENerator of BOundary state pairs), a novel test generator for ADS testing.GenBomutates the driving conditions of the ego vehicle (position, velocity and orientation), collected in a failure-free environment instance, and efficiently generates challenging driving conditions at the behavior boundary (i.e., where the model starts to misbehave) in the same environment instance. We use such boundary conditions to augment the initial training dataset and retrain the DNN model under test. Our evaluation results show that the retrained model has, on average, up to 3$\times$higher success rate on a separate set of evaluation tracks with respect to the original DNN model.

Background: Most quality assessment approaches for Deep Learning (DL) focus on finding misbehaviour-inducing inputs. However, it is difficult to clearly understand the causes of misbehaviours, due to the DL software opaqueness. Recent research proposed different techniques to explain DL misbehaviours, producing input explanations either at a “low level” (raw input elements) or at a “high level” (input features). Aims: We aim to compare the similarity between different explanations and assess to what extent they are understandable. Method: We have conducted an empirical study involving 3 state-of-the-art techniques for DL explanation in 13 configurations, applied to 2 different DL tasks. We have also collected answers from 48 questionnaires submitted to SE experts. Results: Low- and high-level techniques provide dissimilar explanations for the same inputs. However, experts deemed none of the explanations as useful in 28% of the cases. Conclusion: Despite the complementarity of existing explanations, further research is needed to produce better explanations.

Testing Deep Learning (DL) based systems inherently requires large and representative test sets to evaluate whether DL systems generalise beyond their training datasets. Diverse Test Input Generators (TIGs) have been proposed to produce artificial inputs that expose issues of the DL systems by triggering misbehaviours. Unfortunately, such generated inputs may be invalid, i.e., not recognisable as part of the input domain, thus providing an unreliable quality assessment. Automated validators can ease the burden of manually checking the validity of inputs for human testers, although input validity is a concept difficult to formalise and, thus, automate. In this paper, we investigate to what extent TIGs can generate valid inputs, according to both automated and human validators. We conduct a large empirical study, involving 2 different automated validators, 220 human assessors, 5 different TIGs and 3 classification tasks. Our results show that 84% artificially generated inputs are valid, according to automated validators, but their expected label is not always preserved. Automated validators reach a good consensus with humans (78% accuracy), but still have limitations when dealing with feature-rich datasets.

With the advent of deep learning, AI components have achieved unprecedented performance on complex, human competitive tasks, such as image, video, text and audio processing. Hence, they are increasingly integrated into sophisticated software systems, some of which (e.g., autonomous vehicles) are required to deliver certified dependability warranties. In this talk, I will consider the unique features of AI based systems and of the faults possibly affecting them, in order to revise the testing fundamentals and redefine the overall goal of testing, taking a statistical view on the dependability warranties that can be actually delivered. Then, I will consider the key elements of a revised testing process for AI based systems, including the test oracle and the test input generation problems. I will also introduce the notion of runtime supervision, to deal with unexpected error conditions that may occur in the field. Finally, I will identify the future steps that are essential to close the loop from testing to operation, proposing an empirical framework that reconnects the output of testing to its original goals.

Unmanned aerial vehicles (UAVs), also known as drones, are acquiring increasing autonomy. With their commercial adoption, the problem of testing their functional and non-functional, and in particular their safety requirements has become a critical concern. Simulation-based testing represents a fundamental practice, but the testing scenarios considered in software-in-the-loop testing may not be representative of the actual scenarios experienced in the field.In this paper, we propose SURREALIST (teSting UAVs in the neighboRhood of REAl flIghtS), a novel search-based approach that analyses the logs from real UAV flights and automatically generates simulation-based test cases in the neighborhood of such real flights, thereby improving the realism and representativeness of the simulation-based tests. This is done in two steps: first, SURREALIST faithfully replicates the given UAV flight in the simulation environment, generating a simulation-based test that mirrors a pre-logged real-world behavior. Then, it smoothly manipulates the replicated flight conditions to discover slightly modified test cases that are challenging or trigger misbehaviors of the UAV under test in simulation. In our experiments, we were able to replicate a real flight accurately in the simulation environment and to expose unstable and potentially unsafe behavior in the neighborhood of a replicated flight, which even led to crashes.

As Deep Neural Networks (DNNs) are rapidly being adopted within large software systems, software developers are increasingly required to design, train, and deploy such models into the systems they develop. Consequently, testing and improving the robustness of these models have received a lot of attention lately. However, relatively little effort has been made to address the difficulties developers experience when designing and training such models: if the evaluation of a model shows poor performance after the initial training, what should the developer change? We survey and evaluate existing state-of-the-art techniques that can be used to repair model performance, using a benchmark of both real-world mistakes developers made while designing DNN models and artificial faulty models generated by mutating the model code. The empirical evaluation shows that random baseline is comparable with or sometimes outperforms existing state-of-the-art techniques. However, for larger and more complicated models, all repair techniques fail to find fixes. Our findings call for further research to develop more sophisticated techniques for Deep Learning repair.

When deployed in the operation environment, Deep Learning (DL) systems often experience the so-called development to operation (dev2op) data shift, which causes a lower prediction accuracy on field data as compared to the one measured on the test set during development. To address the dev2op shift, developers must obtain new data with the newly observed features, as these are under-represented in the train/test set, and must use them to fine tune the DL model, so as to reach the desired accuracy level. In this paper, we address the issue of acquiring new data with the specific features observed in operation, which caused a dev2op shift, by proposing DeepAtash, a novel search-based focused testing approach for DL systems. DeepAtash targets a cell in the feature space, defined as a combination of feature ranges, to generate misbehaviour-inducing inputs with predefined features. Experimental results show that DeepAtash was able to generate up to 29X more targeted, failure-inducing inputs than the baseline approach. The inputs generated by DeepAtash were useful to significantly improve the quality of the original DL systems through fine tuning not only on data with the targeted features, but quite surprisingly also on inputs drawn from the original distribution.

Abstract Offline model-level testing of autonomous driving software is much cheaper, faster, and diversified than in-field, online system-level testing. Hence, researchers have compared empirically model-level vs system-level testing using driving simulators. They reported the general usefulness of simulators at reproducing the same conditions experienced in-field, but also some inadequacy of model-level testing at exposing failures that are observable only in online mode. In this work, we replicate the reference study on model vs system-level testing of autonomous vehicles while acknowledging several assumptions that we had reconsidered. These assumptions are related to several threats to validity affecting the original study that motivated additional analysis and the development of techniques to mitigate them. Moreover, we also extend the replicated study by evaluating the original findings when considering a physical, radio-controlled autonomous vehicle. Our results show that simulator-based testing of autonomous driving systems yields predictions that are close to the ones of real-world datasets when using neural-based translation to mitigate the reality gap induced by the simulation platform. On the other hand, model-level testing failures are in line with those experienced at the system level, both in simulated and physical environments, when considering the pre-failure site, similar-looking images, and accurate labels.

Abstract Deep Neural Networks (DNNs) are becoming a crucial component of modern software systems, but they are prone to fail under conditions that are different from the ones observed during training (out-of-distribution inputs) or on inputs that are truly ambiguous, i.e., inputs that admit multiple classes with nonzero probability in their labels. Recent work proposed DNN supervisors to detect high-uncertainty inputs before their possible misclassification leads to any harm. To test and compare the capabilities of DNN supervisors, researchers proposed test generation techniques, to focus the testing effort on high-uncertainty inputs that should be recognized as anomalous by supervisors. However, existing test generators aim to produce out-of-distribution inputs. No existing model- and supervisor independent technique targets the generation of truly ambiguous test inputs, i.e., inputs that admit multiple classes according to expert human judgment. In this paper, we propose a novel way to generate ambiguous inputs to test DNN supervisors and used it to empirically compare several existing supervisor techniques. In particular, we propose AmbiGuess to generate ambiguous samples for image classification problems. AmbiGuess is based on gradient-guided sampling in the latent space of a regularized adversarial autoencoder. Moreover, we conducted what is $${-}$$ - to the best of our knowledge $${-}$$ - the most extensive comparative study of DNN supervisors, considering their capabilities to detect 4 distinct types of high-uncertainty inputs, including truly ambiguous ones. We find that the tested supervisors’ capabilities are complementary: Those best suited to detect true ambiguity perform worse on invalid, out-of-distribution and adversarial inputs and vice-versa.

Summary Modern software systems accommodate complex configurations and execution conditions that depend on the environment where the software is run. While in house testing can exercise only a fraction of such execution contexts, in vivo testing can take advantage of the execution state observed in the field to conduct further testing activities. In this paper, we present the Groucho approach to in vivo testing. Groucho can suspend the execution, run some in vivo tests, rollback the side effects introduced by such tests, and eventually resume normal execution. The approach can be transparently applied to the original application, even if only available as compiled code, and it is fully automated. Our empirical studies of the performance overhead introduced by Groucho under various configurations showed that this may be kept to a negligible level by activating in vivo testing with low probability. Our empirical studies about the effectiveness of the approach confirm previous findings on the existence of faults that are unlikely exposed in house and become easy to expose in the field. Moreover, we include the first study to quantify the coverage increase gained when in vivo testing is added to complement in house testing.

Summary Deep neural networks (DNN) are increasingly used as components of larger software systems that need to process complex data, such as images, written texts, audio/video signals. DNN predictions cannot be assumed to be always correct for several reasons, amongst which the huge input space that is dealt with, the ambiguity of some inputs data, as well as the intrinsic properties of learning algorithms, which can provide only statistical warranties. Hence, developers have to cope with some residual error probability. An architectural pattern commonly adopted to manage failure prone components is the supervisor , an additional component that can estimate the reliability of the predictions made by untrusted (e.g., DNN) components and can activate an automated healing procedure when these are likely to fail, ensuring that the deep learning‐based system (DLS) does not cause damages, despite its main functionality being suspended. In this paper, we consider DLS that implement a supervisor by means of uncertainty estimation. After overviewing the main approaches to uncertainty estimation and discussing their pros and cons, we motivate the need for a specific empirical assessment method that can deal with the experimental setting in which supervisors are used, where accuracy of the DNN matters only as long as the supervisor lets the DLS continue to operate. Then we present a large empirical study conducted to compare the alternative approaches to uncertainty estimation. We distilled a set of guidelines for developers that are useful to incorporate a supervisor based on uncertainty monitoring into a DLS.

Assessing the quality of Deep Learning (DL) systems is crucial, as they are increasingly adopted in safety-critical domains. Researchers have proposed several input generation techniques for DL systems. While such techniques can expose failures, they do not explain which features of the test inputs influenced the system’s (mis-) behaviour. DeepHyperion was the first test generator to overcome this limitation by exploring the DL systems’ feature space at large. In this article, we propose DeepHyperion-CS , a test generator for DL systems that enhances DeepHyperion by promoting the inputs that contributed more to feature space exploration during the previous search iterations. We performed an empirical study involving two different test subjects (i.e., a digit classifier and a lane-keeping system for self-driving cars). Our results proved that the contribution-based guidance implemented within DeepHyperion-CS outperforms state-of-the-art tools and significantly improves the efficiency and the effectiveness of DeepHyperion . DeepHyperion-CS exposed significantly more misbehaviours for five out of six feature combinations and was up to 65% more efficient than DeepHyperion in finding misbehaviour-inducing inputs and exploring the feature space. DeepHyperion-CS was useful for expanding the datasets used to train the DL systems, populating up to 200% more feature map cells than the original training set.

Safe deployment of self-driving cars (SDC) necessitates thorough simulated and in-field testing. Most testing techniques consider virtualized SDCs within a simulation environment, whereas less effort has been directed towards assessing whether such techniques transfer to and are effective with a physical real-world vehicle. In this paper, we shed light on the problem of generalizing testing results obtained in a driving simulator to a physical platform and provide a characterization and quantification of the sim2real gap affecting SDC testing. In our empirical study, we compare SDC testing when deployed on a physical small-scale vehicle vs its digital twin. Due to the unavailability of driving quality indicators from the physical platform, we use neural rendering to estimate them through visual odometry, hence allowing full comparability with the digital twin. Then, we investigate the transferability of behavior and failure exposure between virtual and real-world environments, targeting both unintended abnormal test data and intended adversarial examples. Our study shows that, despite the usage of a faithful digital twin, there are still critical shortcomings that contribute to the reality gap between the virtual and physical world, threatening existing testing solutions that only consider virtual SDCs. On the positive side, our results present the test configurations for which physical testing can be avoided, either because their outcome does transfer between virtual and physical environments, or because the uncertainty profiles in the simulator can help predict their outcome in the real world.

Software is constantly changing as developers add new features or make changes. This directly impacts the effectiveness of the test suite associated with that software, especially when the new modifications are in an area where no test case exists. This article addresses the issue of developing a high-quality test suite to repeatedly cover a given point in a program, with the ultimate goal of exposing faults affecting the given program point. Our approach, IFRIT, uses Deep Reinforcement Learning to generate diverse inputs while keeping a high level of reachability of the desired program point. IFRIT achieves better results than state-of-the-art and baseline tools, improving reachability, diversity and fault detection.

Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently, saving computation and labelling costs. This is particularly true for large scale, deployed systems, where inputs observed in production are recorded to serve as potential test or training data for next versions of the system. Feng et. al. propose DeepGini, a very fast and simple TIP and show that it outperforms more elaborate techniques such as neuron- and surprise coverage. In a large-scale study (4 case studies, 8 test datasets, 32’200 trained models) we verify their findings. However, we also find that other comparable or even simpler baselines from the field of uncertainty quantification, such as the predicted softmax likelihood or the entropy of the predicted softmax likelihoods perform equally well as DeepGini

Automated online recognition of unexpected conditions is an indispensable component of autonomous vehicles to ensure safety even in unknown and uncertain situations. In this paper we propose a runtime monitoring technique rooted in the attention maps computed by explainable artificial intelligence techniques. Our approach, implemented in a tool called ThirdEye, turns attention maps into confidence scores that are used to discriminate safe from unsafe driving behaviours. The intuition is that uncommon attention maps are associated with unexpected runtime conditions.

Safe handling of hazardous driving situations is a task of high practical relevance for building reliable and trustworthy cyber-physical systems such as autonomous driving systems. This task necessitates an accurate prediction system of the vehicle's confidence to prevent potentially harmful system failures on the occurrence of unpredictable conditions that make it less safe to drive. In this paper, we discuss the challenges of adapting a misbehavior predictor with knowledge mined during the execution of the main system. Then, we present a framework for the continual learning of misbehavior predictors, which records in-field behavioral data to determine what data are appropriate for adaptation. Our framework guides adaptive retraining using a novel combination of in-field confidence metric selection and reconstruction error-based weighing. We evaluate our framework to improve a misbehavior predictor from the literature on the Udacity simulator for self-driving cars. Our results show that our framework can reduce the false positive rate by a large margin and can adapt to nominal behavior drifts while maintaining the original capability to predict failures up to several seconds in advance.

The dataset available for pre-release training of a machine-learning based system is often not representative of all possible execution contexts that the system will encounter in the field. Reinforcement Learning (RL) is a prominent approach among those that support continual learning, i.e., learning continually in the field, in the post-release phase. No study has so far investigated any method to test the plasticity of RL-based systems, i.e., their capability to adapt to an execution context that may deviate from the training one. We propose an approach to test the plasticity of RL-based systems. The output of our approach is a quantification of the adaptation and anti-regression capabilities of the system, obtained by computing the adaptation frontier of the system in a changed environment. We visualize such frontier as an adaptation/anti-regression heatmap in two dimensions, or as a clustered projection when more than two dimensions are involved. In this way, we provide developers with information on the amount of changes that can be accommodated by the continual learning component of the system, which is key to decide if online, in-the-field learning can be safely enabled or not.

The state space of Android apps is huge, and its thorough exploration during testing remains a significant challenge. The best exploration strategy is highly dependent on the features of the app under test. Reinforcement Learning (RL) is a machine learning technique that learns the optimal strategy to solve a task by trial and error, guided by positive or negative reward, rather than explicit supervision. Deep RL is a recent extension of RL that takes advantage of the learning capabilities of neural networks. Such capabilities make Deep RL suitable for complex exploration spaces such as one of Android apps. However, state-of-the-art, publicly available tools only support basic, Tabular RL. We have developed ARES, a Deep RL approach for black-box testing of Android apps. Experimental results show that it achieves higher coverage and fault revelation than the baselines, including state-of-the-art tools, such as TimeMachine and Q-Testing. We also investigated the reasons behind such performance qualitatively, and we have identified the key features of Android apps that make Deep RL particularly effective on them to be the presence of chained and blocking activities. Moreover, we have developed FATE to fine-tune the hyperparameters of Deep RL algorithms on simulated apps, since it is computationally expensive to carry it out on real apps.

The race for deploying AI-enabled autonomous vehicles (AVs) on public roads is based on the promise that such self-driving cars will be as safe as or safer than human drivers. Numerous techniques have been proposed to test AVs, which however lack oracle definitions that account for the quality of driving, due to the lack of a commonly used set of metrics. Towards filling this gap, we first performed a systematic analysis of the literature concerning the assessment of the quality of driving of human drivers and extracted 126 metrics. Then, we measured the correlation between such metrics and the human perception of driving quality when AVs are driving. Lastly, we performed a study based on mutation analysis to assess whether the 26 metrics that best capture the quality of AV driving according to the human study can be used as functional oracles. Our results, targeting the Udacity platform, indicate that our automated oracles can kill a high proportion of mutants at a zero or very low false alarm rate, and therefore can be used as effective functional oracles for the quality of driving of AVs.

Automated end-to-end (E2E) Web testing relying on frameworks such as Selenium Web Driver is commonly used to assess the quality of web applications. However, the resulting test scripts may require long execution times, due to their interaction with the browser GUI and backend services. To avoid repeated and costly setup of the Web application state, testers tend to build test suites whose test scripts depend on each other (i.e., one test case sets up the application state expected by another test case). In this paper we present Stile, a tool for the parallel execution of Web test scripts that ensures the compliance of all execution schedules with the dependencies among the involved test scripts, while at the same time minimizing the execution time and the computation time required for such parallel execution. Experimental results show that execution times can be approximately halved thanks to Stile.

The degree of code coverage reached by a test suite is an important indicator of the thoroughness of testing. Most coverage tools for Android apps work at the bytecode level and provide no information to developers about which source code lines have not yet been exercised by any test case. In this paper, we present COSMO, the first fully automated Android app instrumenter publicly available that operates at the source code level in a completely transparent way, making it fully compatible with existing system level testing technologies and Android test generators. The experiments that we have conducted on a large benchmark of Android apps show that COSMO can successfully instrument most apps without altering their execution traces, introducing a small, acceptable runtime overhead.

The peculiar novelty of smart contracts is a computational model where irreversible transactions are stored in a distributed persistent data storage, namely the blockchain. The technical nature of this new type of software opens to new kinds of faults, which require specific test capabilities to be revealed. In this paper we present SOCRATES, an extensible and modular framework to automatically test smart contracts. The distinctive features of SOCRATES are: (1) a collection of composable behaviours that exercise smart contracts in the blockchain; (2) it deploys a society of bots, with the purpose of detecting defects arising from multi-user interactions, which are impossible to reveal when deploying a single bot. Our empirical investigation demonstrates that SOCRATES is able expose both known and previously unknown faults in smart contracts that are actively run in the official Ethereum blockchain. Moreover, we show that a society of multiple bots is more efficient in fault exposure than a single bot alone.

Modern software systems rely on Deep Neural Networks (DNN) when processing complex, unstructured inputs, such as images, videos, natural language texts or audio signals. Provided the intractably large size of such input spaces, the intrinsic limitations of learning algorithms and the ambiguity about the expected predictions for some of the inputs, not only there is no guarantee that DNN's predictions are always correct, but rather developers must safely assume a low, though not negligible, error probability. A fail-safe Deep Learning based System (DLS) is one equipped to handle DNN faults by means of a supervisor, capable of recognizing predictions that should not be trusted and that should activate a healing procedure bringing the DLS to a safe state.In this paper, we propose an approach to use DNN uncertainty estimators to implement such supervisor. We first discuss advantages and disadvantages of existing approaches to measure uncertainty for DNNs and propose novel metrics for the empirical assessment of the supervisor that rely on such approaches. We then describe our publicly available tool UNCERTAINTY-WIZARD, which allows transparent estimation of uncertainty for regular tf.keras DNNs. Lastly, we discuss a large-scale study conducted on four different subjects to empirically validate the approach, reporting the lessons-learned as guidance for software engineers who intend to monitor uncertainty for fail-safe execution of DLS.

Uncertainty and confidence have been shown to be useful metrics in a wide variety of techniques proposed for deep learning testing, including test data selection and system supervision. We present Uncertainty-Wizard, a tool that allows to quantify such uncertainty and confidence in artificial neural networks. It is built on top of the industry-leading TF.KERAS deep learning API and it provides a near-transparent and easy to understand interface. At the same time, it includes major performance optimizations that we benchmarked on two different machines and different configurations.

Deep Learning (DL) solutions are increasingly adopted, but how to test them remains a major open research problem. Existing and new testing techniques have been proposed for and adapted to DL systems, including mutation testing. However, no approach has investigated the possibility to simulate the effects of real DL faults by means of mutation operators. We have defined 35 DL mutation operators relying on 3 empirical studies about real faults in DL systems. We followed a systematic process to extract the mutation operators from the existing fault taxonomies, with a formal phase of conflict resolution in case of disagreement. We have implemented 24 of these DL mutation operators into DeepCrime, the first source-level pre-training mutation tool based on real DL faults. We have assessed our mutation operators to understand their characteristics: whether they produce interesting, i.e., killable but not trivial, mutations. Then, we have compared the sensitivity of our tool to the changes in the quality of test data with that of DeepMutation++, an existing post-training DL mutation tool.

Deep Learning (DL) has been successfully applied to a wide range of application domains, including safety-critical ones. Several DL testing approaches have been recently proposed in the literature but none of them aims to assess how different interpretable features of the generated inputs affect the system's behaviour.

Deep Learning (DL) components are routinely integrated into software systems that need to perform complex tasks such as image or natural language processing. The adequacy of the test data used to test such systems can be assessed by their ability to expose artificially injected faults (mutations) that simulate real DL faults.In this paper, we describe an approach to automatically generate new test inputs that can be used to augment the existing test set so that its capability to detect DL mutations increases. Our tool DeepMetis implements a search based input generation strategy. To account for the non-determinism of the training and the mutation processes, our fitness function involves multiple instances of the DL model under test. Experimental results show that DeepMetis is effective at augmenting the given test set, increasing its capability to detect mutants by 63% on average. A leave-one-out experiment shows that the augmented test set is capable of exposing unseen mutants, which simulate the occurrence of yet undetected faults.

One of the major challenges in the verification of complex industrial Cyber-Physical Systems is the difficulty of determining whether a particular system output or behaviour is correct or not, the so-called test oracle problem. Metamorphic testing alleviates the oracle problem by reasoning on the relations that are expected to hold among multiple executions of the system under test, which are known as Metamorphic Relations (MRs). However, the development of effective MRs is often challenging and requires the involvement of domain experts. In this paper, we present a case study aiming at automating this process. To this end, we implemented GAssertMRs, a tool to automatically generate MRs with genetic programming. We assess the cost-effectiveness of this tool in the context of an industrial case study from the elevation domain. Our experimental results show that in most cases GAssertMRs outperforms the other baselines, including manually generated MRs developed with the help of domain experts. We then describe the lessons learned from our experiments and we outline the future work for the adoption of this technique by industrial practitioners.

Summary By ensuring adequate functional coverage, End‐to‐End (E2E) testing is a key enabling factor of continuous integration. This is even more true for web applications, where automated E2E testing is the only way to exercise the full stack used to create a modern application. The test code used for web testing usually relies on DOM locators, often expressed as XPath expressions, to identify the web elements and to extract the data checked in assertions. When applications evolve, the most dominant cost for the evolution of test code is due to broken locators, which fail to locate the target element in the novel versions and must be repaired. In this paper, we formulate the robust XPath locator generation problem as a graph exploration problem, instead of relying on ad‐hoc heuristics as the one implemented by the state of the art tool robula+. Our approach is based on a statistical adaptive algorithm implemented by the tool sidereal, which outperforms robula+'s heuristics in terms of robustness by learning the potential fragility of HTML properties from previous versions of the application under test. sidereal was applied to six applications and to a total of 611 locators and was compared against two baseline algorithms, robula+ and Montoto. The adoption of sidereal results in a significant reduction of the number of broken locators (respectively ‐55% and ‐70%). The time for generating such robust locators was deemed acceptable being in the order of hundredths of second.

Software changes constantly, because developers add new features or modifications. This directly affects the effectiveness of the test suite associated with that software, especially when these new modifications are in a specific area that no test case covers. This article tackles the problem of generating a high-quality test suite to cover repeatedly a given point in a program, with the ultimate goal of exposing faults possibly affecting the given program point. Both search-based software testing and constraint solving offer ready, but low-quality, solutions to this: Ideally, a maximally diverse covering test set is required, whereas search and constraint solving tend to generate test sets with biased distributions. Our approach, Diversified Focused Testing (DFT), uses a search strategy inspired by GödelTest. We artificially inject parameters into the code branching conditions and use a bi-objective search algorithm to find diverse inputs by perturbing the injected parameters, while keeping the path conditions still satisfiable. Our results demonstrate that our technique, DFT, is able to cover a desired point in the code at least 90% of the time. Moreover, adding diversity improves the bug detection and the mutation killing abilities of the test suites. We show that DFT achieves better results than focused testing, symbolic execution, and random testing by achieving from 3% to 70% improvement in mutation score and up to 100% improvement in fault detection across 105 software subjects.

We propose a human-in-the-loop approach for oracle improvement and analyse whether the proposed oracle improvement process is helping developers to create better oracles. For this, we conducted two human studies with 68 participants overall: an oracle assessment study and an oracle improvement study. Our results show that developers exhibit poor performance (29 percent accuracy) when manually assessing whether an assertion oracle contains a false positive, a false negative or none of the two. This shows that automated detection of these oracle deficiencies is beneficial for the users. Our tool OASIs (Oracle ASsessment and Improvement) helps developers produce assertions with higher quality. Participants who used OASIs in the improvement study were able to achieve 33 percent of full and 67 percent of partial correctness as opposed to participants without the tool who achieved only 21 percent of full and 43 percent of partial correctness.

Deep Neural Networks (DNNs) are the core component of modern autonomous driving systems. To date, it is still unrealistic that a DNN will generalize correctly to all driving conditions. Current testing techniques consist of offline solutions that identify adversarial or corner cases for improving the training phase.

The growing application of deep neural networks in safety-critical domains makes the analysis of faults that occur in such systems of enormous importance. In this paper we introduce a large taxonomy of faults in deep learning (DL) systems. We have manually analysed 1059 artefacts gathered from GitHub commits and issues of projects that use the most popular DL frameworks (TensorFlow, Keras and PyTorch) and from related Stack Overflow posts. Structured interviews with 20 researchers and practitioners describing the problems they have encountered in their experience have enriched our taxonomy with a variety of additional faults that did not emerge from the other two sources. Our final taxonomy was validated with a survey involving an additional set of 21 developers, confirming that almost all fault categories (13/15) were experienced by at least 50% of the survey participants.

The outcome of test case execution depends on the state of the object under test. While testers can carefully choose meaningful and representative object states for test execution, it is unaffordable to cover the combinatorial space of possible object states exhaustively. An appealing option is to delegate part of the testing activities to the runtime and to execute test cases in the field whenever a new or uncommon state is observed. We have designed and developed Groucho, a framework for in-vivo testing of Java applications. Among the challenges that we faced, the most important ones are isolation of the test session from the user session and minimal performance overhead. Experimental results show that if the activation probability is kept reasonably small (e.g., $10 ^{- {4}}$), the impact of the framework is imperceptible(i.e., either statistically insignificant or with a negligible effect size).

Web crawlers can perform long running in-depth explorations of a web application, achieving high coverage of the navigational structure. However, a crawling trace cannot be easily turned into a minimal test suite that achieves the same coverage. In fact, when the crawling trace is segmented into test cases, two problems arise: (1) test cases are dependent on each other, therefore they may raise errors when executed in isolation, and (2) test cases are redundant, since the same targets are covered multiple times by different test cases. In this paper, we propose DANTE, a novel web test generator that computes the test dependencies associated with the test cases obtained from a crawling session, and uses them to eliminate redundant tests and produce executable test schedules. DANTE can effectively turn a web crawler into a test case generator that produces minimal test suites, composed only of feasible tests that contribute to achieve the final coverage. Experimental results show that DANTE, on average, (1) reduces the error rate of the test cases obtained by crawling traces from 85% to zero, (2) produces minimized test suites that are 84% smaller than the initial ones, and (3) outperforms two competing crawling-based and model-based techniques in terms of coverage and breakage rate.

The ecosystem in which mobile applications run is highly heterogeneous and configurable. All layers upon which mobile apps are built offer wide possibilities of variations, from the device and the hardware, to the operating system and middleware, up to the user preferences and settings. Testing all possible configurations exhaustively, before releasing the app, is unaffordable. As a consequence, the app may exhibit different, including faulty, behaviours when executed in the field, under specific configurations.In this paper, we describe a framework that can be instantiated to support in-vivo testing of a mobile app. The framework monitors the configuration in the field and triggers in-vivo testing when an untested configuration is recognized. Experimental results show that the overhead introduced by monitoring is unnoticeable to negligible (i.e., 0-6%) depending on the device being used (high- vs. low-end). In-vivo test execution required on average 3s: if performed upon screen lock activation, it introduces just a slight delay before locking the device.

Deep Learning (DL) is increasingly adopted to solve complex tasks such as image recognition or autonomous driving. Companies are considering the inclusion of DL components in production systems, but one of their main concerns is how to assess the quality of such systems. Mutation testing is a technique to inject artificial faults into a system, under the assumption that the capability to expose (kilt) such artificial faults translates into the capability to expose also real faults. Researchers have proposed approaches and tools (e.g., Deep-Mutation and MuNN) that make mutation testing applicable to deep learning systems. However, existing definitions of mutation killing, based on accuracy drop, do not take into account the stochastic nature of the training process (accuracy may drop even when re-training the un-mutated system). Moreover, the same mutation operator might be effective or might be trivial/impossible to kill, depending on its hyper-parameter configuration. We conducted an empirical evaluation of existing operators, showing that mutation killing requires a stochastic definition and identifying the subset of effective mutation operators together with the associated most effective configurations.

Automated web testing is an appealing option, especially when continuous testing practices are adopted. However, web test cases are known to be fragile and to break easily when a web application evolves. The Page Object (PO) design pattern addresses such problem by providing a layer of indirection that decouples test cases from the internals of the web page, where web page elements are located and triggered by the web tests. However, PO development could potentially introduce an additional burden to the already strictly constrained testing activities. This paper reports an empirical investigation of costs and benefits due to the introduction of the PO pattern in web test suite development. In particular, we conducted a family of controlled experiments in which test cases were developed with and without the PO pattern. While the benefits of POs did not compensate for the extra development effort they require in the limited experimental setting of our study, results indicate that when the test suite to be developed is at least 10× larger, test development becomes more efficient with than without POs.

With the increasing adoption of Deep Learning (DL) for critical tasks, such as autonomous driving, the evaluation of the quality of systems that rely on DL has become crucial. Once trained, DL systems produce an output for any arbitrary numeric vector provided as input, regardless of whether it is within or outside the validity domain of the system under test. Hence, the quality of such systems is determined by the intersection between their validity domain and the regions where their outputs exhibit a misbehaviour.

Assertion oracles are executable boolean expressions placed inside the program that should pass (return true) for all correct executions and fail (return false) for all incorrect executions. Because designing perfect assertion oracles is difficult, assertions often fail to distinguish between correct and incorrect executions. In other words, they are prone to false positives and false negatives. In this paper, we propose GAssert (Genetic ASSERTion improvement), the first technique to automatically improve assertion oracles. Given an assertion oracle and evidence of false positives and false negatives, GAssert implements a novel co-evolutionary algorithm that explores the space of possible assertions to identify one with fewer false positives and false negatives. Our empirical evaluation on 34 Java methods from 7 different Java code bases shows that GAssert effectively improves assertion oracles. GAssert outperforms two baselines (random and invariant-based oracle improvement), and is comparable with and in some cases even outperformed human-improved assertions.

Abstract Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

E2E web test suites are prone to test dependencies due to the heterogeneous multi-tiered nature of modern web apps, which makes it difficult for developers to create isolated program states for each test case. In this paper, we present the first approach for detecting and validating test dependencies present in E2E web test suites. Our approach employs string analysis to extract an approximated set of dependencies from the test code. It then filters potential false dependencies through natural language processing of test names. Finally, it validates all dependencies, and uses a novel recovery algorithm to ensure no true dependencies are missed in the final test dependency graph. Our approach is implemented in a tool called TEDD and evaluated on the test suites of six open-source web apps. Our results show that TEDD can correctly detect and validate test dependencies up to 72% faster than the baseline with the original test ordering in which the graph contains all possible dependencies. The test dependency graphs produced by TEDD enable test execution parallelization, with a speed-up factor of up to 7×.

Existing web test generators derive test paths from a navigational model of the web application, completed with either manually or randomly generated input values. However, manual test data selection is costly, while random generation often results in infeasible input sequences, which are rejected by the application under test. Random and search-based generation can achieve the desired level of model coverage only after a large number of test execution at- tempts, each slowed down by the need to interact with the browser during test execution. In this work, we present a novel web test generation algorithm that pre-selects the most promising candidate test cases based on their diversity from previously generated tests. As such, only the test cases that explore diverse behaviours of the application are considered for in-browser execution. We have implemented our approach in a tool called DIG. Our empirical evaluation on six real-world web applications shows that DIG achieves higher coverage and fault detection rates significantly earlier than crawling-based and search-based web test generators.

The oracle problem remains one of the key challenges in software testing, for which little automated support has been developed so far. We introduce OASIs, a search-based tool for Java that assists testers in oracle assessment and improvement. It does so by combining test case generation to reveal false positives and mutation testing to reveal false negatives. In this work, we describe how OASIs works, provide details of its implementation, and explain how it can be used in an iterative oracle improvement process with a human in the loop. Finally, we present a summary of previous empirical evaluation showing that the fault detection rate of the oracles after improvement using OASIs increases, on average, by 48.6%.

Several criteria have been proposed over the years for measuring test suite adequacy. Each criterion can be converted into a specific objective function to optimize with search-based techniques in an attempt to generate test suites achieving the highest possible coverage for that criterion. Recent work has tried to optimize for multiple-criteria at once by constructing a single objective function obtained as a weighted sum of the objective functions of the respective criteria. However, this solution suffers the problem of sum scalarization, i.e., differences along the various dimensions being optimized get lost when such dimensions are projected into a single value. Recent advances in SBST formulated coverage as a many-objective optimization problem rather than applying sum scalarization. Starting from this formulation, in this work, we apply many-objective test generation that handles multiple adequacy criteria simultaneously. To scale the approach to the big number of objectives to be optimized at the same time, we adopt an incremental strategy, where only coverage targets in the control dependency frontier are considered until the frontier is expanded by covering a previously uncovered target.

Summary Test automation tools are widely adopted for testing complex Web applications. Three generations of tools exist: first, based on screen coordinates; second, based on DOM–based commands; and third, based on visual image recognition. In our previous work, we proposed Pesto, a tool able to migrate second‐generation Selenium WebDriver test suites towards third‐generation Sikuli ones. In this work, we extend Pesto to manage Web elements having (1) complex visual interactions and (2) multiple visual appearances. Pesto relies on aspect‐oriented programming, computer vision, and code transformations. Our new improved tool has been evaluated on two Web test suites developed by an independent tester. Experimental results show that Pesto manages and transforms correctly test suites with Web elements having complex visual interactions and multistate elements. By using Pesto, the migration of existing DOM–based test suites to the visual approach requires a low manual effort, since our approach proved to be very accurate.

The test case generation is intrinsically a multi-objective problem, since the goal is covering multiple test targets (e.g., branches). Existing search-based approaches either consider one target at a time or aggregate all targets into a single fitness function (whole-suite approach). Multi and many-objective optimisation algorithms (MOAs) have never been applied to this problem, because existing algorithms do not scale to the number of coverage objectives that are typically found in real-world software. In addition, the final goal for MOAs is to find alternative trade-off solutions in the objective space, while in test generation the interesting solutions are only those test cases covering one or more uncovered targets. In this paper, we present Dynamic Many-Objective Sorting Algorithm (DynaMOSA), a novel many-objective solver specifically designed to address the test case generation problem in the context of coverage testing. DynaMOSA extends our previous many-objective technique Many-Objective Sorting Algorithm (MOSA) with dynamic selection of the coverage targets based on the control dependency hierarchy. Such extension makes the approach more effective and efficient in case of limited search budget. We carried out an empirical study on 346 Java classes using three coverage criteria (i.e., statement, branch, and strong mutation coverage) to assess the performance of DynaMOSA with respect to the whole-suite approach (WS), its archive-based variant (WSA) and MOSA. The results show that DynaMOSA outperforms WSA in 28 percent of the classes for branch coverage (+8 percent more coverage on average) and in 27 percent of the classes for mutation coverage (+11 percent more killed mutants on average). It outperforms WS in 51 percent of the classes for statement coverage, leading to +11 percent more coverage on average. Moreover, DynaMOSA outperforms its predecessor MOSA for all the three coverage criteria in 19 percent of the classes with +8 percent more code coverage on average.

Code protections aim at blocking (or at least delaying) reverse engineering and tampering attacks to critical assets within programs. Knowing the way hackers understand protected code and perform attacks is important to achieve a stronger protection of the software assets, based on realistic assumptions about the hackers' behaviour. However, building such knowledge is difficult because hackers can hardly be involved in controlled experiments and empirical studies. The FP7 European project Aspire has given the authors of this paper the unique opportunity to have access to the professional penetration testers employed by the three industrial partners. In particular, we have been able to perform a qualitative analysis of three reports of professional penetration test performed on protected industrial code. Our qualitative analysis of the reports consists of open coding, carried out by 7 annotators and resulting in 459 annotations, followed by concept extraction and model inference. We identified the main activities: understanding, building attack, choosing and customizing tools, and working around or defeating protections. We built a model of how such activities take place. We used such models to identify a set of research directions for the creation of stronger code protections.

Software engineers often use record/replay tools to enable the automated testing of web applications. Tests created in this manner can then be used to regression test new versions of the web applications as they evolve. Web application tests recorded by record/replay tools, however, can be quite brittle, they can easily break as applications change. For this reason, researchers have begun to seek approaches for automatically repairing record/replay tests. To date, however, there have been no comprehensive attempts to characterize the causes of breakagesin record/replay tests for web applications. In this work, wepresent a taxonomy classifying the ways in which record/replay tests for web applications break, based on an analysis of 453 versions of popular web applications for which 1065 individual test breakages were recognized. The resulting taxonomy can help direct researchers in their attempts to repair such tests. It can also help practitioners by suggesting best practices when creating tests or modifying programs, and can help researchers with other tasks such as test robustness analysis and IDE design.

We introduce a technique for assessing and improving test oracles by reducing the incidence of both false positives and false negatives. We prove that our approach can always result in an increase in the mutual information between the actual and perfect oracles. Our technique combines test case generation to reveal false positives and mutation testing to reveal false negatives. We applied the decision support tool that implements our oracle improvement technique to five real-world subjects. The experimental results show that the fault detection rate of the oracles after improvement increases, on average, by 48.6% (86% over the implicit oracle). Three actual, exposed faults in the studied systems were subsequently confirmed and fixed by the developers.

Obfuscation techniques are a general category of software protections widely adopted to prevent malicious tampering of the code by making applications more difficult to understand and thus harder to modify. Obfuscation techniques are divided in code and data obfuscation, depending on the protected asset. While preliminary empirical studies have been conducted to determine the impact of code obfuscation, our work aims at assessing the effectiveness and efficiency in preventing attacks of a specific data obfuscation technique - VarMerge. We conducted an experiment with student participants performing two attack tasks on clear and obfuscated versions of two applications written in C. The experiment showed a significant effect of data obfuscation on both the time required to complete and the successful attack efficiency. An application with VarMerge reduces by six times the number of successful attacks per unit of time. This outcome provides a practical clue that can be used when applying software protections based on data obfuscation.

Automated test scripts are used with success in many web development projects, so as to automatically verify key functionalities of the web application under test, reveal possible regressions and run a large number of tests in short time. However, the adoption of automated web testing brings advantages but also novel problems, among which the test code fragility problem. During the evolution of the web application, existing test code may easily break and testers have to correct it. In the context of automated DOM-based web testing, one of the major costs for evolving the test code is the manual effort necessary to repair broken web page element locators – lines of source code identifying the web elements (e.g. form fields and buttons) to interact with. In this work, we present Robula+, a novel algorithm able to generate robust XPath-based locators – locators that are likely to work correctly on new releases of the web application. We compared Robula+ with several state of the practice/art XPath locator generator tools/algorithms. Results show that XPath locators produced by Robula+ are by far the most robust. Indeed, Robula+ reduces the locators' fragility on average by 90% w.r.t. absolute locators and by 63% w.r.t. Selenium IDE locators. Copyright © 2016 John Wiley & Sons, Ltd.

The main reason for the fragility of web test cases is the inability of web element locators to work correctly when the web page DOM evolves. Web elements locators are used in web test cases to identify all the GUI objects to operate upon and eventually to retrieve web page content that is compared against some oracle in order to decide whether the test case has passed or not. Hence, web element locators play an extremely important role in web testing and when a web element locator gets broken developers have to spend substantial time and effort to repair it. While algorithms exist to produce robust web element locators to be used in web test scripts, no algorithm is perfect and different algorithms are exposed to different fragilities when the software evolves. Based on such observation, we propose a new type of locator, named multi-locator, which selects the best locator among a candidate set of locators produced by different algorithms. Such selection is based on a voting procedure that assigns different voting weights to different locator generation algorithms. Experimental results obtained on six web applications, for which a subsequent release was available, show that the multi-locator is more robust than the single locators (about -30% of broken locators w.r.t. the most robust kind of single locator) and that the execution overhead required by the multiple queries done with different locators is negligible (2-3% at most).

Test data generation has been extensively investigated as a search problem, where the search goal is to maximize the number of covered program elements (e.g., branches). Recently, the whole suite approach, which combines the fitness functions of single branches into an aggregate, test suite-level fitness, has been demonstrated to be superior to the traditional single-branch at a time approach. In this paper, we propose to consider branch coverage directly as a many-objective optimization problem, instead of aggregating multiple objectives into a single value, as in the whole suite approach. Since programs may have hundreds of branches (objectives), traditional many-objective algorithms that are designed for numerical optimization problems with less than 15 objectives are not applicable. Hence, we introduce a novel highly scalable many-objective genetic algorithm, called MOSA (Many-Objective Sorting Algorithm), suitably defined for the many- objective branch coverage problem. Results achieved on 64 Java classes indicate that the proposed many-objective algorithm is significantly more effective and more efficient than the whole suite approach. In particular, effectiveness (coverage) was significantly improved in 66% of the subjects and efficiency (search budget consumed) was improved in 62% of the subjects on which effectiveness remains the same.

Several techniques and tools have been proposed for the automatic generation of test cases. Usually, these tools are evaluated in terms of fault-revealing or coverage capability, but their impact on the manual debugging activity is not considered. The question is whether automatically generated test cases are equally effective in supporting debugging as manually written tests. We conducted a family of three experiments (five replications) with humans (in total, 55 subjects) to assess whether the features of automatically generated test cases, which make them less readable and understandable (e.g., unclear test scenarios, meaningless identifiers), have an impact on the effectiveness and efficiency of debugging. The first two experiments compare different test case generation tools (Randoop vs. EvoSuite). The third experiment investigates the role of code identifiers in test cases (obfuscated vs. original identifiers), since a major difference between manual and automatically generated test cases is that the latter contain meaningless (obfuscated) identifiers. We show that automatically generated test cases are as useful for debugging as manual test cases. Furthermore, we find that, for less experienced developers, automatic tests are more useful on average due to their lower static and dynamic complexity.

Programs often run under strict usage conditions (e.g., license restrictions) that could be broken in case of code tampering. Possible attacks include malicious reverse engineering, tampering using static, dynamic and hybrid techniques, on standard devices as well as in labs with additional special purpose hardware equipment. ASPIRE (http://www.aspire-fp7.eu) is a European FP7 research project devoted to the elaboration of novel techniques to mitigate and prevent attacks to code integrity, to code/data confidentiality and to code lifting. This paper presents the ongoing activity to define a set of metrics aimed at quantifying the effect on code of the ASPIRE protections. The metrics have been conceived based on a measurement framework, which prescribes the identification of the relevant code features to consider and of their relationships with attacks and protections.

Models - in particular finite state machine models - provide an invaluable source of information for the derivation of effective test cases. However, models usually approximate part of the program semantics and capture only some of the relevant dependencies and constraints. As a consequence, some of the test cases that are derived from models are infeasible.

To isolate and fix failures that occur in the field, after deployment, developers must be able to reproduce and investigate such failures in-house. In practice, however, bug reports rarely provide enough information to recreate field failures, thus making in-house debugging an arduous task. This task becomes even more challenging for programs whose input must adhere to a formal specification, such as a grammar. To help developers address this issue, we propose an approach for automatically generating inputs that recreate field failures in-house. Given a faulty program and a field failure for this program, our approach exploits the potential of grammar-guided genetic programming to iteratively find legal inputs that can trigger the observed failure using a limited amount of runtime data collected in the field. When applied to 11 failures of 5 real-world programs, our approach was able to reproduce all but one of the failures while imposing a limited amount of overhead.

Automated testing of web applications reduces the effort needed in manual testing. Old 1st generation tools, based on screen coordinates, produce quite fragile test suites, tightly coupled with the specific screen resolution, window position and size experienced during test case recording. These tools have been replaced by a 2nd generation of tools, which offer easy selection and interaction with the web elements, based on DOM-oriented commands. Recently, a new 3rd generation of tools came up based on visual image recognition, bringing the promise of wider applicability and simplicity. A tester might ask if the migration towards such new technology is worthwhile, since the manual effort to rewrite a test suite might be overwhelming. In this paper, we propose PESTO, a tool facing the problem of the automated migration of 2nd generation test suites to the 3rd generation. PESTO determines automatically the screen position of each web element located on the DOM by a 2nd generation test case. It then calculates a screenshot image centred around the web element so as to ensure unique visual matching. Then, the entire source code of the DOM-based test suite is transformed into a visual test suite, based on such automatically extracted images and using specific visual commands.

Software components are usually redundant, since their interface offers different operations that are equivalent in their functional behavior. Several reliability techniques exploit this redundancy to either detect or tolerate faults in software. Metamorphic testing, for instance, executes pairs of sequences of operations that are expected to produce equivalent results, and identifies faults in case of mismatching outcomes. Some popular fault tolerance and self-healing techniques execute redundant operations in an attempt to avoid failures at runtime. The common assumption of these techniques, though, is that such redundancy is known a priori. This means that the set of operations that are supposed to be equivalent in a given component should be available in the specifications. Unfortunately, inferring this information manually can be expensive and error prone. This paper proposes a search-based technique to synthesize sequences of method invocations that are equivalent to a target method within a finite set of execution scenarios. The experimental results obtained on 47 methods from 7 classes show that the proposed approach correctly identifies equivalent method sequences in the majority of the cases where redundancy was known to exist, with very few false positives.

ABSTRACT Program comprehension is a prerequisite to any maintenance and evolution task. In particular, when performing feature location, developers perform program comprehension by abstracting software features and identifying the links between high‐level abstractions (features) and program elements. We present Segment Concept AssigNer (SCAN), an approach to support developers in feature location. SCAN uses a search‐based approach to split execution traces into cohesive segments. Then, it labels the segments with relevant keywords and, finally, uses formal concept analysis to identify relations among segments. In a first study, we evaluate the performances of SCAN on six Java programs by 31 participants. We report an average precision of 69% and a recall of 63% when comparing the manual and automatic labels and a precision of 63% regarding the relations among segments identified by SCAN. After that, we evaluate the usefulness of SCAN for the purpose of feature location on two Java programs. We provide evidence that SCAN (i) identifies 69% of the gold set methods and (ii) is effective in reducing the quantity of information that developers must process to locate features—reducing the number of methods to understand by an average of 43% compared to the entire execution traces. Copyright © 2014 John Wiley & Sons, Ltd.

The effectiveness of evolutionary test case generation based on Genetic Algorithms (GAs) can be seriously impacted by genetic drift, a phenomenon that inhibits the ability of such algorithms to effectively diversify the search and look for alternative potential solutions. In such cases, the search becomes dominated by a small set of similar individuals that lead GAs to converge to a sub-optimal solution and to stagnate, without reaching the desired objective. This problem is particularly common for hard-to-cover program branches, associated with an extremely large solution space. In this paper, we propose an approach to solve this problem by integrating a mechanism for orthogonal exploration of the search space into standard GA. The diversity in the population is enriched by adding individuals in orthogonal directions, hence providing a more effective exploration of the solution space. To the best of our knowledge, no prior work has addressed explicitly the issue of evolution direction based diversification in the context of evolutionary testing. Results achieved on 17 Java classes indicate that the proposed enhancements make GA much more effective and efficient in automating the testing process. In particular, effectiveness (coverage) was significantly improved in 47% of the subjects and efficiency (search budget consumed) was improved in 85% of the subjects on which effectiveness remains the same.

Reproducing field failures in-house, a step developers must perform when assigned a bug report, is an arduous task. In most cases, developers must be able to reproduce a reported failure using only a stack trace and/or some informal description of the failure. The problem becomes even harder for the large class of programs whose input is highly structured and strictly specified by a grammar. To address this problem, we present SBFR, a search-based failure-reproduction technique for programs with structured input. SBFR formulates failure reproduction as a search problem. Starting from a reported failure and a limited amount of dynamic information about the failure, SBFR exploits the potential of genetic programming to iteratively find legal inputs that can trigger the failure.

Even for small programs, the input space is huge - often unbounded. Partition testing divides the input space into disjoint equivalence classes and combinatorial testing selects a subset of all possible input class combinations, according to criteria such as pairwise coverage. The down side of this approach is that the partitioning of the input space into equivalence classes (input classification) is done manually. It is expensive and requires deep domain and implementation understanding. In this paper, we propose a novel approach to classify test inputs and their dependencies automatically. Firstly, random (or automatically generated) input vectors are sent to the system under test (SUT). For each input vector, an observed “hit vector” is produced by monitoring the execution of the SUT. Secondly, hit vectors are grouped into clusters using machine learning. Each cluster contains similar hit vectors, i.e., similar behaviors, and from them we obtain corresponding clusters of input vectors. Input classes are then extracted for each input parameter straightforwardly. Our experiments with a number of subjects show good results as the automatically generated classifications are the same or very close to the expected ones.

Software testing is an effective, yet expensive, method to improve software quality. Test automation, a potential way to reduce testing cost, has received enormous research attention recently, but the so-called “oracle problem” (how to decide the PASS/FAIL outcome of a test execution) is still a major obstacle to such cost reduction. We have extensively investigated state-of-the-art works that contribute to address this problem, from areas such as specification mining and model inference. In this paper, we compare three types of automated oracles: Data invariants, Temporal invariants, and Finite State Automata. More specifically, we study the training cost and the false positive rate; we evaluate also their fault detection capability. Seven medium to large, industrial application subjects and real faults have been used in our empirical investigation.

The IEEE International Conference on Software Maintenance (ICSM) is the premiere international venue in software maintenance and evolution, where participants from academia, government, and industry meet and share ideas and experiences for solving critical software maintenance problems. In 2011, the 27th edition of ICSM was held at the Williamsburg Lodge, in the heart of Williamsburg, VA, USA. Williamsburg, once the capital of England's oldest, wealthiest, and most populous North American colony, is one of the most important American historical landmarks. ICSM 2011 was held from September 25 to 30, 2011. The conference was preceded by two co-located events, the 11th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) and the International Workshop on the Maintenance and Evolution of Service-Oriented and Cloud-Based Systems (MESOCA). It was followed by two other co-located events, the 13th IEEE International Symposium on Web Systems Evolution (WSE) and the 6th IEEE International Workshop on Visualizing Software for Understanding and Analysis (VISSOFT). ICSM 2011 attracted 127 submissions, each of which was reviewed by at least three members of the ICSM Program Committee. Following a week and a half of online discussion supported by EasyChair, 36 papers were selected for publication and presentation in the Technical Program of the conference. Consensus was aimed for and reached in almost all cases. The topics of accepted papers ranged from Reverse Engineering and Program Comprehension to Impact Analysis, Traceability, Migration and Evolution, Refactoring, Software Clones, Linguistic Analysis, and Regression Testing. The conference program included the ICSM Doctoral Symposium, Tutorials, Tool Demonstrations, Posters, the Industry track, and the Early Research Achievements track. This special section of the Journal of Software: Evolution and Process contains extended versions of the papers selected by the ICSM 2011 Program Committee among the best papers presented at the conference. These extended versions have additionally undergone the rigorous JSEP journal review process: they were reviewed by three anonymous referees under the supervision and coordination of the guest editors. We are proud to present you the three excellent papers that are the results of this process. Paper 1, ‘An empirical study of faults in late propagation clone genealogies’, by Liliane Barbour, Foutse Khomh, and Ying Zou, investigates the history of clone evolution over time, focusing on the phenomenon of late propagation. Late propagation occurs whenever one of two clones diverges from the other, as a result of code evolution, while later such changes are reconciled, so as to remove the divergence. The authors have defined eight types of late propagation, and they have evaluated empirically the relationship between fault proneness and late propagation of various type. They have also investigated whether reconciling changes are fault-fixing changes. This paper contributes to our knowledge of the risks associated with clones and with their evolution over time. Paper 2, ‘Evaluating test-to-code traceability recovery methods through controlled experiments’, by Abdallah Qusef, Gabriele Bavota, Rocco Oliveto, Andrea De Lucia, and David Binkley, presents a combination of two controlled experiments comparing the effectiveness of four different approaches to recovering traceability links between test cases and code. Traceability addresses the problem of identifying links between software artifacts of different types, and test-to-code traceability addresses the issue of identifying precisely which sections of code are actually tested by a given test case. In the experiments presented in this paper, the effectiveness of the four approaches on three large Java systems is compared using a manual consensus of three PhD student programmers as a reference answer. Both accuracy and practical usefulness to programmers are explored, demonstrating that the authors' own SCOTCH method can be significantly more effective than existing techniques. Paper 3, ‘How developers perform feature location tasks: a human-centric and process-oriented exploratory study’, by Jinshui Wang, Xin Peng, Zhenchang Xing, and Wenyun Zhao, describes an empirical study of feature location in code by software developers. Feature location addresses the problem of identifying which parts of the source code of a system are concerned with implementing a specific business or technical concern. Automation in feature location has been a popular topic in the software maintenance research community for some time. In this paper, the authors choose instead to seek insight from how programmers actually perform the task in practice, concluding that the process can be understood at three levels of granularity, each of which is influenced by a number of external factors. The empirical insights reported here can be used to inform future work in supporting and automating this important aspect of software maintenance. We wish to thank the authors for having contributed to this special section, the ICSM 2011 Program Committee for their indications on which papers to include in the special section, and the journal referees, for their detailed and constructive comments, which greatly helped the authors to improve their papers. James R. Cordy is Professor and past Director of the School of Computing at Queen's University at Kingston, Canada. From 1995 to 2000, he was Vice President and Chief Research Scientist at Legasys Corporation, a software technology company specializing in legacy software system analysis and renovation. Dr. Cordy received his PhD degree from the University of Toronto, where he served several years as lecturer and senior research associate before moving to Queen's University in 1985. As leader of the TXL project, with hundreds of academic and industrial users worldwide, he is the author of more than 140 refereed contributions in programming languages, software engineering, and artificial intelligence and has served widely as member and chair of many conferences and workshops in programming languages and software engineering, most recently as Program Chair of ICSM 2011, IWSC 2011, and CASCON 2013 and as General Chair of SCAM 2012 and WCRE 2012. Dr. Cordy is an ACM Distinguished Scientist, a senior member of the IEEE, and an IBM Visiting Scientist and Faculty Fellow. Paolo Tonella is head of the Software Engineering Research Unit at Fondazione Bruno Kessler (FBK), in Trento, Italy. He received his PhD degree in Software Engineering from the University of Padova in 1999, with the thesis ‘Code Analysis in Support to Software Maintenance’. In 2011, he was awarded the ICSE 2001 MIP (Most Influential Paper) award, for his paper: ‘Analysis and testing of web applications’. He is the author of ‘Reverse engineering of object oriented code’, Springer, 2005. He participated in several industrial and EU projects on software analysis and testing. Dr. Tonella was Program Chair of ICSM 2011 and ICPC 2007 and General Chair of ISSTA 2010 and ICSM 2012. Among the others, he served in the program committees of ICSE, FSE, ICSM, ISSTA, ICST, and ICPC. In 2007, Paolo Tonella was ranked among the top 50 Software Engineering scholars (Communications of the ACM, vol. 50, n. 6, pp. 81–85). He has been recognized as a distinguished TOSEM referee several times in recent years (including 2011–2012). His key contributions to the research in software engineering include the following: a comprehensive reverse engineering method for object oriented software, pioneering research on model extraction for web testing and foundational work on evolutionary testing of object oriented software.

SUMMARY Web applications are often used to expose business processes implemented as software systems. This paper describes a technique for recovering business processes based on a dynamic analysis of the applications behavior. The technique described here does not require any access to internal software artifacts of the application, such as source code or documentation. An initial process is inferred to by means of the analysis of execution traces, in which the execution of GUI elements such as forms and links is recorded. The recovered process is then abstracted by clustering its elements according to four different criteria: structural, page‐based, dependency‐based and semantical. A case study has been conducted with the aim of evaluating understandability and readability of the reverse engineered processes as well as the clustering techniques used in refining them. Copyright © 2010 John Wiley & Sons, Ltd.

Automatically generated test cases are usually evaluated in terms of their fault revealing or coverage capability. Beside these two aspects, test cases are also the major source of information for fault localization and fixing. The impact of automatically generated test cases on the debugging activity, compared to the use of manually written test cases, has never been studied before. In this paper we report the results obtained from two controlled experiments with human subjects performing debugging tasks using automatically generated or manually written test cases. We investigate whether the features of the former type of test cases, which make them less readable and understandable (e.g., unclear test scenarios, meaningless identifiers), have an impact on accuracy and efficiency of debugging. The empirical study is aimed at investigating whether, despite the lack of readability in automatically generated test cases, subjects can still take advantage of them during debugging.

Automated web crawlers can be used to explore and exercise portions of a web application under test. However, the possibility to achieve full exploration of a web application through automated crawling is severely limited by the choice of the input values submitted with forms. Depending on the crawler's capabilities, a larger or smaller portion of web application will be automatically explored. In this paper, we introduce web crawl ability metrics to quantify properties of application pages and forms that affect crawl ability. Moreover, we show that our metrics can be used to identify the boundaries between those parts of the application that can be successfully crawled automatically and those parts that will require manual intervention or other crawl ability support. We have validated our crawl ability metrics on real web applications, for which low crawl ability was indeed associated with the existence of pages never exercised during automated crawling.

Models inferred from execution traces (logs) may admit more behaviours than those possible in the real system (over-approximation) or may exclude behaviours that can indeed occur in the real system (under-approximation). Both problems negatively affect model based testing. In fact, over-approximation results in infeasible test cases, i.e., test cases that cannot be activated by any input data. Under-approximation results in missing test cases, i.e., system behaviours that are not represented in the model are also never tested. In this paper we balance over- and under-approximation of inferred models by resorting to multi-objective optimization achieved by means of two search-based algorithms: A multi-objective Genetic Algorithm (GA) and the NSGA-II. We report the results on two open-source web applications and compare the multi-objective optimization to the state-of-the-art KLFA tool. We show that it is possible to identify regions in the Pareto front that contain models which violate fewer application constraints and have a higher bug detection ratio. The Pareto fronts generated by the multi-objective GA contain a region where models violate on average 2% of an application's constraints, compared to 2.8% for NSGA-II and 28.3% for the KLFA models. Similarly, it is possible to identify a region on the Pareto front where the multi-objective GA inferred models have an average bug detection ratio of 110 : 3 and the NSGA-II inferred models have an average bug detection ratio of 101 : 6. This compares to a bug detection ratio of 310928 : 13 for the KLFA tool.

Specifications mined from execution traces are largely used to support testing and analysis of software applications with little runtime variability. However, when models are mined from applications that evolve at runtime, the resulting models become quickly obsolete, and thus of little support for any testing and analysis activity. To cope with such systems, mined specifications must be consistently updated every time the software changes. In principle, models can be periodically mined from scratch, but in many cases this solution is too expensive or even impossible. In this paper we describe Revolution, an approach for the automatic evolution of specifications mined by applying state abstraction techniques. Revolution produces models that are continuously updated and thus remain aligned with the actual implementation. Empirical results show that Revolution can suitably address run-time evolving applications.

Model-based testing relies on the assumption that effective adequacy criteria can be defined in terms of model coverage achieved by a set of test paths. However, such test paths are only abstract test cases and input test data must be specified to make them concrete. We propose a novel approach that combines model-based and combinatorial testing in order to generate executable and effective test cases from a model. Our approach starts from a finite state model and applies model-based testing to generate test paths that represent sequences of events to be executed against the system under test. Such paths are transformed to classification trees, enriched with domain input specifications such as data types and partitions. Finally, executable test cases are generated from those trees using t-way combinatorial criteria.

The articles in this special section contain selected papers from the International Symposium on Software Testing and Analysis 2010.

Enriching business process models with semantic annotations that are taken from an ontology has become a crucial need in service provisioning, integration and composition, and business processes management. We represent semantically annotated business processes as part of an Web ontology lanuage knowledge base that formalizes the business process structure, the business domain, a set of criteria that describe correct semantic annotations, and a set of constraints that describe requirements on the business process itself. In this paper, we show how the Semantic Web representation and reasoning techniques can be 1) exploited by our aspect-oriented approach to modularize exception-handling (as well as other crosscutting) mechanisms and 2) effectively applied to formalize and automatically verify constraints on the management of exceptional flows (as well as other relevant flows) in business processes. The benefits of the Semantic Web and the aspect-oriented technologies are illustrated in a case study, where exceptional flows are modularized separately and managed at the semantic level due to the proposed approach.

Welcome to the 27thIEEE International Conference on Software Maintenance in Williamsburg, Virginia, USA. ICSM is continuing its journey around the world in a new location. The conference is hosted by the Williamsburg Lodge, situated in the heart of Historical Williamsburg. We hope you will enjoy the location as much as the conference.

Web services evolve frequently to meet new business demands and opportunities. However, service changes may affect service compositions that are currently consuming the services. Hence, audit testing (a form of regression testing in charge of checking for compatibility issues) is needed. As service compositions are often in continuous operation and the external services have limited (expensive) access when invoked for testing, audit testing has severe time and resources constraints, which make test prioritization a crucial technique (only the highest priority test cases will be executed).This paper presents a novel approach to the prioritization of audit test cases using information retrieval. This approach matches a service change description with the code portions exercised by the relevant test cases. So, test cases are prioritized based on their relevance to the service change. We evaluate the proposed approach on a system that composes services from eBay and Google.

We present an algorithm for constructing fitness functions that improve the efficiency of search-based testing when trying to generate branch adequate test data. The algorithm combines symbolic information with dynamic analysis and has two key advantages: It does not require any change in the underlying test data generation technique and it avoids many problems traditionally associated with symbolic execution, in particular the presence of loops. We have evaluated the algorithm on industrial closed source and open source systems using both local and global search-based testing techniques, demonstrating that both are statistically significantly more efficient using our approach. The test for significance was done using a one-sided, paired Wilcoxon signed rank test. On average, the local search requires 23.41% and the global search 7.78% fewer fitness evaluations when using a symbolic execution based fitness function generated by the algorithm.

The current Web applications are in continuous evolution to provide new and more complex functionalities, which can improve the user experience by means of adaptivity and dynamic changes. Since testing is the most frequently used technique to evaluate the quality of software applications in industry, novel testing approaches will be necessary to evaluate the quality of future (and more complex) web applications. In this paper, we investigate the testing challenges of future web applications and propose a testing methodology that addresses these challenges by the integration of search-based testing, model-based testing, oracle learning, concurrency testing, combinatorial testing, regression testing, and coverage analysis. This paper also presents a testing metamodel that states testing concepts and their relationships, which are used as the theoretical basis of the proposed testing methodology.

Experienced programmers choose identifier names carefully, in the attempt to convey information about the role and behavior of the labeled code entity in a concise and expressive way. In fact, during program understanding the names given to code entities represent one of the major sources of information used by developers. We conjecture that lexicon bad smells, such as, extreme contractions, inconsistent term use, odd grammatical structure, etc., can hinder the execution of maintenance tasks which rely on program understanding. We propose an approach to determine the extent of this impact and instantiate it on the task of concept location. In particular, we conducted a study on two open source software systems where we investigated how lexicon bad smells affect Information Retrieval-based concept location. In this study, the classes changed in response to past modification requests are located before and after lexicon bad smells are identified and removed from the source code. The results indicate that lexicon bad smells impact concept location when using IR-based techniques.

The prioritization of requirements is a crucial activity in the early phases of the software development process. It consists of finding an order relation among requirements, considering several requirements characteristics, such as stakeholder preferences, technical constraints, implementation costs and user perceived value.

Abstract Semantic annotations are a way to provide a precise meaning to business process elements, which supports reasoning on properties and constraints. Among the obstacles preventing widespread adoption of semantic annotations are the technical skills required to manage the formalization of the semantics and the difficulty of reconciling the different viewpoints of different analysts working on the same business process. In this paper, we support business analysts in the collaborative annotation of business processes by means of a tool inspired to the Wiki pages model. Using this tool, analysts can concurrently work on process elements, ontology concepts, process annotation or constraint specification. The underlying formalism is not exposed in the Wiki pages, where natural language templates are used. Copyright © 2011 John Wiley & Sons, Ltd.

Often, legacy data management systems provide no native support to transactions. Programmers protect data from concurrent access by adopting commonly agreed patterns, relying on low level concurrency primitives, such as semaphores. In such cases, consistent data access is granted only if all code components are compliant with the adopted mutual exclusion patterns. When migrating legacy systems to modern data management systems, the ad hoc mechanisms for data protection must be replaced with modern constructs for transaction management. In such cases, a literal translation may expose problems and bugs, which were originally masked by the specific implementation and patterns in use. In this paper, we propose a static flow analysis that determines the existence of potentially incompatible locks within the same thread, which require specific code re-engineering before migrating to a modern data management system. We report the results obtained on a concrete instance of this problem.

To support programmers during program maintenance we present an approach which extracts concepts and relations from the source code. Our approach applies natural language parsing to sentences constructed from the terms that appear in program element identifiers. The result of parsing can be represented as a dependency tree. Then, we automatically extract an ontology by mapping linguistic entities (nodes and relations between nodes in the dependency tree) to concepts and relations among concepts. We applied our approach to a case study and assessed the result in terms of the support it can give to concept location, executed in the context of bug fixing.

Abstract Legacy information systems, such as banking systems, are usually organized around their data model. Hence, when these systems are migrated to modern environments, translation of the data model involves the most critical decisions, having strong implications on the rest of the translation. In this paper, we report our experience and describe the approaches adopted in migrating a large banking system (ten million lines of code) to Java, starting from a proprietary data model which gives programmers explicit control of the variable overlay in memory. After presenting the basic translation scheme, we discuss the exceptions that may occur in practice. Then, we consider two heuristic approaches useful to reduce the number of cases where a behavior equivalent to that of unions must be reproduced in Java. Finally, we comment on the experimental results obtained so far. Copyright © 2009 John Wiley & Sons, Ltd.

In recent years, several design notations have been proposed to model domain-specific applications or reference architectures. In particular, Conallen has proposed the UML Web Application Extension (WAE): a UML extension to model Web applications. The aim of our empirical investigation is to test whether the usage of the Conallen notation supports comprehension and maintenance activities with significant benefits, and whether such benefits depend on developers ability and experience. This paper reports and discusses the results of a series of four experiments performed in different locations and with subjects possessing different experience-namely, undergraduate students, graduate students, and research associates-and different ability levels. The experiments aim at comparing performances of subjects in comprehension tasks where they have the source code complemented either by standard UML diagrams or by diagrams stereotyped using the Conallen notation. Results indicate that, although, in general, it is not possible to observe any significant benefit associated with the usage of stereotyped diagrams, the availability of stereotypes reduces the gap between subjects with low skill or experience and highly skilled or experienced subjects. Results suggest that organizations employing developers with low experience can achieve a significant performance improvement by adopting stereotyped UML diagrams for Web applications.

Pair-wise comparison has been successfully utilised in order to prioritise test cases by exploiting the rich, valuable and unique knowledge of the tester. However, the prohibitively large cost of the pair-wise comparison method prevents it from being applied to large test suites. In this paper, we introduce a cluster-based test case prioritisation technique. By clustering test cases, based on their dynamic runtime behaviour, we can reduce the required number of pair-wise comparisons significantly. The approach is evaluated on seven test suites ranging in size from 154 to 1,061 test cases. We present an empirical study that shows that the resulting prioritisation is more effective than existing coverage-based prioritisation techniques in terms of rate of fault detection. Perhaps surprisingly, the paper also demonstrates that clustering (even without human input) can outperform unclustered coverage-based technologies, and discusses an automated process that can be used to determine whether the application of the proposed approach would yield improvement.

Source code obfuscation is a protection mechanism widely used to limit the possibility of malicious reverse engineering or attack activities on a software system. Although several code obfuscation techniques and tools are available, little knowledge is available about the capability of obfuscation to reduce attackers' efficiency, and the contexts in which such an efficiencymay vary.

Test-driven software development tackles the problem of operationally defining the features to be implemented by means of test cases. This approach was recently ported to the early development phase, when requirements are gathered and clarified. Among the existing proposals, Fit (Framework for Integrated Testing) supports the precise specification of requirements by means of so called Fit tables, which express relevant usage scenarios in a tabular format, easily understood also by the customer. Fit tables can be turned into executable test cases through the creation of pieces of glue code, called fixtures.

Ajax supports the development of rich-client Web applications, by providing primitives for the execution of asynchronous requests and for the dynamic update of the page structure and content. Often, Ajax Web applications consist of a single page whose elements are updated in response to callbacks activated asynchronously by the user or by a server message. These features give rise to new kinds of faults that are hardly revealed by existing Web testing approaches. In this paper, we propose a novel state-based testing approach, specifically designed to exercise Ajax Web applications. The document object model (DOM) of the page manipulated by the Ajax code is abstracted into a state model. Callback executions triggered by asynchronous messages received from the Web server are associated with state transitions. Test cases are derived from the state model based on the notion of semantically interacting events. We evaluate the approach on a case study in terms of fault revealing capability. We also measure the amount of manual interventions involved in constructing and refining the model required by this approach.

The remote trust problem aims to address the issue of verifying the execution of a program running on an un-trusted host which communicates regularly with a trusted server. One proposed solution to this problem relies on a centralized scheme using assertions and replication to withhold usable services from a tampered client. We show how to extend such a scheme to a distributed trusted hardware such as tamper-resistant smartcards. We compared the performance and security of the proposed distributed system to the original centralized scheme on a case study. Our results indicate that, compared to a centralized scheme, our distributed trust scheme has dramatically lower network traffic, and smaller memory and computational requirements on the trusted server.

An appropriate translation of the data model is central to any language migration effort. Finding a mapping between original and target data models may be challenging for legacy languages (e.g., Assembly) which lack a structured data model and rely instead on explicit programmer control of the overlay of variables. Before legacy applications written in languages with an unstructured data model can be migrated to modern languages, a structured data model must be inferred. This paper describes a set of source transformations used to create such a model as part of a migration of eight million lines of code to Java. The original application is written in a proprietary language supporting variable layout by memory relocation.

Abstract Web applications are becoming more and more complex and difficult to maintain. To satisfy the customer's demands, they need to be updated often and quickly. In the maintenance phase, Web site understanding is a central activity. In this phase, programmers spend a lot of time and effort in the comprehension of the internal Web site structure. Such activity is often required because the available documentation is not aligned with the implementation, if not missing at all. Reverse engineering techniques have the potential to support Web site understanding, by providing views that show the organization of a site and its navigational structure. However, representing each Web page as a node in a diagram recovered from the source code of the Web site often leads to huge and unreadable graphs. Moreover, since the level of connectivity is typically high, the edges in such graphs make the overall result even less usable. In this paper, we propose an approach to Web site understanding based on clustering of client‐side HTML pages with similar content. This approach works well with content‐oriented sites rather than application‐oriented ones and uses a crawler to download the Web pages of the target Web site. The presence of common keywords is exploited to decide when it is appropriate to group pages together. An experimental work, including 17 Web sites, validates our approach and shows that the clusters produced automatically are close to those that a human would produce for a given Web site. Copyright © 2007 John Wiley & Sons, Ltd.

Proponents of design notations tailored for specific application domains or reference architectures, often available in the form of UML stereotypes, motivate them by improved understandability and modifiability. However, empirical studies that tested such claims report contradictory results, where the most intuitive notations are not always the best performing ones. This indicates the possible existence of relevant influencing factors, other than the design notation itself. In this work we report the results of a family of three experiments performed at different locations and with different subjects, in which we assessed the effectiveness of UML stereotypes for Web design in support to comprehension tasks. Replications with different subjects allowed us to investigate whether subjects' ability and experience play any role in the comprehension of stereotyped diagrams. We observed different behaviors of users with different degrees of ability and experience, which suggests alternative comprehension strategies of (and tool support for) different categories of users.

Identifiers represent an important source of information for programmers understanding and maintaining a system. Self-documenting identifiers reduce the time and effort necessary to obtain the level of understanding appropriate for the task at hand. While the role of the lexicon in program comprehension has long been recognized, only a few works have studied the quality and enhancement of the identifiers and no works have studied the evolution of the lexicon. In this paper, we characterize the evolution of program identifiers in terms of stability metrics and occurrences of renaming. We assess whether an evolution process similar to the one occurring for the program structure exists for identifiers. We report data and results about the evolution of three large systems, for which several releases are available. We have found evidence that the evolution of the lexicon is more limited and constrained than the evolution of the structure. We argue that the different evolution results from several factors including the lack of advanced tool support for lexicon construction, documentation, and evolution.

Abstract Web application design requires the modeling of multiple, separate concerns, such as the navigational structure, the business logic and the data persistence. To this aim, several methodologies have been conceived. One of them, the Web Application Extension (WAE), extends the UML notation by means of stereotypes and tagged values intended to capture Web‐specific concepts (e.g., the navigational structure). Although the WAE methodology is nowadays quite mature and ready for industrial adoption, the question whether it is able to actually facilitate the task of developers and maintainers has still to be empirically investigated. This paper reports and discusses the results from a controlled experiment on the benefits associated with the use of the WAE notation in the execution of comprehension tasks, carried out before maintenance. The WAE notation was compared against the use of pure unified modified language. Results indicate that the use of the WAE notation significantly improves the level of comprehension, although it does not increase the time needed to perform the comprehension task in a significant way. Copyright © 2007 John Wiley & Sons, Ltd.

The test case execution order affects the time at which the objectives of testing are met. If the objective is fault detection, an inappropriate execution order might reveal most faults late, thus delaying the bug fixing activity and eventually the delivery of the software. Prioritizing the test cases so as to optimize the achievement of the testing goal has potentially a positive impact on the testing costs, especially when the test execution time is long. Test engineers often possess relevant knowledge about the relative priority of the test cases. However, this knowledge can be hardly expressed in the form of a global ranking or scoring. In this paper, we propose a test case prioritization technique that takes advantage of user knowledge through a machine learning algorithm, case-based ranking (CBR). CBR elicits just relative priority information from the user, in the form of pairwise test case comparisons. User input is integrated with multiple prioritization indexes, in an iterative process that successively refines the test case ordering. Preliminary results on a case study indicate that CBR overcomes previous approaches and, for moderate suite size, gets very close to the optimal solution

This paper introduces the Species per Path approach to search-based software test data generation. The approach transforms the program under test into a version in which multiple paths to the search target are factored out. Test data are then sought for each individual path by dedicated 'species' operating in parallel. The factoring out of paths results in several individual search landscapes, with feasible paths giving rise to landscapes that are potentially more conducive to test data discovery than the original overall landscape.The paper presents the results of two empirical studies that validate and verify the approach. The validation study supports the claim that the approach is widely applicable and practical. The verification study shows that it is possible to generate test data for targets with the approach that are troublesome for the standard evolutionary method.

Multilingual Web sites are expected to provide the same content expressed in various languages, presented according to a common style, with the same interaction facilities. To this extent, most Web developers start from a source language version of the site and produce the multilingual versions by providing translations in all supported languages. Translation pages are usually generated by replicating the HTML structure and the scripting language sections of the original pages and by translating the textual sections into the target languages. This practice exposes the site to several problems during its evolution. Updates may be not properly propagated to all translations, and unwanted divergences can be introduced over time in content, presentation and interaction. In this paper, we propose a prototype toolkit, limited to Western languages, that can help restructuring an existing static Web site, and migrating its multilingual content to a unified and consistent representation. First of all, pages are classified according to the language of their content. Then, correspondences among pages in the original language and their translations are determined. Based on the computation of the edit operations necessary to make each page consistent with its translations, the site is updated to a new version where all pages are aligned. In the last phase, a unified representation of the structure and of the multilingual content of each page is inserted into a Content Management System. This ensures a consistent future evolution of the site. The prototype toolkit has been tested on 10 existing static Web sites, with texts in Italian, English, German and Spanish. For some of the above-mentioned phases, alternative solutions have been considered and their relative advantages have been evaluated against a manually constructed gold standard. We are quite confident that with some adaptation, most of the results we obtained can be extended to any pair of Western languages. Copyright © 2005 John Wiley & Sons, Ltd.

Aspect-oriented programming (AOP) provides mechanisms for the separation of crosscutting concerns - functionalities scattered through the system and tangled with the base code. Existing systems are a natural testbed for the AOP approach since they often contain several crosscutting concerns which could not be modularized using traditional programming constructs. This paper presents an automated approach to the problem of migrating systems developed according to the object-oriented programming (OOP) paradigm into aspect-oriented programming (AOP). A simple set of six refactorings has been defined to transform OOP to AOP and has been implemented in the AOP-migrator tool, an Eclipse plug-in. A set of enabling transformations from OOP to OOP complement the initial set of refactorings. The paper presents the results of four case studies, which use the approach to migrate selected crosscutting concerns from medium-sized Java programs (in the range of 10K to 40K lines of code) into equivalent programs in AspectJ. The case study results show the feasibility of the migration and indicate the importance of the enabling transformations as a preprocessing step

During software evolution, programmers devote most of their effort to the understanding of the structure and behavior of the system. For Object-Oriented code, this might be particularly hard, when multiple, scattered objects contribute to the same function. Design views offer an invaluable help, but they are often not aligned with the code, when they are not missing at all.This tutorial describes some of the most advanced techniques that can be employed to reverse engineer several design views from the source code. The recovered diagrams, represented in UML (Unified Modeling Language), include class, object, interaction (collaboration and sequence), state and package diagrams. A unifying static code analysis framework used by most of the involved algorithms is presented at the beginning of the tutorial. A single running example is referred all over the presentation. Trade-offs (e.g., static vs. dynamic analysis), limitations and expected benefits are also discussed.

This paper presents a human-guided automated approach to refactoring object oriented programs to the aspect oriented paradigm. The approach is based upon the iterative application of four steps: discovery, enabling, selection, and refactoring. After discovering potentially applicable refactorings, the enabling step transforms the code to improve refactorability. During the selection phase the particular refactorings to apply are chosen. Finally, the refactoring phase transforms the code by moving the selected code to a new aspect. This paper presents the results of an evaluation in which one of the crosscutting concerns of a 40,000 LoC program (JHotDraw) is refactored.

Aspect oriented programming aims at addressing the problem of the crosscutting concerns, i.e., those functionalities that are scattered among several modules in a given system. Aspects can be defined to modularize such concerns. In this work, we focus on a specific kind of crosscutting concerns, the scattered implementation of methods declared by interfaces that do not belong to the principal decomposition. We call such interfaces aspectizable. All the aspectizable interfaces identified within a large number of classes from the Java Standard Library and from three Java applications have been automatically migrated to aspects. To assess the effects of the migration on the internal and external quality attributes of these systems, we collected a set of metrics and we conducted an empirical study, in which some maintenance tasks were executed on the two alternative versions (with and without aspects) of the same system. In this paper, we report the results of such a comparison.

Given a binary relationship between objects and attributes, concept analysis is a powerful technique to organize pairs of related sets of objects and attributes into a concept lattice, where higher level concepts represent general features shared by many objects, while lower level concepts represent the object-specific features. Concept analysis was recently applied to several software engineering problems, such as: restructuring the code into more cohesive components, identifying class candidates, locating features in the code by means of dynamic analysis, reengineering class hierarchies. This paper provides the background knowledge required by such applications. Moreover, the methodological issues involved in the different applications of this technique are considered by giving a detailed presentation of three of them: module restructuring, design pattern inference and impact analysis based on decomposition slicing. The paper is concluded by an overview on other kinds of applications.

Separation of concerns and modularization are the cornerstones of software engineering. However, when a system is decomposed into units, functionalities often emerge which cannot be assigned to a single element of the decomposition. The implementation of interfaces represents a typical instance of this problem. In fact, the code that defines the interface methods is often scattered across several classes in the system and tangled with the original code. Aspect oriented programming provides mechanisms for the dynamic and static composition of transversal functionalities, that can be used to factor out the implementation of interfaces. We describe a technique for the identification of those interface implementations that are most likely to represent crosscutting concerns. Moreover, the code transformation (refactoring) to migrate such interfaces to aspects is also presented. Experimental results validate the approach.

Object oriented programming promotes reuse of classes in multiple contexts. Thus, a class is designed and implemented with several usage scenarios in mind, some of which possibly open and generic. Correspondingly, the unit testing of classes cannot make too strict assumptions on the actual method invocation sequences, since these vary from application to application.In this paper, a genetic algorithm is exploited to automatically produce test cases for the unit testing of classes in a generic usage scenario. Test cases are described by chromosomes, which include information on which objects to create, which methods to invoke and which values to use as inputs. The proposed algorithm mutates them with the aim of maximizing a given coverage measure. The implementation of the algorithm and its application to classes from the Java standard library are described.

Abstract Web sites are becoming increasingly important for companies and organizations. Crucial information as well as economic transactions are managed by Web sites, so that their reliability, usability and overall quality are central issues. The advent of new technologies, such as Web services, makes the domain of Web applications a very dynamic one, where the evolution is rapid and unavoidable. Moreover, existing systems are also eligible for migration to the Web, thus requiring specific reengineering processes and methods. This special issue is focused on the recent achievements produced by software engineering research that has been conducted on Web sites. Reverse engineering, restructuring, testing and dynamic analysis are well‐established disciplines that have been investigated in the new domain of the Web. Moreover, migration of legacy code to the Web poses technical problems related to domain analysis, adoption challenges and performance assessment. This issue provides an overview of the most promising results achieved in these fields. Copyright © 2004 John Wiley & Sons, Ltd.

Abstract The World Wide Web, initially intended as a way to publish static hypertexts on the Internet, is moving toward complex applications. Static Web sites are being gradually replaced by dynamic sites, where information is stored in databases and non‐trivial computation is performed. In such a scenario, ensuring the quality of a Web application from the user's perspective is crucial. Techniques are being investigated for the analysis and testing of Web applications for such a purpose. However, a static analysis of the source code may be extremely difficult (and, in general, infeasible) because of the presence of dynamic generation of the HTML code that is part of the application under analysis. In this paper, a dynamic analysis technique is proposed for the extraction of a Web application model through its execution. Availability of statistical data about the accesses to the pages generated by the Web application is exploited for statistical testing, based on the recovered model. Test cases can be prioritized, so as to exercise the most frequently followed paths first. Moreover, statistical reproduction of the user's navigation paths allows for an estimation of the reliability of the application. Copyright © 2004 John Wiley & Sons, Ltd.

In object oriented programming, the functionalities of a system result from the interactions (message exchanges) among the objects allocated by the system. While designing object interactions is far more complex than designing the object structure in forward engineering, the problem of understanding object interactions during code evolution is even harder, because the related information is spread across the code. In this paper, a technique for the automatic extraction of UML interaction diagrams from C++ code is proposed. The algorithm is based on a static, conservative flow analysis that approximates the behavior of the system in any execution and for any possible input. Applicability of the approach to large software is achieved by means of two mechanisms: partial analysis and focusing. Usage of our method on a real world, large C++ system confirmed its viability.

The decomposition slice graph and concept lattice are two program representations used to abstract the details of code into a higher-level view of the program. The decomposition slice graph partitions the program into computations performed on different variables and shows the dependence relation between computations, holding when a computation needs another computation as a building block. The concept lattice groups program entities which share common attributes and organizes such groupings into a hierarchy of concepts, which are related through generalizations/specializations. This paper investigates the relationship existing between these two program representations. The main result of this paper is a novel program representation, called concept lattice of decomposition slices, which is shown to be an extension of the decomposition slice graph, and is obtained by means of concept analysis, with additional nodes associated with weak interferences between computations, i.e., shared statements which are not decomposition slices. The concept lattice of decomposition slices can be used to support software maintenance by providing relevant information about the computations performed by a program and the related dependences/interferences, as well as by representing a natural data structure on which to conduct impact analysis. Preliminary results on small to medium size code support the applicability of this method at the intraprocedural level or when investigating the dependences among small groups of procedures.

When a software system enters the maintenance phase, the availability of accurate and consistent information about its organization can help alleviate the difficulties of program understanding. Reverse engineering methods aim at extracting such information directly from the code. While several tools support the recovery of the class diagram from object oriented code, so far no work has attacked the problem of statically characterizing the behavior of an object oriented system by means of diagrams which represent the class instances (objects) and their mutual relationships. In this paper a novel static analysis algorithm is proposed for extraction of the object diagram from the code, based on a program representation called the object flow graph. Partial object diagrams can be associated dynamically to the system by executing and tracing the program on a set of test cases. The complementary nature of these two views is discussed, and a novel approach to object oriented testing is derived from such a comparison. The usefulness of the proposed technique is illustrated on a real world, public domain C++ system.

Current practice of Web site development does not address explicitly the problems related to multilingual sites. The same information, as well as the same navigation paths, page formatting and organization, are expected to be provided by the site independently from the chosen language. This is typically ensured by adopting personal conventions on the way pages are named and on their location in the file system. Updates are then performed manually and consistency depends on the ability of the programmers not to miss any impact of the change. In this paper an extension to XHTML, called MLHTML (MultiLingual XHTML), is proposed as the target representation of a restructuring process aimed at producing a maintainable and consistent multilingual Web site. MLHTML centralizes the language dependent variants of a page in a single representation, where shared parts are not duplicated Existing sites can be migrated to MLHTML by means of the algorithms described in this paper. After classifying the pages according to their language, a page alignment technique is exploited to identify corresponding pages and to eliminate inconsistencies. Transformation into MLHTML can then be achieved automatically.

The C++ programming language offers a wide range of coding alternatives to programmers. Some language characteristics inherited from C are potential sources of problems during the evolution of a system. Global variables, arbitrary branches, pointers, memory management instructions and macro directives are examples of them. The resulting code may be difficult to understand, maintain and test, if such language features are used without proper discipline. The Java language was designed with the explicit goal of excluding all troublesome features that can be found in C++, by disregarding some of them (e.g., globals), and disciplining other ones (e.g., pointers). For several companies the migration to Java is not strategically feasible. Nevertheless, the adoption of its philosophy within C++ code could help improve the quality of the software. In this paper a subset of C++, named Cjj, is defined which is compliant with Java, in that it enforces a Java-like programming style, while forbidding all C++ constructs that are in contrast with it. A tool for the verification of the compliance of an input source with Cjj was developed. It verifies if any of the constraints defining Cjj is violated.

The economic relevance of Web applications increases the importance of controlling and improving their quality. Moreover, the newly available technologies for their development allow the insertion of sophisticated functions, but often leave the developers responsible for their organization and evolution. As a consequence, a high demand is emerging for methodologies and tools for the quality assurance of Web-based systems. In this paper, a UML model of Web applications is proposed for their high-level representation. Such a model is the starting point for several analyses, which can help in the assessment of the static site structure. Moreover, it drives Web application testing, in that it can be exploited to define white-box testing criteria and to semi-automatically generate the associated test cases. The proposed techniques were applied to several real-world Web applications. The results suggest that automatic support for verification and validation activities can be extremely beneficial. In fact, it guarantees that all paths in the site which satisfy a selected criterion are properly exercised before delivery. The high level of automation that is achieved in test case generation and execution increases the number of tests that are conducted and simplifies the regression checks.

Program slicing revealed a useful way to limit the search of software defects during debugging and to better understand the decomposition of the application into computations. We propose to extend the extraction of slices to Web applications, in order to produce a reduced Web application which behaves as the original one with respect to some criterion, i.e., some displayed information of interest. After presenting the theoretical implications of applying slicing to Web applications, we demonstrate its usefulness with reference to an example, derived from a survey of a set of travel agency sites. Web application slicing helps to disclose relevant information and understand the internal system structure.

UML diagrams, and in particular the most frequently used one, the class diagram, represent a valuable source of information even after the delivery of the system, when it enters the maintenance phase. Several tools provide a reverse engineering engine to recover it from the code. In this paper an algorithm is proposed for the improvement of the accuracy of the UML class diagram extracted from the code. Specifically, important information about inter-class relations may be missed in a reverse engineered class diagram, when weakly typed containers, i.e., containers collecting objects whose type is the top of the inheritance hierarchy, are employed. In fact, the class of the contained objects is not directly known, and therefore no relation with it is apparent from the container declaration. The proposed approach was applied to several software components developed at CERN. Experimental results highlight that a substantial improvement is achieved when the container type information is refined with the inferred data. The number of relations otherwise missed is relevant and the connectivity of the associated class diagrams is radically different when containers are considered.

Abstract When designing a new application, experienced software engineers usually adopt solutions that have proven successful in previous projects. Such reuse of code organizations is seldom made explicit. Nevertheless, it represents important information, which can be extremely valuable in the maintenance phase by documenting the design choices underlying the implementation. In addition it can be reused whenever a similar problem is encountered. In this paper an approach for the inference of recurrent design patterns directly from the code is proposed. No assumption is made on the availability of any pattern library, and the concept analysis algorithm—adapted for this purpose—is able to infer the presence of class groups which instantiate a common, repeated pattern. In fact, concept analysis provides sets of objects sharing attributes, which—in the case of object‐oriented design patterns—become class members or inter‐class relations. The approach was applied to three C++ applications for which the structural relations among classes led to the extraction of a set of design patterns, which could be enriched with non‐structural information about class members and method invocations. The resulting patterns could be interpreted as meaningful organizations aimed at solving general problems which have several instances in the applications analyzed. Copyright © 2001 John Wiley & Sons, Ltd.

Low coupling between modules and high cohesion inside each module are the key features of good software design. This paper proposes a new approach to using concept analysis for module restructuring, based on the computation of extended concept subpartitions. Alternative modularizations, characterized by high cohesion around the internal structures that are being manipulated, can be determined by such a method. To assess the quality of the restructured modules, the trade-off between encapsulation violations and decomposition is considered, and proper measures for both factors are defined. Furthermore, the cost of restructuring is evaluated through a measure of distance between the original and the new modularizations. Concept subpartitions were determined for a test suite of 20 programs of variable size: 10 public-domain and 10 industrial applications. The trade-off between encapsulation and decomposition was measured on the resulting module candidates, together with an estimate of the cost of restructuring. Moreover, the ability of concept analysis to determine meaningful modularizations was assessed in two ways. First, programs without encapsulation violations were used as oracles, assuming the absence of violations as an indicator of careful decomposition. Second, the suggested restructuring interventions were actually implemented in some case studies to evaluate the feasibility of restructuring and to deeply investigate the code organization before and after the intervention. Concept analysis was experienced to be a powerful tool supporting module restructuring.

The identifiers chosen by programmers as entity names contain valuable information. They are often the starting point for the program understanding activities, especially when high level views, like the call graph, are available. An approach for the restructuring of program identifier names is proposed, aimed at improving their meaningfulness. It considers two forms of standardization, associated respectively to the lexicon of the composing terms and to the syntax of their arrangement. Automatic and semiautomatic techniques are described which can help the restructuring intervention. Their application to a real world case study is also presented.

Web sites are becoming important assets for several companies, which need to incorporate sophisticated technologies into complex and large Web based systems. As a consequence, methodologies and tools are required for their design, implementation and maintenance. In particular the possibility for a site to evolve so as to provide updated and accessible information is a fundamental need. Web sites are considered the object of several analyses, focused on their structure and their history, with the purpose of supporting maintenance activities. Structural information may help understanding the organization of the pages in the site, while history analysis provides indications on modifications that do not correspond to the original design or that produce undesirable effects. A tool was developed to implement the analysis of Web site structure and evolution. Its application to some examples downloaded from the Web highlights several areas where the extracted information can improve the control on the maintenance phase and provide valuable support.

The ITC-Irst Reverse Engineering group was charged with analyzing a software application of approximately 4.7 million lines of C code. It was an old legacy system, maintained for a long time, on which several successive adaptive and corrective maintenance interventions had led to the degradation of the original structure. The company decided to re-engineer the software instead of replacing it, because the complexity and costs of re-implementing the application from scratch could not be afforded, and the associated risk could not be run. Several problems were encountered during re-engineering, including identifying dependencies and detecting redundant functions that were not used anymore. To accomplish these goals, we adopted a conservative approach. Before performing any kind of analysis on the whole code, we carefully evaluated the expected costs. To this aim, a small but representative sample of modules was preliminarily analyzed, and the costs and outcomes were extrapolated so as to obtain some indications on the analysis of the whole system. When the results of the sample modules were found to be useful as well as affordable for the entire system, the resources involved were carefully distributed among the different reverse engineering tasks to meet the customer's deadline. This paper summarizes that experience, discussing how we approached the problem, the way we managed the limited resources available to complete the task within the assigned deadlines, and the lessons we learned. Copyright © 2000 John Wiley & Sons, Ltd.

This paper presents an approach to automatic unit test data generation for branch coverage using mixed-integer linear programming, execution trees, and symbolic execution. This approach can be useful to both general testing and regression testing after software maintenance and reengineering activities. Several strategies, including original algorithms, to move towards practical test data generation have been investigated in this paper. Methods include: the analysis of minimum path-length partial execution trees for unconstrained arcs, thus increasing the generation performance and reducing the difficulties originated by infeasible paths the reduction of the difficulties originated by nonlinear path conditions by considering alternative linear paths the reduction of the number of test cases, which are needed to achieve the desired coverage, based on the concept of unconstrained arcs in a control flow graph the extension of symbolic execution to deal with dynamic memory allocation and deallocation, pointers and pointers to functions system. Preliminary results are encouraging and show that a high percentage of the program branches can be covered by the test data automatically produced. The approach is flexible to branch selection criteria coming from general testing as well as regression testing.

When designing a new application, experienced software engineers usually try to employ solutions that proved successful in previous projects. Such reuse of code organizations is seldom made explicit. Nevertheless it represents important information about the system, that can be extremely valuable in the maintenance phase by documenting the design choices underlying the implementation. In addition, having it available, it can be reused whenever a similar problem is encountered. In this paper an approach is proposed to the inference of recurrent design patterns directly from the code or the design. No assumption is made on the availability of any pattern library and the concept analysis algorithm, adapted for this purpose, is able to infer the presence of class groups which instantiate a common, repeated pattern. In fact, concept analysis provides sets of objects sharing attributes, which, in the case of object oriented design patterns, become class members or inter-class relations. The approach was applied to a C++ application, for which the structural relations among classes led to the extraction of a set of structural design patterns, which could be enriched with non structural information about class members and method invocations. The resulting patterns could be interpreted as meaningful organizations aimed at solving general problems which have several instances in the analyzed application.

When programmers perform maintenance tasks, program understanding is often required. One of the first activities in understanding a software system is identifying its subsystems and their relations, i.e., its software architecture. Since a large part of the effort is spent in creating a mental model of the system under study, tools can help maintainers in managing the evolution of legacy systems by showing them architectural information. This paper describes an environment for the architectural recovery of software systems called the architectural recovery tool (ART). The environment is based on a hierarchical architectural model that drives the application of a set of recognizers, each producing a different architectural view of a system or of some of its parts. Recognizers embody knowledge about architectural clichés and use flow analysis techniques to make their output more accurate. To test the accuracy and effectiveness of the ART, a suite of public domain applications containing interesting architectural organizations was selected as a benchmark. Results are presented by showing ART performance in terms of precision and recall of the architectural concept retrieval process. The results obtained show that cliché-based architectural recovery is feasible and the recovered information can be valuable support in reengineering and maintenance activities. Copyright © 1999 John Wiley & Sons, Ltd.

Ascertaining the reaching definitions from the source code can give views of the linkages in that source code. These views can aid source code analyses, such as impact analysis and program slicing, and can assist in the reverse engineering and re-engineering of large legacy systems. Maintainers like to do such activities interactively and value fast responses from program analysis tools. Therefore the control of the trade-off between accuracy and efficiency should be given to the maintainer. Since some real world programs, especially in languages like C, make much use of pointers, and efficient points-to analysis should be integrated within the computation of the data dependencies during the process of ascertaining the reaching definitions. This paper proposes three different approaches to the analysis of the reaching definitions based on different levels of precision, reflecting differences in their sensitivity to the calling context and the control flow. The least precise approach produces an overestimate by an average of 41% of data dependencies compared to the approach with the highest degree of precision. The result for the least precise approach is conservative because all detectable data dependencies are included, and is far faster than the more precise approaches. Runs on a test suite show an almost 2000 to 1 reduction in execution time by the least precise approach compared with the most precise approach. The intermediate approach is more than 30 times faster than the most precise approach, and much more precise than the least precise one (an average of 2% extra dependencies compared to the most precise approach). Therefore, while on medium size systems the intermediate approach could be a good compromise, on large systems the least precise approach becomes extremely valuable, being the only one feasible. Copyright © 1999 John Wiley & Sons, Ltd.

The dynamic evolution of ecological systems in which predators and prey compete for survival has been investigated by applying suitable mathematical models. Dynamic systems theory provides a useful way to model interspecies competition and thus the evolution of predator and prey populations. This kind of mathematical framework has been shown to be well suited to describe evolution of economical systems as well, where instead of predators and prey there are consumers and resources. Maintenance and testing activities absorb the most relevant part of the total life-cycle cost of software. Such economic relevance strongly suggests to investigate the maintenance and testing processes in order to find new models allowing software engineers to better estimate, plan and manage costs and activities. We show how dynamic systems theory could be usefully applied to maintenance and testing context, namely to model the dynamic evolution of the effort. When programmers start trying to recognize and correct code defects, while the number of residual defects decreases, the effort spent to find out any new defect has an initial increase, followed by a decline, in a similar way as prey and predator populations do. The feasibility of this approach is supported by the experimental data about two real world software projects.

Good software design is characterized by low coupling between modules and high cohesion inside each module. This is obtained by encapsulating the details about the internal structure of data and exporting only public functions with a clean interface. For programming languages such as C, which offer little support for encapsulation, code analysis tools may help in assessing and improving the access to data structures. In this paper a new representation of the accesses of functions to dynamic locations, called the O-A diagram, is proposed. By isolating meaningful groups of functions working on common dynamic data, such a diagram can be used to evaluate the encapsulation in a program and to drive possible interventions to improve it. Experimental results suggest that the aggregations identified by the O-A diagram are actually cohesive functions operating on a shared data structure. The results are useful in themselves, by providing the programmer with information about the organization of the accesses to dynamic memory. In addition the O-A diagram permits highlighting violations of encapsulation, so that proper restructuring actions can be performed.

Large software systems are difficult to understand and maintain.Code analysis tools can provide programmers with different views of the software which may help their understanding activity.To be applicable to real programs written in modern programming languages, these tools need to efficiently handle pointers.In the case of C++ analysis, object oriented peculiarities (like, e.g., polymorphism) have to be accounted for as well.We propose a flow insensitive, context insensitive points-to analysis capable of dealing with the features of the object oriented code.It is extremely promising because of the positive trade-off between complexity and accuracy.The integration of the points-to results with other analyses, such as reaching definitions and slicing, is also discussed in the context of our program understanding environment.

During maintenance activities, the availability of integrated conceptual views that present software at different levels of abstraction, from software architecture to control and data flow relations at code level, is fundamental to understand and modify legacy systems. This paper presents CANTO (Code and Architecture Analysis Tool), a comprehensive program understanding and maintenance environment which integrates fine grained information with architectural views extracted from source code, giving the user control of what is being computed by analyses. The capabilities and usefulness of CANTO are illustrated with reference to a real understanding and maintenance task

When programmers perform maintenance tasks, program understanding is required. One of the first activities in understanding a software system is identifying its subsystems and their relations, i.e. its software architecture. Since a large part of the effort is spent in creating a mental model of the system under study, tools can help maintainers in managing the evolution of legacy systems, by showing them architectural information. An environment for the architectural analysis of software systems is described. The environment is based on a hierarchical architectural model that drives the application of a set of recognizers, each producing a different architectural view of the system or of some of its parts. Recognizers embody knowledge about architectural cliches and use flow analysis techniques to make their output more accurate.

An approach to reengineer BASIC PC legacy code into modern graphical systems is proposed. BASIC has historically been one of the first languages available on PCs. Based on it, small or medium size companies have developed systems that represent valuable company assets to be preserved. Our goal is the automatic migration from the BASIC character oriented user interface to a graphical environment which includes a GUI builder, and compiles event driven C/C++ code. For this purpose a conceptual representation in terms of abstract graphical objects and call-backs has been inferred from the original code, and a translator from BASIC to C has been developed. Moreover the GUI builder internal representation has been generated, so that the user interface can be interactively fine-tuned by the programmer. We present and discuss BASIC peculiarities, with preliminary results on code translation. For the explanation of our approach to user interface migration an example is used throughout the text.