Model-based diagnosis is a generally applicable, principled approach to the systematic debugging of a wide range of system types such as circuits, knowledge bases, physical devices, or software. Based on a formal description of the system, it enables precise and deterministic reasoning about potential faults responsible for observed misbehavior. In software, such a formal system description can often even be extracted from the buggy program fully automatically. As logical reasoning is central to diagnosis, the performance of model-based debuggers is largely influenced by reasoning efficiency, which in turn depends on the complexity and expressivity of the system description. Since highly detailed models capturing exact semantics often exceed the capabilities of current reasoning tools, researchers have proposed more abstract representations. In this work, we thoroughly analyze system modeling techniques with a focus on fault localization in spreadsheets—one of the most widely used end-user programming paradigms. Specifically, we present three constraint model types characterizing spreadsheets at different abstraction levels, show how to extract them automatically from faulty spreadsheets, and provide theoretical and empirical investigations of the impact of abstraction on both diagnostic output and computational performance. Our main conclusions are that (i) for the model types, there is a trade-off between the conciseness of generated fault candidates and computation time, (ii) the exact model is often impractical, and (iii) a new model based on qualitative reasoning yields the same solutions as the exact one in up to more than half the cases while being orders of magnitude faster. Due to their ability to restrict the solution space in a sound way, the explored model-based techniques, rather than being used as standalone approaches, are expected to realize their full potential in combination with iterative sequential diagnosis or indeterministic but more performant statistical debugging methods.

Many systems in our daily lives control physical processes, which are parametrized and adapted, such as heating systems in buildings. Faults and non-optimized settings lead to a high energy demand and, therefore, need to be detected as early as possible. Unfortunately, due to specific adaptations, only the basic principles remain the same, but not the concrete implementations, making the use of techniques like machine learning difficult. Therefore, we suggest using abstract models that cover the basic behavior in a way that allows us to reuse the models in different installations. In particular, we discuss the application of qualitative simulation for fault detection and introduce a formal definition of conformance between the results of qualitative simulation and the monitored behavior. We discuss arising difficulties and provide a basis for further research and applications.

This paper introduces Spectral Fault Receptive Fields (SFRFs), a biologically inspired technique for degradation state assessment in bearing fault diagnosis and remaining useful life (RUL) estimation. Drawing on the center-surround organization of retinal ganglion cell receptive fields, we propose a frequency-domain feature extraction algorithm that enhances the detection of fault signatures in vibration signals. SFRFs are designed as antagonistic spectral filters centered on characteristic fault frequencies, with inhibitory surrounds that enable robust characterization of incipient faults under variable operating conditions. A multi-objective evolutionary optimization strategy based on NSGA-II algorithm is employed to tune the receptive field parameters by simultaneously minimizing RUL prediction error, maximizing feature monotonicity, and promoting smooth degradation trajectories. The method is demonstrated on the XJTU-SY bearing run-to-failure dataset, confirming its suitability for constructing condition indicators in health monitoring applications. Key contributions include: (i) the introduction of SFRFs, inspired by the biology of vision in the primate retina; (ii) an evolutionary optimization framework guided by condition monitoring and prognosis criteria; and (iii) experimental evidence supporting the detection of early-stage faults and their precursors. Furthermore, we confirm that our diagnosis-informed spectral representation achieves accurate RUL prediction using a bagging regressor. The results highlight the interpretability and principled design of SFRFs, bridging signal processing, biological sensing principles, and data-driven prognostics in rotating machinery.

Improving sustainability in the building sector requires more efficient operation of energy-intensive systems such as Heating, Ventilation, and Air Conditioning (HVAC). We present a novel diagnostic framework for HVAC systems that integrates Answer Set Programming (ASP) with Functional Event Calculus (FEC). Our approach exploits the declarative nature of ASP for modeling and incorporates FEC to capture temporal system dynamics. We demonstrate the feasibility of our approach through a case study on a real-world heating system, where we model key components and system constraints. Our evaluation on nominal and faulty traces shows that exploiting ASP in combination with FEC can identify plausible diagnoses. Moreover, we explore the difference between static and rolling-window strategies and provide insights into runtime versus soundness on those variants. Our work provides a step toward the practical application of ASP-based temporal reasoning in building diagnostics.

Fault detection in heating systems is critical for ensuring energy efficiency and operational reliability. Traditional approaches rely on labeled fault data and expert-defined rules, which are often unavailable or costly to obtain. We introduce GEMMA-FD (GEMMA for Fault Detection), a novel zero-shot framework for fault detection in heat pumps that leverages large language models (LLMs) without requiring labeled anomalies or predefined fault signatures. Our method transforms multivariate sensor time series into structured natural language prompts and augments them with visual features, such as line plots of key variables, to facilitate multimodal reasoning. Using GEMMA-3, an open-weight multimodal LLM, we classify heat pump system states as either normal or faulty. Experiments on a real-world heat pump dataset show that GEMMA-FD can identify unseen faults with reasonable precision, although its performance remains lower than a supervised XGBoost baseline trained on the same prompts. Specifically, GEMMA-FD achieves a macro-F1 score of 0.252, compared to 0.69 for XGBoost, underscoring the trade-off between generalization and targeted accuracy. Nevertheless, GEMMA-FD demonstrates the potential of foundation models for interpretable, multilingual fault detection in cyber-physical systems, while highlighting the need for prompt engineering, few-shot augmentation, and multimodal inputs to improve the classification of rare and complex fault types.

Identifying and localizing bugs in programs has always been considered a complex but essential topic. Whereas the former has led to substantial progress in areas like formal verification and testing with a high degree of automation, the latter has not been satisfactorily automated. Approaches like program slicing, model-based diagnosis, and, more recently, spectrum-based fault localization can be used to find possible causes of a misbehaving program automatically, but often come with high computational complexity or a larger list of diagnoses, which require additional manual effort. In this paper, we present the first experimental results of an approach that combines program slicing with spectrum-based fault localization aiming at improving the outcome of automated debugging methods. In contrast to previous work, where we illustrated potential improvements only by considering a particular use case, we present an evaluation based on 22 different example programs in this paper. The approach improves the wasted effort on average by around 5 to 15% on average.

Extensive research has been applied to autonomous systems, particularly in the field of autonomous driving, encompassing a range of tasks, from object recognition to planning and control. In the presentation, we discuss research activities and solutions while also focusing on verification. The latter is essential for ensuring the safety and robustness of such systems under any scenario driven by interactions between the autonomous system and its environment. We further outline that verification during development appears to be insufficient, necessitating runtime monitoring based on regulations and physical laws, where knowledge-based systems provide an excellent foundation for this purpose.

Assuring quality for cyber-physical systems has been a significant concern, leading to various proposed solutions. Faults in cyber-physical systems lead to security and safety issues in communication and operation, respectively. To prevent harm, verification and validation methodologies are applied during development. However, there might be no guarantee that the final deployed system is fault-free, i.e., a residual risk always remains. This paper focuses on involved risks, identifies their sources, and discusses methods for risk reduction in cyber-physical systems. For this purpose, a holistic approach to risk reduction in cyber-physical systems is utilized. Further, different stages of system development and operation are explained, and methodologies for finding defects and evaluating risks are discussed. Finally, concepts and methods using an industrial battery management system are presented. Specifically, the benefits of using formal methods to reduce risks in the context of autonomous driving and ADAS functionality are illustrated.

Visualizations in teaching have become a common practice as they effectively convey theoretical concepts. Compiler construction, a heavily theory-based subject in computer science education, is particularly challenging for students to understand. While many tools simulate a compiler's front end, or analysis phase, applications that focus on the back end, or synthesis phase, are scarce. This paper describes VisOpt, a web-based visualization tool designed for a master's level Compiler Construction course. VisOptfocuses on the synthesis phase, i.e., code optimization and code generation. Its primary objective is to help students comprehend various local compiler optimizations, which can be visualized on the original code, an intermediate representation, or an assembler-like target code. A quasi-experiment with a pre-test-post-test design revealed that students who used VisOpt reported higher self-efficacy compared to those who did not. Although no significant improvement in learning outcomes was observed overall, we propose VisOpt as an engaging pedagogical tool that effectively complements traditional methods for teaching the synthesis phase of compilers.

Modern cyber-physical systems integrate data from many sensors as a regular part of their operations. Over the years, researchers have proposed methods ranging from statistical approaches to neural networks to achieve this sensor fusion, along with high-level paradigms such as early and late fusion. However, quality assurance of sensor fusion algorithms typically focuses on highlighting their accuracy or ability to reduce uncertainty under given conditions. This paper aims to establish a qualitative approach to testing sensor fusion. We formulate an answer set program based on desirable properties for sensor fusion and show how to apply it to test fusion algorithms. Our results indicate that our approach is effective at finding faults, but does not easily find minimal models for large inputs.

Recent advancements in automated and autonomous driving systems have facilitated their integration into modern vehicles, enabling them to accurately perceive their surroundings and support or even fully undertake complex driving tasks. Given the complexity and unpredictable nature of driving environments and traffic situations, ensuring the correct behavior of such systems is essential to prevent hazardous situations, increase user acceptance, and avoid human harm. However, the increased complexity of these systems and the extensive search space of possible scenarios introduce significant challenges to testing and real-time fault management. Hence, besides rigorous testing during the development phase, there is a need for additional validation and verification during operation. This paper proposes utilizing Answer Set Programming (ASP), a form of declarative programming, for continuous real-time monitoring, fault detection, and explanation to ensure the correct functioning of automated and autonomous driving systems. Our approach aims to enhance the reliability and safety of such systems by detecting violations and providing explanations that can support fault-adaptive control or mitigation strategies. We demonstrate the effectiveness of our methodology across diverse scenarios executed within a simulation environment, discuss the main challenges encountered, and outline future research directions.

This paper examines two prominent Fault Detection and Isolation methodologies: the Signature Matrix approach, traditionally used in hardware systems, and the Spectrum-based approach, applied in software fault localization. Despite their distinct operational domains, both methods share the objective of precisely identifying and isolating faults. This study aims to compare these approaches and to highlight their similarities in principle. Through a comparative analysis, we assess how the structured pattern recognition of the Signature Matrix method and the statistical analysis capabilities of the Spectrum-based approach can be synergized to enhance diagnostic processes of cyber-physical systems that are composed of both hardware and software components. The investigation is motivated by the prospect of developing a hybrid Fault Detection and Isolation strategy that incorporates the robust detection mechanisms of hardware diagnostics with the techniques used in software fault localization. The findings are intended to advance the theoretical framework of Fault Detection and Isolation systems and suggest practical implementations across varied technological platforms, thereby improving the reliability and efficiency of fault detection and isolation in both hardware and software contexts.

The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.

Over the past few years, increasingly complex machine learning methods have been applied for various Software Engineering (SE) tasks, particularly for the important task of automated fault prediction and localization. It, however, becomes much more difficult for scholars to reproduce the results that are reported in the literature, especially when the applied deep learning models and the evaluation methodology are not properly documented and when code and data are not shared. Given some recent—and very worrying—findings regarding reproducibility and progress in other areas of applied machine learning, this study aims to analyze to what extent the field of software engineering, in particular in the area of software fault prediction, is plagued by similar problems. We have therefore conducted a systematic review of the current literature and examined the level of reproducibility of 56 research articles that were published between 2019 and 2022 in top-tier software engineering conferences. Our analysis revealed that scholars are apparently largely aware of the reproducibility problem, and about two-thirds of the papers provide code for their proposed deep-learning models. However, it turned out that in the vast majority of cases, crucial elements for reproducibility are missing, such as the code of the compared baselines, code for data pre-processing, or code for hyperparameter tuning. In these cases, it, therefore, remains challenging to reproduce the results in the current research literature exactly. Overall, our meta-analysis, therefore, calls for improved research practices to ensure the reproducibility of machine-learning-based research.

Since the advent of Large Language Models (LLM[s]) a few years ago, they have not only reached the mainstream but have become a commodity. Their application areas steadily expand because of sophisticated model architectures and enormous training corpora. However, accessible chatbot user interfaces and human-like responses may cause a tendency to overestimate their abilities. This study contributes to demonstrating the strengths and weaknesses of LLMs. In this work, we bridge methods from sub-symbolic and symbolic AI. In particular, we evaluate the capabilities of LLMs to convert textual requirements documents into their logical representation, enabling analysis and reasoning. This task demonstrates a use case close to industry, as requirements analysis is key in requirements and system engineering. Our experiments evaluate the popular model family used in OpenAI’s ChatGPT, GPT-3.5, and GPT-4. The underlying goal of testing for the correct abstraction of meaning is not trivial, as the relationship between input and output semantics is not directly measurable. Thus, it is necessary to approximate translation correctness through quantifiable criteria. Most notably, we defined consistency-based metrics for the plausibility and stability of translations. Our experiments give insights into syntactical validity, semantic plausibility, stability of translations, and parameter configurations for LLM translations. We use real-world requirements and test the LLMs’ performance out of the box and after pre-training. Experimentally, we demonstrated the strong relation between ChatGPT parameters and the stability of translations. Finally, we showed that even the best model configurations produced syntactically faulty (5%) or semantically implausible (7%) output and are not stable in their results.

Abstract In this paper, we outline an approach for automatically generating challenging road networks for virtual testing of an automated lane‐keeping system. Based on a set of control points, we construct a parametric curve representing a road network, defining the dynamic driving task an automated lane‐keeping system‐equipped vehicle must perform. Changing control points has a global influence on the resulting road geometry. Our approach uses search to find control‐point sets that result in a challenging road, eventually forcing the vehicle to leave the intended path. We apply our approach in different search variants to evaluate their performance regarding test efficiency and the diversity of failing tests. In addition, we evaluate different genetic algorithm control parameter configurations to investigate the most influential parameters and if specific configurations can be seen asoptimal, leading to better results than others. For both studies, we consider another search‐based test method and two different random test generators as a baseline for comparison. The empirical results indicate that specific control parameter settings increase the overall performance for each search variant. While the population size is the most influential control parameter for all methods, the performance improvement when usingoptimalsettings is only significant for one method.

Abstract As software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser‐known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open‐source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power.

Vehicle and traffic simulation is a common practice for testing and evaluating advanced driver-assistance systems (ADAS) and autonomous driving (AD). As a result, the literature mentions numerous simulators capable of simulating ADAS/AD implementations. In this study, we investigate previous surveys that cover multiple scenarios and initiate a systematic review targeting simulators for testing ADAS/AD. Our results show that the literature mentions, in total, 181 simulators capable of evaluating one or more ADAS/AD implementations. Furthermore, according to previous surveys and reviews, the most popular simulators are CARLA, Airsim, and SUMO. Finally, our results uncover that every five years, the number of novel simulators added to the literature grows at least quadratically, showing that further review is necessary to address the differences between these simulators and understand the simulator landscape from an ADAS/AD testing perspective.

As the automotive industry increasingly relies on wireless technologies, a new attack surface emerges, posing significant security threats to modern cars. This paper investigates the vulnerabilities and risks of wireless vehicle attacks, including eavesdropping, message tampering, spoofing, and jamming. It highlights vulnerabilities in the CAN bus communication interface. By exploring these attacks and their potential consequences, this paper aims to shed light on the urgent need for robust security measures to safeguard the safety and privacy of vehicle owners. The focus is on understanding the evolving landscape of wireless threats in the automotive industry, providing valuable insights for researchers, practitioners, and stakeholders involved in developing effective countermeasures and enhancing overall vehicle security. In contrast to other research articles, this paper presents the ISO/SAE DIS 21434 standard, which offers a systematic and structured approach to enhance cybersecurity in the automotive industry, even in the face of emerging wireless threats. In addition, this paper highlights notable examples of attacks on modern cars, where researchers gained access to vehicle systems through wireless vulnerabilities, demonstrating the potential dangers of interconnected car systems to illustrate the real-world implications.

Data abstraction plays a crucial role in various application domains, allowing for simplification and representation of complex data sets. This paper focuses on data abstraction in the context of data clustering for test case generation, specifically in the automotive domain. Our main objective is to investigate whether we can use data abstraction to enhance the clustering outcome. We propose different abstraction functions for vehicle sensor data obtained from real-world driving data. We use these abstracted data sets as input to a clustering approach that identifies similar driving scenarios and extracts driving episodes. We evaluate the quality of the clusters using three clustering validation metrics and a Pearson correlation-based metric that assesses the similarity between the extracted driving episodes. To evaluate the effectiveness of data abstraction, we compare the metrics results to those obtained using clustering based on the original data sets comprising numerical data. The findings indicate that data abstraction primarily improves the three clustering validation metrics while delivering nearly comparable results regarding the Pearson correlation-based metric and comes with a substantially reduced runtime.

Monitoring the functionality of systems during operation is vital for detecting faults and preventing their consequences.In autonomous driving, monitoring is even more critical because of hardly being able to verify all implemented functionality.Today, systems comprise many interacting components making centralized monitoring less feasible and hard to handle.Hence, we suggest a distributed but connected monitoring system that reflects the system's conceptual structure.In this paper, we outline the foundations of a monitoring system, present some applications and show how we use concepts like the operational design domain and requirements for obtaining the required monitoring knowledge in the application area of autonomous driving.

More automated and autonomous systems are becoming daily use that implements safety–critical functions, e.g., autonomous driving or mobile robots. Testing such systems people depend on is challenging because some environmental interactions may not be expected during development but occur when those systems are in operation. Deciding when to stop testing or answering how to ensure sufficient testing is challenging and very expensive. For generating critical environmental interactions, i.e., critical scenarios, we present and compare two testing solutions focusing on generating critical scenarios utilizing combinatorial and search-based testing, respectively. For combinatorial testing, we suggest using ontologies that describe the environment of an autonomous or highly automated system. For search-based testing, we rely on genetic algorithms. We experimentally compared the two testing approaches using two implementations of an industrial emergency braking function and random testing as the baseline. Furthermore, we compared the approaches qualitatively using several categories. From the experiments, we see that the combinatorial testing approach can find all different types of faults listed in Table 5 considering a combinatorial strength of 3. This is not the case for search-based and random testing in all experiments. Combinatorial testing comes with the highest combinatorial coverage. However, all approaches can reveal faulty behavior utilizing appropriate environmental models. We present the results of an in-depth comparison of combinatorial and search-based testing. The be as fair as possible, the comparison relied on the same environmental model and other parameters like the number of generated test cases. The results show that combinatorial testing comes with the highest coverage and can find all different kinds of failures summarized in Table 5 providing a certain strength. Meanwhile, search-based testing is also capable of finding different failures depending on the coverage it can reach. Both approaches seem complementary and of use for the application domain of autonomous and automated driving functions.

A variety of automated software fault prediction techniques was proposed in recent years, in particular for the important class of spreadsheet programs. Software fault prediction techniques commonly create ranked lists of “suspicious” program statements for developers to inspect. Existing research, however, suggests that solely providing such ranked lists may not always be effective. In particular, it was found that developers often seek for explanations for the outcomes provided by a debugging tool and that such explanations may be key for developers to trust and rely on the tool. Research on how to explain the outcomes of fault prediction techniques, which are often based on complex machine learning models, is scarce, and little is known regarding how such explanations are perceived by developers. With this work, we aim to narrow this research gap and study the perception of different forms of explanations by spreadsheet users in the context of a machine learning based fault prediction tool. A between-subjects user study (N=120) revealed significant differences between the explored explanation styles. In particular, we found that well-designed natural language explanations can indeed help users better understand why certain spreadsheet cells were marked by the debugging tool and that such explanations can be effective to increase the users’ trust compared to a black box system. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Scenario-based approaches have been receiving a huge amount of attention in research and engineering of automated driving systems. Due to the complexity and uncertainty of the driving environment, and the complexity of the driving task itself, the number of possible driving scenarios that an Automated Driving System or Advanced Driving-Assistance System may encounter is virtually infinite. Therefore it is essential to be able to reason about the identification of scenarios and in particular critical ones that may impose unacceptable risk if not considered. Critical scenarios are particularly important to support design, verification and validation efforts, and as a basis for a safety case. In this paper, we present the results of a systematic mapping study in the context of autonomous driving. The main contributions are: (i) introducing a comprehensive taxonomy for critical scenario identification methods; (ii) giving an overview of the state-of-the-art research based on the taxonomy encompassing 86 papers between 2017 and 2020; and (iii) identifying open issues and directions for further research. The provided taxonomy comprises three main perspectives encompassing the problem definition (the why), the solution (the methods to derive scenarios), and the assessment of the established scenarios. In addition, we discuss open research issues considering the perspectives of coverage, practicability, and scenario space explosion.

As the size of software projects increases, their test suites usually grow accordingly. Test suite size, however, has a direct impact on the efficiency of software testing. Hence, test suite reduction (TSR) procedures aim at removing redundant test cases while maintaining the suites fault detection capabilities (FDC). This paper explores checked coverage as a coverage metric for TSR; checked coverage not only investigates if a part of code was executed but also if it was checked by a test oracle. Previously, this metric has been applied successfully as an indicator for oracle quality. To assess how suitable checked coverage is in comparison to traditional metrics, such as line or method coverage, we developed a TSR tool for Java programs. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open-source Java projects. Our study investigates both the efficiency of the TSR as well as effectiveness in regard to the FDC and size of the reduced test suites.

Faults in spreadsheets are not uncommon and they can have significant negative consequences in practice. Various approaches for fault localization were proposed in recent years, among them techniques that transferred ideas from spectrum-based fault localization (SFL) to the spreadsheet domain. Applying SFL to spreadsheets proved to be effective, but has certain limitations. Specifically, the constrained computational structures of spreadsheets may lead to large sets of cells that have the same assumed fault probability according to SFL and thus have to be inspected manually. In this work, we propose to combine SFL with a fault prediction method based on spreadsheet metrics in a machine learning (ML) approach. In particular, we train supervised ML models using two orthogonal types of features: (i) variables that are used to compute similarity coefficients in SFL and (ii) spreadsheet metrics that have shown to be good predictors for faulty formulas in previous work. Experiments with a widely-used corpus of faulty spreadsheets indicate that the combined model helps to significantly improve fault localization performance in terms of wasted effort and accuracy.

The testing objective is to find interactions with a system under test leading to unexpected behavior. Such interactions are test cases that can be either manually specified or automatically generated. For the latter, we find many methods and techniques in the research literature, including combinatorial testing or model-based testing. In this paper, we focus on automated test case generation based on models where we are interested in extracting models from available data. In particular, we consider automotive testing, where cars and other vehicles must behave correctly in typical driving situations. The idea is to use available driving data from which we want to extract driving models that we can later use for generating test cases, i.e., arbitrary driving patterns for vehicle testing. Besides outlining the foundations, we discuss the first experimental results we obtain using available open-access driving data.

About two decades ago, two algorithms, i.e., DDMin and QuickXPlain, for minimizing collections, were independently proposed and gained attention in the two research areas of Software Engineering and Artificial Intelligence, respectively.Whereas DDMin was developed for reducing a given test case, QuickXPlain was intended to be used for obtaining minimal conflicts efficiently.In this paper, we compare the performance of both algorithms with respect to their capabilities of minimizing collections.We found out that one algorithm outperforms the other under given prerequisites and vice versa.These findings help to select the suitable algorithm for a given task.Index Terms-test case minimization, conflict minimization, software testing, application to diagnosis and configuration

Abstract Diagnosis, i.e., the detection and identification of faults, provides the basis for bringing systems back to normal operation in case of a fault. Diagnosis is a very important task of our daily live, assuring safe and reliable behavior of systems. The automation of diagnosis has been a successful research topic for several decades. However, there are limitations due to complexity issues and lack of expressiveness of the underlying reasoning mechanisms. More recently logic reasoning like answer set programming has gained a lot of attention and practical use. In this paper, we tackle the question whether answer set programming can be used for automating diagnosis, focusing on industrial applications. We discuss a formalization of the diagnosis problem based on answer set programming, introduce a general framework for modeling systems, and present experimental results of an answer set programming based diagnosis algorithm. Past limitations like not being able to deal with numerical operations for modeling can be solved to some extent. The experimental results indicate that answer set programming is efficient enough for being used in diagnosis applications, providing that the underlying system is of moderate size. For digital circuits having less than 500 components, diagnosis time has been less than one second even for computing triple fault diagnoses.

Spreadsheets are widely used in organizations for various purposes such as data aggregation, reporting and decision-making. Since spreadsheets, like other types of software, can contain faulty formulas, it is important to provide developers with appropriate methods to find and fix such faults. Recently, various heuristic and statistics-based fault identification methods were proposed, which point developers to potentially faulty parts of the spreadsheets. Due to their heuristic nature, these methods might, however, miss some faults. As a result, if spreadsheet developers rely too strongly on these methods, they might not pay sufficient attention to problems that are not pinpointed by the methods. In this research, we are the first to study this potential problem of over-reliance in spreadsheet debugging, which may lead to limited debugging effectiveness. We report the outcome of a controlled experiment where 59 participants were tasked to find faulty formulas in a given spreadsheet with and without support of a novel spreadsheet debugging tool. Our results indicate that tool over-reliance can indeed result as a phenomenon of using heuristic debugging techniques. However, the study also provides evidence that making users aware of potential tool limitations within the debugging environment may help to address this problem. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Systems exhibiting anticipatory behavior are controlling devices that are influencing decisions critical to business with increasing frequency, but testing such systems has received little attention from the artificial intelligence or software engineering communities. In this article, we describe research activities being carried out to test anticipatory systems and explore how this research contributes to the body of knowledge. In addition, we review the types of addressed anticipatory applications and point out open issues and trends. This systematic mapping study was conducted to classify and analyze the literature on testing anticipatory systems, enabling us to highlight the most relevant topics and potential gaps in this field. We identified 206 studies that contribute to the testing of systems that exhibit anticipatory behavior. The papers address testing at stages such as context sensing, inferring higher-level concepts from the sensed data, predicting the future context, and intelligent decision-making. We also identified agent testing as a trend, among others. The existing literature on testing anticipatory systems has originated from various research communities, such as those on autonomous agents and quality engineering. Although researchers have recently exhibited increasing interest in testing anticipatory systems, theoretical knowledge about testing such systems is lacking.

Vision systems, i.e., systems that enable the detection and tracking of objects in images, have gained substantial importance over the past decades. They are used in quality assurance applications, e.g., for finding surface defects in products during manufacturing, surveillance, but also automated driving, requiring reliable behavior. Interestingly, there is only little work on quality assurance and especially testing of vision systems in general. In this paper, we contribute to the area of testing vision software, and present a framework for the automated generation of tests for systems based on vision and image recognition with the focus on easy usage, uniform usability and expandability. The framework makes use of existing libraries for modifying the original images and to obtain similarities between the original and modified images. We show how such a framework can be used for testing a particular industrial application on identifying defects on riblet surfaces and present preliminary results from the image classification domain.

Test automation, i.e., the generating and executing test cases without human intervention, is inevitable for today's agile software engineering processes. However, there are still areas where test automation is not applied due to various reasons like in the domain of digital signal processing and audio software. In this paper, we focus on test automation of audio plugins used in digital audio workstations. In particular, we present a framework behind an automatic regression testing tool, intended to support developers in the course of the continuous integration. The framework allows integrating different test case generation methods and supports regression testing by comparing the rendered audio output of two different audio plugin versions. The corresponding tool supports combinatorial and random testing. Furthermore, we present experimental results obtained making use of several real-world audio plugins. The experimental results show that random testing requires less test cases to reach the maximum achievable mutation score.

The security of cyber-physical systems is constantly threatened through cyber-attacks using available networking infrastructure. To prevent such attacks, anomaly-based intrusion detection systems are used in practice. Unfortunately, it is considered a hard task to detect the constantly improving attacks without prior knowledge of the attacks themselves. Hence, improved intrusion detection systems are of uttermost importance for preventing successful attacks of our today's network-based infrastructure. In this paper, we focus on improving intrusion detection systems. We build on former work on intrusion detection of power grid substation SCADA network traffic where a real-world data set is available. In contrast to previous work, we take imbalances of data used to learn attack patterns into account. Besides outlining the underlying foundations, the models used, and the experimental setup, we present and discuss the experimental results obtained using the available data set.

Software product metrics allow practitioners to improve their products and to optimize development processes based on quantifiable characteristics of source code. To facilitate similar benefits for spreadsheet programs, researchers proposed various product metrics for spreadsheets over the last decades. However, to our knowledge, no comprehensive overview of those efforts is currently available. In this paper, we close this gap by conducting a literature review of research works that either inherently or explicitly define product metrics for spreadsheets. We scanned five major digital libraries for scientific papers that define or use spreadsheet product metrics. Based on the identified 37 papers, we created a novel catalog of product metrics for spreadsheets. The catalog can be used by practitioners and researchers as a central reference for spreadsheet product metrics. In the paper, we (i) describe the proposed metrics in detail, (ii) report how often and for what purposes the metrics are used, (iii) identify significant discrepancies in the naming and definition of the metrics, and (iv) investigate how the appropriateness of the metrics was evaluated.

Electronic spreadsheets are widely used in organizations for various data analytics and decision-making tasks. Even though faults within such spreadsheets are common and can have significant negative consequences, today's tools for creating and handling spreadsheets provide limited support for fault detection, localization, and repair. Being able to predict whether a certain part of a spreadsheet is faulty or not is often central for the implementation of such supporting functionality. In this work, we propose a novel approach to fault prediction in spreadsheet formulas, which combines an extensive catalog of spreadsheet metrics with modern machine learning algorithms. An analysis of the individual metrics from our catalog reveals that they are generally suited to discover a wide range of faults. Their predictive power is, however, limited when considered in isolation. Therefore, in our approach we apply supervised learning algorithms to obtain fault predictors that utilize all data provided by multiple spreadsheet metrics from our catalog. Experiments on different datasets containing faulty spreadsheets show that particularly Random Forests classifiers are often effective. As a result, the proposed method is in many cases able to make highly accurate predictions whether a given formula of a spreadsheet is faulty.11.Results of a preliminary study were published in[1].

In Model-Based Diagnosis (MBD), we concern ourselves with the health and safety of physical and software systems. Although we often use different knowledge representations and algorithms, some tools like satisfiability (SAT) solvers and temporal logics, are used in both domains. In this paper we introduce Finite Trace Next Logic (FTNL) models of sequential circuits and propose an enhanced algorithm for computing minimal-cardinality diagnoses. Existing state-of-the-art satisfiability algorithms for minimal diagnosis use Sorting Networks (SNs) for constraining the cardinality of the diagnostic candidates. In our approach we exploit Multi-Operand Adders (MOAs). Based on extensive tests with ISCAS-89 circuits, we found that MOAs enable Conjunctive Normal Form (CNF) encodings that are significantly more compact. These encodings lead to 19.7 to 67.6 times fewer variables and 18.4 to 62 times fewer clauses. For converting an FTNL model to CNF, we could achieve a speed-up ranging from 6.2 to 22.2. Using SNs fosters 3.4 to 5.5 times faster on-line satisfiability checking though. This makes MOAs preferable for applications where RAM and off-line time are more limited than on-line CPU time.

Testing AI-based systems and especially when they rely on machine learning is considered a challenging task. In this paper, we contribute to this challenge considering testing neural networks utilizing mutation testing. A former paper focused on applying mutation testing to the configuration of neural networks leading to the conclusion that mutation testing can be effectively used. In this paper, we discuss a substantially extended empirical evaluation where we considered different test data and the source code of neural network implementations. In particular, we discuss whether a mutated neural network can be distinguished from the original one after learning, only considering a test evaluation. Unfortunately, this is rarely the case leading to a low mutation score. As a consequence, we see that the testing method, which works well at the configuration level of a neural network, is not sufficient to test neural network libraries requiring substantially more testing effort for assuring quality.

Abstract Abductive inference derives explanations for encountered anomalies and thus embodies a natural approach for diagnostic reasoning. Yet its computational complexity, which is inherent to the expressiveness of the underlying theory, remains a disadvantage. Even when restricting the representation to Horn formulae the problem is NP-complete. Hence, finding procedures that can efficiently solve abductive diagnosis problems is of particular interest from a research as well as practical point of view. In this paper, we aim at providing guidance on choosing an algorithm or tool when confronted with the issue of computing explanations in propositional logic-based abduction. Our focus lies on Horn representations, which provide a suitable language to describe most diagnostic scenarios. We illustrate abduction via two contrasting problem formulations: direct proof methods and conflict-driven techniques. While the former is based on determining logical consequences, the later searches for suitable refutations involving possible causes. To reveal runtime performance trends we conducted a case study, in which we compared publicly available general purpose tools, established Horn reasoning engines, as well as new variations of known methods as a means for abduction.

Abstract Web applications are deployed on machines around the globe and offer almost universal accessibility. These applications assure functional interconnectivity between different components on a 24/7 basis. One of the most important requirements is data confidentiality and secure authentication. However, implementation flaws and unfulfilled requirements often result in security leaks that malicious users eventually exploited. In this context, the application of different testing methods is of utmost importance in order to detect software defects during development and to prevent unauthorized access in advance. In this paper, we contribute to test automation for web applications. In particular, we focus on using planning for testing where we introduce underlying models covering attacks and their use in testing of web applications. The planning model offers a high degree of extendibility and configurability and as well overcomes limits of traditional graphical representations. New testing possibilities emerge that eventually lead to better vulnerability detection, therefore ensuring more secure web services and applications.

Abstract Having systems that can adapt themselves in case of faults or changing environmental conditions is of growing interest for industry and especially for the automotive industry considering autonomous driving. In autonomous driving, it is vital to have a system that is able to cope with faults in order to enable the system to reach a safe state. In this paper, we present an adaptive control method that can be used for this purpose. The method selects alternative actions so that given goal states can be reached, providing the availability of a certain degree of redundancy. The action selection is based on weight models that are adapted over time, capturing the success rate of certain actions. Besides the method, we present a Java implementation and its validation based on two case studies motivated by the requirements of the autonomous driving domain. We show that the presented approach is applicable both in case of environmental changes but also in case of faults occurring during operation. In the latter case, the methods provide an adaptive behavior very much close to the optimal selection.

In this paper, we outline the use of a genetic algorithm for test parameter optimization in the context of autonomous and automated driving. Our approach iteratively optimizes test parameters to aim at obtaining critical scenarios that form the basis for virtual verification and validation of Advanced Driver Assistant Systems (ADAS). We consider a test scenario to be critical if the underlying parameter set causes a malfunction of the system equipped with the ADAS function (i.e., near crash or crash of the vehicle). For evaluating the effectiveness of our approach, we set up an automated simulation framework, where we simulated the Euro NCAP car-to-car rear scenario. To assess the criticality of each test scenario we rely on time-to-collision (TTC), which is a well-known and often used time-based safety indicator for recognizing rear-end conflicts. Our genetic algorithm approach showed a higher chance to generate a critical scenario, compared to a random selection of test parameters.

Faults in spreadsheets can represent a major risk for businesses. To minimize such risks, various automated testing and debugging approaches for spreadsheets were proposed. In such approaches, often one main assumption is that the spreadsheet developer is able to indicate if the outcomes of certain calculations correspond to the intended values. This, however, might require that the user performs calculations manually, a process which can easily become tedious and error-prone for more complex spreadsheets. In this work, we propose an interactive spreadsheet algorithmic debugging method, which is based on partitioning the spreadsheet into fragments. Test cases can then be automatically or manually created for each of these smaller fragments, whose correctness or faultiness can be easier assessed by users than test cases that cover the entire spreadsheet. The annotated test cases are then fed into an algorithmic debugging technique, which returns a set of formulas that could have caused any observed failures, i.e., discrepancies between the expected and computed calculation outcomes. Simulation experiments demonstrate that the suggested decomposition approach can speed up the algorithmic debugging process and significantly reduce the number of fault candidates returned by the algorithm. An additional laboratory study shows that fragmenting a spreadsheet with our method furthermore reduces the time needed by users for creating test cases for a spreadsheet.

Feature engineering is a fundamental step in data mining and yet it is both difficult and expensive. Hand-crafting features is not only a time-consuming task that requires specific domain knowledge, it also may prevent new information to emerge. The extraction of meaningful features from relational data is particularly difficult due to complex relationships between tables. In the last decade there is an emerging trend towards automating the process of constructing propositional features from relational data and such approaches have been successfully used for solving numerous real-world problems. Despite their success, most of them lack an adequate support of nominal attributes. We present a new approach helping propositionalization methods to extract meaningful features from nominal attributes and improve their predictive performance. In an experimental evaluation on three datasets we demonstrate that the proposed technique is capable of producing novel features that are highly correlated with the target attribute. Furthermore, those features can reveal relationships among the distinct categorical values allowing to compare and order them. Finally, experimental results show that those new features can significantly improve the predictive performance in classification tasks.

Propositionalization has been proven to be a very effective solution for multi-relational data mining tasks.Traditional propositionalization approaches follow a two-step principle: transforming the relational data into a single, flat table and applying a propositional learning algorithm.During the transformation the target table gets expanded by adding many new features summarizing the information of the non-target tables.Based on the used feature construction strategy, this leads to a table of very high dimensionality with a lot of irrelevant and/or redundant features that has a negative effect on the predictive performance.In this paper, we propose an alternative propositionalization approach that evaluates the features already during the construction phase and reports only a subset of highly predictive features to the propositional learner.We present an implementation of this approach that adapts a state-ofthe-art propositionalization technique and combines it with a genetic algorithm to search for an optimal feature subset.Our experiments on a number of benchmark datasets reveal superior predictive performance of the approach compared to traditional two-step methods making it a considerable extension for any propositionalization algorithm.

The complexity of technical systems requires increasingly advanced fault diagnosis methods to ensure safety and reliability during operation. Particularly in domains where maintenance constitutes an extensive portion of the entire operation cost, efficient and effective failure identification holds the potential to provide large economic value. Abduction offers an intuitive concept for diagnostic reasoning relying on the notion of logical entailment. Nevertheless, abductive reasoning is an intractable problem and computing solutions for instances of reasonable size and complexity persists to pose a challenge. In this paper, we investigate algorithm selection as a mechanism to predict the “best” performing technique for a specific abduction scenario within the framework of model-based diagnosis. Based on a set of structural attributes extracted from the system models, our meta-approach trains a machine learning classifier that forecasts the most runtime efficient abduction technique given a new diagnosis problem. To assess the predictor’s selection capabilities and the suitability of the meta-approach in general, we conducted an empirical analysis featuring seven abductive reasoning approaches. The results obtained indicate that applying algorithm selection is competitive in comparison to always choosing a single abductive reasoning method.

Propositionalization has been proven to be a very effective solution for multi-relational data mining problems. The approaches usually follow a two-step principle: transforming the relational data into a single, flat table and applying a propositional learning algorithm. During the transformation, the target table gets expanded by adding many new features summarizing the information of the non-target tables. Based on the used feature construction strategy, this leads to a table of very high dimensionality with a lot of irrelevant and/or redundant features that can negatively affect the predictive performance. In this paper, we propose a modification of the traditional two-step framework to overcome such problems. The proposed approach evaluates the features during the construction phase and reports only a subset of highly predictive features to the propositional learner. We present an implementation of this approach using a genetic algorithm to search for an optimal feature subset. Our experiments on a number of benchmark datasets suggest that the modified framework can help propositionalization methods to significantly improve their predictive performance.

The diagnosis model is certainly a key element for any model-based diagnosis process. Experience shows though that in practice we often have no such model available for one or the other reason, so that in many projects we cannot draw on diagnosis processes when tackling problems. In this paper, we thus show how to improve on available automated processes for deriving a diagnostic model from standard simulation models as usually created during development. We delve in particular into the question how research in the context of combinatorial testing and fault injection can help in this respect, and consider several questions that arise.

Spreadsheets often contain faults that are difficult to localize. Spectrum-based Fault Localization (SFL) assists users in the fault localization process by ranking cells by their suspiciousness to contain a fault. Since the ranking of the basic SFL approach is often imprecise, we propose three techniques to improve it, i.e., dynamic cones, grouping, and tie-breaking. We evaluate these techniques with three spreadsheet corpora comprising more than 1,000 faulty spreadsheets of different size and structure. While dynamic cones do not come with large improvements, grouping and tie-breaking do. Grouping has a positive impact on about 50% of the spreadsheets with almost no negative effects. The same holds for tie-breaking, where some of the strategies offer high positive impact while keeping the risk of negative influences low.

Spreadsheets serve as a basis for decision-making processes in many companies and bugs in spreadsheets can therefore represent a considerable risk to businesses. Systematic tests can help to locate such bugs, but providing test cases can be cumbersome and complex for large real-world spreadsheets. To make the specification of test cases easier, we propose to split spreadsheets into smaller logically connected parts (called fragments) which can be individually tested for correctness. We present an algorithmic approach to compute such fragments, which we validated with a laboratory study in the form of a spreadsheet debugging exercise involving 57 subjects. The results show that the fragmentation approach can help to significantly reduce the required efforts to test a spreadsheet.

Fault prediction on high quality industry grade software often suffers from strong imbalanced class distribution due to a low bug rate. Previous work reports on low predictive performance, thus tuning parameters is required. As the State of the Art recommends sampling methods for imbalanced learning, we analyse effects when under- and oversampling the training data evaluated on seven different classification algorithms. Our results demonstrate settings to achieve higher performance values but the various classifiers are influenced in different ways. Furthermore, not all performance reports can be tuned at the same time.

Spreadsheets are the most prominent example of end-user programing, but they unfortunately are often erroneous, and thus, they compute wrong values. Localizing the true cause of such an observed misbehavior can be cumbersome and frustrating especially for large spreadsheets. Therefore, supporting techniques and tools for fault localization are highly required. Model-based software debugging (MBSD) is a well-known technique for fault localization in software written in imperative and object-oriented programing languages like C, C++, and Java. In this paper, we explain how to use MBSD for fault localization in spreadsheets and compare three types of models for MBSD, namely the value-based model (VBM), the dependency based model (DBM), and an improved version of the DBM. Whereas the VBM computes the lowest number of diagnoses, both DBMs convince by their low computational complexity. Hence, a combination of these two types of models is desired, and we present a solution that combines value-based and DBM in this paper. Moreover, we discuss a detailed evaluation of the models and the combined approach, which indicates that the combined approach computes the same number of diagnoses like the VBMs while requiring less computation time. Hence, the proposed approach is more appropriate to be used in tools for fault localization in spreadsheets.

Software fault prediction promises to be a powerful tool in supporting test engineers upon their decision where to define testing hotspots. However, there are limitations on a cross project prediction and a lack of reports upon application to industrial software, as well as the power of metrics to represent bugs. In this paper, we present a novel analysis based upon faults discovered in model-based automotive software projects and their relationship to metrics used to perform fault prediction. Using our previously released dataset on software metrics, we report bug classes discovered during heavy testing of those automotive software. As the software has been developed following strict coding and development guidelines, we present the results based on a comparison between the discovered error classes and those which might derive a reduced potential error set. Using the three projects from our dataset we determine if any of these bug classes are project specific.

Software fault localization, the act of identifying the locations of faults in a program, is widely recognized to be one of the most tedious, time consuming, and expensive - yet equally critical - activities in program debugging. Due to the increasing scale and complexity of software today, manually locating faults when failures occur is rapidly becoming infeasible, and consequently, there is a strong demand for techniques that can guide software developers to the locations of faults in a program with minimal human intervention. This demand in turn has fueled the proposal and development of a broad spectrum of fault localization techniques, each of which aims to streamline the fault localization process and make it more effective by attacking the problem in a unique way. In this article, we catalog and provide a comprehensive overview of such techniques and discuss key issues and concerns that are pertinent to software fault localization as a whole.

In this paper, we present a novel industry dataset on static software and change metrics for Matlab/Simulink models and their corresponding auto-generated C source code. The data set comprises data of three automotive projects developed and tested accordingly to industry standards and restrictive software development guidelines. We present some background information of the projects, the development process and the issue tracking as well as the creation steps of the dataset and the used tools during development. A specific highlight of the dataset is a low measurement error on change metrics because of the used issue tracking and commit policies.

Security testing of web applications remains a major problem of software engineering. In order to reveal vulnerabilities, manual and automatic testing approaches use different strategies for detection of certain kinds of inputs that might lead to a security breach. In this paper we compared a state-of-the-art manual testing tool with an automated one that is based on model-based testing. The first tool requires user input from the tester whereas the second one reduces the necessary amount of manual manipulation. Both approaches depend on the corresponding test case generation technique and its produced inputs are executed against the system under test (SUT). For this case we enhance a novel technique, which combines a combinatorial testing technique for input generation and a model-based technique for test execution. In this work the input parameter modelling is improved by adding constraints to generate more comprehensive and sophisticated testing inputs. The evaluated results indicate that both techniques succeed in detecting security leaks in web applications with different results, depending on the background logic of the testing approach. Last but not least, we claim that attack pattern-based combinatorial testing with constraints can be an alternative method for web application security testing, especially when we compare our method to other test generation techniques like fuzz testing.

Spreadsheets may be large, containing several thousand formulas, and thus they may be hard to comprehend and analyze. Unfortunately, they are also prone to errors. Identifying the cells which are responsible for an observed error is time-consuming, tedious, and frustrating. Spectrum-based Fault Localization (SFL) helps users to faster identify those cells that have to be modified in order to eliminate any observed misbehavior. SFL requires information about the correctness of certain cell values, and users might wrongly classify such cell values. A misclassification may influence the outcome of SFL substantially. In this paper, we investigate the influence of incorrect user information on the quality of SFL. In particular, we present a theoretical analysis of the impact of a misclassification on the Ochiai similarity coefficient and an empirical evaluation based on 33 spreadsheets with 218 faulty versions.

Spreadsheets are amongst the most successful examples of end user programming. Because of their, still increasing, importance for companies, spreadsheets have drastic economical and societal impact. Hence, locating and fixing spreadsheet faults is important and deserves attention from the research community. A state-of-the-art technique uses genetic programming for generating repair candidates, but a limitation that hinders real-world application is that it still computes too many repair candidates. In this paper, we discuss a novel technique based on constraint solving that uses distinguishing test cases to narrow down the number of repair candidates.

Locating faults in spreadsheets can be difficult. Therefore, tools supporting the localization of faults are needed. This paper presents a novel dependency-based model that can be used in Model-based software debugging (MBSD). This model allows improvements of the diagnostic accuracy while keeping the computation times short. In an empirical evaluation, we show that dependency-based models of spreadsheets whose value-based models are often not solvable in an acceptable amount of time can be solved in less than one second. Furthermore, the amount of diagnoses is reduced by 15 % on average when using the novel instead of the original dependency-based model.

Spreadsheets are by far the most used programs that are written by end-users. They often build the basis for decisions in companies and governmental organizations and therefore they have a high impact on our daily life. Ensuring correctness of spreadsheets is thus an important task. But what happens after detecting a faulty behavior? This question has not been sufficiently answered. Therefore, we focus on fault localization techniques for spreadsheets. In this paper, we introduce a novel dependency-based approach for model-based fault localization in spreadsheets. This approach improves diagnostic accuracy while keeping computation times short, thus making the automated fault localization more appropriate for practical applications. The presented approach allows for an acceptable fault localization time of less than a second, and reduces the number of computed root cause candidates by 15 % on average, when compared with another dependency-based approach.

SUMMARY Tools for automated fault localization usually generate too many bug candidates depending on the underlying technique. Hence, more information is required in order to further restrict the number of bug candidates. Approaches that rely on specific knowledge of the program to be debugged, such as variable values at specific positions in the source code, are not easily accessible for users especially in the case of software maintenance. In order to avoid this problem, we suggest integrating testing to restrict the number of bug candidates. In particular, we propose computing possible corrections of the program and from this distinguishing test cases. A distinguishing test case is a test that reveals different output values for two program variants given the same input values. Besides the formal definitions and algorithms, we extend the first empirical results of our approach in this paper. The obtained empirical results show that the use of mutations and distinguishing test cases reduces the number of bug candidates by more than 90% on average. Copyright © 2012 John Wiley & Sons, Ltd.

Automated debugging, i.e., localizing faults in programs automatically using a failing test case and the source code, is still and open research topic despite the fact that there have been many different techniques already published. In this paper we focus on constraints for debugging. Constraints cannot only be effectively used for verification and testing, but also provide a foundation for automated fault localization. We briefly discuss the basic idea of constraint based debugging and outline some of the most important challenges.

In this article we report on novel insights in model-based software debugging of hardware description languages (HDLs). Today's simulation driven working process emphasizes the need for exploiting test suites not only for detecting but also for localizing the root cause of misbehavior. We discuss the modeling approaches for the various artifacts of the Verilog hardware description language (blocking and non-blocking statements, expressions, execution ordering) and present a novel model incorporating test suites. The evaluation of our approach on the well-known ISCAS89 benchmarks concerning single and double-fault diagnoses clearly indicates that incorporating test suites into the fault localization technique (and development process) considerably improves the accuracy of the obtained diagnosis candidates.

There are two driving questions in software test automation: first, which tests to select out of a potentially infinite set of inputs, and second, whether the system under test exhibits an error when executing the chosen tests. Model-based testing (MBT) offers an answer to both of these questions. Models serve as a rich source of tests, and researchers have devised a wide range of techniques for selecting representative sets of tests. At the same time, the models encode the expected behaviour of the systems under test, and so the model answers the question of the expected output as well. The idea of MBT dates back to the 1970s, when people started using finite state machines for testing. Since then, an active international community has grown around this topic. Today, there are several workshops with years-long traditions (Workshop on Advances in Model-based Testing (A-MOST), Workshop on Model-based Testing (MBT), Workshop on Model-based Testing in Practice (MOTIP), Workshop on Model-Driven Engineering, Verification, and Validation (MoDeVVa), Workshop on Model-Based Verification and Validation (MVV), Model-Based Testing User Conference (MBTUC), etc.), papers on MBT appear at all major software engineering conferences and journals, there have been three Dagstuhl seminars dedicated to the topic, and several books on MBT have been published. Furthermore, reports of industrial success and companies that earn their living by providing MBT tools are clear indications that the topic has matured from a pure research topic into a successfully applied industrial technology. We received 29 submissions for this special issue. Reviewing followed the same process as for regular papers. Each paper was reviewed by at least three reviewers, and after a rigorous selection process requiring a total of 25 revisions and 100 reviews, nine papers remain for publication. These papers are spread across three special issues of STVR; this issue includes the first three papers, covering the foundations of MBT and offering an industrial perspective. The first paper, ‘A Taxonomy of Model-Based Testing Approaches’ by Mark Utting, Alexander Pretschner and Bruno Legeard, aims to organize a taxonomy describing the technologies and characteristics of the different MBT approaches that have been proposed over the years. This paper is based on a technical report from 2006, which is already very well known in the community and has 136 citations at the time of this writing, showing that it does an excellent job at capturing the essence of MBT. The second paper, ‘Obstacles and Opportunities in Deploying Model-Based GUI Testing of Mobile Software: A Survey’ by Marek Janicki, Mika Katara and Tuula Pääkkönen, investigates possible obstacles and opportunities towards wider deployment of MBT in industry. The paper is based on a survey conducted among engineers and managers involved in testing of mobile software that work in companies domiciled in Finland. The survey is based on a methodology called the TEMA toolset, which is introduced to the participants to give them an idea how MBT could work. Besides an evaluation of the TEMA toolset, the survey helps to identify the challenges in test automation, obstacles to MBT adoption, and identify metrics and reports that can convince managers to adopt MBT. The third paper, ‘Applying Formal Methods to PCEP: An Industrial Case Study from Modelling to Test Generation’ by Iksoon Hwang, Ana Cavalli, Mounir Lallali and Dominique Verchere, offers insights based on an industrial case study. In their study, the authors apply formal methods to model, verify and validate the Path Computation Element Communication Protocol. The protocol is modelled using the IF language, and test cases are produced using the automated test generation tool TestGen-IF. This article not only demonstrates the benefits of automated testing but also describes the authors’ experiences in identifying modelling errors—an aspect that should not be overlooked when applying MBT. All the contributions presented in this and the two upcoming special issues on MBT reveal the broad and dynamic MBT research community. This was possible through the combined efforts of all authors and reviewers, and we wish to thank them all for their time and energy. The following persons were the reviewers: Bernhard Aichernig, Andrea Arcuri, Phillips Aydal, Benoit Baudry, Fevzi Belli, Robert Binder, Paul Black, Gregor von Bochmann, Kirill Bogdanov, Sergiy Boroday, Fabrice Bouquet, Lydie du Bousquet, Harald Brandl, Mario Bravetti, Lionel Briand, José Campos, Ana Cavalli, Charles Colbourn, Mirko Conrad, Steve Counsell, Frederic Dadeau, Hyunsook Do, Juan Dueñas, Khaled El-Fakih, Phyllis Frankl, Angelo Gargantini, Sudipto Ghosh, Wolfgang Grieskamp, Mats Grindal, Florian Gross, Roland Groz, Qiang Guo, Atul Gupta, Mark Harman, Alan Hartman, Robert Hierons, Daniel Hoffman, Antti Huima, Florentin Ipate, Guy-Vincent Jourdan, AbdulSalam Kalaji, Gregory Kapfhammer, Mika Katara, Raimund Kirner, Pieter Koopman, Bogdan Korel, Willibald Krenn, Richard Kuhn, Victor Kuliamin, Raluca Lefticaru, Bruno Legeard, Johan Lilius, Michael Linschulte, Atif Memon, Mercedes Merayo, Marius Mikucionis, Tim Miller, Ralf Mitsching, Laurent Mounier, Henry Muccini, Brian Nielsen, Manuel Nunez, Ana Paiva, Amit Paradkar, Ioannis Parissis, Fabio Paternò, Patrizio Pelliccione, Alexandre Petrenko, Andrea Polini, Alexander Pretschner, Tuula Pääkkönen, Debra Richardson, Ismael Rodríguez, Gregg Rothermel, Vlad Rusu, Manoranjan Satpathy, Ina Schieferdecker, Rudolf Schlatte, Holger Schlingloff, Julien Schmaltz, José Silva, Adenilso Simao, Paul Strooper, Harold Thimbleby, Nikolai Tillmann, Paolo Tonella, Yves Le Traon, Guilherme Travassos, Jan Tretmans, Tugkan Tuglular, Hasan Ural, M. Uyar, Neil Walkinshaw, Martin Weiglhofer, Carsten Weise, Stephan Weissleder, Lee White, Marco Winckler, Jim Woodcock, Fatiha Zaïdi, and Tewfik Ziadi.

Frequent changes in logically coupled source files induced bugs in software. Metrics have been used to identify source files which are logically coupled. In this paper, we propose an approach to compute a set of eight metrics, which measure logical-couplings among source files. We compute these metrics using the historical data of software changes which are related to the fixing of post release bugs. To validate that our propose set of metrics is highly correlated with the number bugs and are more capable to construct a bug prediction model, we performed an experiment. Our experimental results show that our propose set of metrics is highly correlated with the number of bugs, and hence can be used to construct a bug prediction model. In our experiment, the obtained accuracy of our bug predictor model is 97%.

In case of resolved software change requests (SCRs), the names of impacted source files are known. In this paper, we tackle the question whether it is possible to use this information in order to predict the files that have to be changed whenever a new SCR is received. In order to provide a solution, we present two different approaches, which are based on automatic text classification of SCRs. First, we use Latent Semantic Indexing (LSI) to index the key terms of SCRs. Then, for classification we use two different approaches of machine learning i.e., single and multi label classification. We applied our approaches on the SCR's data of Gnome, Mozilla and Eclipse OSS projects. Our initial experimental results are promising, the obtained maximum precision values for single and multi label classification are 58.2% and 47.1% respectively. Furthermore, in case of single and multilabel classification, the maximum attained precision values for any individual label are 86.5% and 92% respectively.

Model-based testing is a well known technique that allows one to validate the correctness of software with respect to its model. If a lot of data is involved, symbolic techniques usually outperform explicit data enumeration. In this paper, we focus on a new symbolic test case generation technique. Our approach is based on symbolic execution and on satisfiability (modulo theory; SMT) solving. Our work was motivated by the complete failure of a well-known existing symbolic test case generator to produce any test cases for an industrial Session Initiation Protocol (SIP) implementation. Hence, we have replaced the BDD-based analysis of the existing tool with a combination of symbolic execution and SMT solving. Our new tool generates the test cases for SIP in seconds. However, further experiments showed that our approach is not a substitutive but a complementary approach: we present the technique and the results obtained for two protocol specifications, the first supporting our new technique, the second being witness for the classic BDD-technique.

Generating test input data is a complex task and nowadays mostly tackled with random approaches. Random testing of methods, which take non primitive data types as parameters, e.g. objects encapsulating database interactions, is a vain endeavor. Especially, if the precondition of the method under test (MUT) requires a particular object state of the method's parameters, random approaches rarely succeed. In this paper we present a technique to automatically synthesize implementations for the parameters of a MUT from a given Design by Contract specification. These implementations behave as described by the Design by Contract specification, but do not interact with their environment (e.g. database, network and file system). Furthermore, we can set the initial state of the synthesized implementations to the state required by the MUT's precondition. Besides a formal discussion of our approach we present results obtained by applying our technique to two case studies: a stack-based calculator and a real-world data collection tool from the telecommunication industry. The presented approach outperforms random data generation on both case studies in terms of amount of methods tested (function coverage) and line coverage.

Although decision trees are frequently used in environmental decision support systems they have shortcomings. In case of an available model decision trees have to be constructed manually from the model. Moreover, not all knowledge is represented in the decision tree. In order to overcome this issues we propose the use of abductive reasoning directly applied to the available cause-effect model. In particular we introduce the abduction problem, i.e., the problem of finding a cause for observed effects, show how this problem can be extended in order to allow distinguishing between competing explanations, and discuss the integration of testing and repair actions within the framework. The latter is especially important in case of environmental decision support systems.

A lot of information can be obtained from configuration management systems and post-release bug databases like Bugzilla. In this paper we focus on the question whether there are language specific bug patterns in large programs. For this purpose we implemented a system for extracting the necessary information from the Mozilla project files. A comparison of the extracted information with respect to the programming language showed that there are bug patterns specific to programming languages. In particular we found that Java files of the Mozilla project are less error prone than C and C++ files. Moreover, we found out that the bug lifetime when using Java was almost double the lifetime of bugs in C or C++ file.

A bug triage system is used for validation and allocation of bug reports to the most appropriate developers. An automatic bug triage system may reduce the software maintenance time and improve its quality by correct and timely assignment of new bug reports to the appropriate developers. In this paper, we present the techniques behind an automatic bug triage system, which is based on the categorization of bug reports. In order to obtain an automatic bug triage system we used these techniques and performed comparative experiments. We downloaded 1,983 resolved bug reports along with the developer activity data from the Mozilla open source project. We extracted the relevant features like report title, report summary etc., from each bug report, and extracted developer's name who resolved the bug reports from the developers activity data. We processed the extracted textual data, and obtained the term-to-document matrix using parsing, filtering and term weighting methods. For term weighting methods we used simple term frequency and TFtimesIDF (term frequency inverse document frequency) methods. Furthermore, we reduced the dimensionality of the obtained term-to-document matrix by applying feature selection and latent semantic indexing methods. Finally we used seven different machine learning methods for the classification of bug reports. The best obtained bug triage system is based on latent semantic indexing and support vector machine having 44.4% classification accuracy. The average precision and recall values are 30% and 28%, respectively.

Change management is a challenging task in software maintenance. Changes are made to the software during its whole life. Some of these changes introduce errors in the code which result in failures. Software changes are composed of small code units called hunks, dispersed in source code files. In this paper we present a technique for classifying software changes based on hunk metrics. We classify individual hunks as buggy or bug-free, thus we provide an approach for bug prediction at the smallest level of granularity. We introduce a set of hunk metrics and build classification models based on these metrics. Classification models are built using logistic regression and random forests. We evaluated the performance of our approach on 7 open source software projects. Our classification approach can classify hunks as buggy or bug free with 81 percent accuracy, 77 percent buggy hunk precision and 67 percent buggy hunk recall on average. Most of the hunk metrics are significant predictors of bugs but the set of significant metrics varies among different projects.

Model-based test case generation allows one to derive almost arbitrary numbers of test cases from models. If resulting test suites are executed against real implementations, there are often huge numbers of failed test cases. Thus, the analysis of the test execution, i.e. the identification of failures for error reporting, becomes a tedious and time consuming task. In this paper we investigate a technique for grouping test runs that most likely reveal the same failure. This reduces the post analysis time and enables the generation of small regression test suites. The test case grouping is implemented by means of spectrum-based fault localization at the level of the specification. We calculate the grouping by relating the spectra of the test cases. Besides a brief discussion of our approach we present results of applying our approach to the Session Initiation Protocol.

Automatic text classification of the software change request (CR) can be used for automating impact analysis, bug triage and effort estimation. In this paper, we focus on the automation of the process for assigning CRs to developers and present a solution that is based on automatic text classification of CRs. In addition our approach provides the list of source files, which are required to be modified and an estimate for the time required to resolve a given CR. To perform experiments, we downloaded the set of resolved CRs from the OSS project's repository for Mozilla. We labeled each CR with multiple labels i.e., the developer name, the list of source files, and the time spent to resolve the CR. To train the classifier, our approach applies the Problem Transformation and Algorithm Adaptation methods of multi-label machine learning to the multi-labeled CR data. With this approach, we have obtained precision levels up to 71.3% with 40.1% recall.

Abstract About a decade after the initial proposal to use model checkers for the generation of test cases we take a look at the results in this field of research. Model checkers are formal verification tools, capable of providing counterexamples to violated properties. Normally, these counterexamples are meant to guide an analyst when searching for the root cause of a property violation. They are, however, also very useful as test cases. Many different approaches have been presented, many problems have been solved, yet many issues remain. This survey paper reviews the state of the art in testing with model checkers. Copyright © 2008 John Wiley & Sons, Ltd.

Nondeterminism is used as a means of under specification or implementation choice in specifications, and it is often necessary if part of a system or the environment is unpredictable. The use of model-checker counterexamples as test-cases is a popular technique in model-based testing. Even though model-checkers can handle nondeterministic models for verification purposes, the use of nondeterministic models for test-case generation is not directly possible. A counterexample is an example execution path where alternative paths might also be valid. Consequently, testing could falsely identify correct implementations as erroneous. This paper describes how to use model-checkers to derive test-cases from nondeterministic models by applying postprocessing to the counterexamples. The influence of nondeterminism on coverage measurement with model-checkers is analyzed, and known coverage criteria are adapted. This is useful for the execution of test-cases on nondeterministic systems, where special treatment is necessary.

Modern drives of mobile robots are complex machines. Because of this complexity, as well as of wear and aging of components, faults occurs in such systems quite frequently at runtime. In order to use such drives in truly autonomous robots it is desirable that the robot is able to automatically react to such faults. Therefore, the robot needs reasoning and reconfiguration capabilities in order to be able to detect, localize and repair such faults on-line. In this paper we propose a model-based diagnosis and reconfiguration framework which allows an autonomous robot to detect and compensate faults in its drive. Moreover, we present an implementation for a real robot platform. Finally, we report experimental results which shows that the proposed framework is able to correctly cope with injected faults in the drive hardware, like broken motors.

This paper describes experiments with the freely available tool Helgrind, results obtained by using it for debugging a server application comprising 500 kLOC. We present improvements to the run time analysis of C++ programs that result in a dramatic reduction of false warnings.

Various research prototypes and a well-founded theory of model based testing (MBT) suggests the application of MBT to real-world problems. In this article we report on applying the well-known TGV tool for protocol conformance testing of a Session Initiation Protocol (SIP) server. Particularly, we discuss the performed abstractions along with corresponding rationales. Furthermore, we show how to use structural and fault-based techniques for test purpose design. We present first empirical results obtained from applying our test cases to a commercial implementation and to a popular open source implementation of a SIP Registrar. Notably, in both implementations our input output labeled transition system model proved successful in revealing severe violations of the protocol.

Automated software testing is an important measure to improve software quality and the efficiency of the software development process. We present a model-checker based approach to automated test-case generation applying mutation to behavioral models and requirements specifications. Unlike previous related approaches, the requirements specification is at the center of this process. A property coverage criterion is used to show that resulting test-cases sufficiently exercise all aspects of the specification. A test-suite derived from the specification can only be as good as the specification itself. We demonstrate that analysis of the testcase generation process reveals important details about the specification, such as vacuity and how much of the model it covers, without requiring additional costly computations.

Model-based diagnosis is a successjid AI technique for locating and identifying faults in technical syslems. Extending previous research on model-based diagnosis support for fault search in technical designs, we are building a model-based debugger for Java programs to provide intelligent support for the programmer trying to locate the so~rce of an error: By using one or more models derived from the source code of the program without additional spec$cations except the Java semantics, the debugger guides the user towards potential sources for incorrect program behaviors, i.e., bugs.

The state of the art in hardware design is the use of hardware description languages such as VHDL. The designs are tested by simulating them and comparing their output to that prescribed by the specification. A significant part of the design effort is spent on detecting unacceptable deviations from this specification and subsequently localizing the sources of such faults. In this paper, we describe an approach to employ model-based diagnosis for fault detection and localization in very large VHDL programs, by automatically generating the diagnosis model from the VHDL code and using observations about the program behavior to derive possible fault locations from the model. In order to achieve sufficient performance for practical applicability, we have developed a representation that provides a highly abstracted view of programs and faults, but is sufficiently detailed to yield substantial reductions in the fault localization costs when compared to the current manpower-intensive approach. The implementation in conjunction with the knowledge representation is designed with openness in mind in order to facilitate use of the highly optimized simulation tools available.