In recent years, the significance of test logs in ensuring system reliability and diagnosing runtime events has grown significantly, particularly with software expanding into various domains, necessitating rigorous verification and validation processes. However, the complexity and cost of testing have prompted a shift towards automation. This paper addresses the challenges of automated software testing through root-cause event detection. The proposed approach initially involves parsing and partitioning logs, followed by representing test events as dense vectors in a continuous space, enabling the capture of semantic similarities and relationships among events based on their sequence positions. Subsequently, test events are clustered in this embedded space, and each log partition is represented as a vector, with its characteristics reflecting the number of events in the log partition present in the clusters. Through two distinct case studies, we demonstrate that the final clustering of log partitions in this new space efficiently identifies root cause events. We evaluate our approach on two applications and anticipate its contribution as a cornerstone for future research and deployment of automated log mining.

Telemetry data (e.g.: CPU and memory usage) is an essential source of information for a software system that projects the system’s health. Anomalies in telemetry data warn system administrators about an imminent failure or deterioration of service quality. However, input events to the system (such as service requests) are the cause of abnormal system behaviour and, thus, anomalous telemetry data. By observing input events, one might predict anomalies even before they appear in telemetry data, thus giving the system administrator even earlier warning before the failure. Finding a correlation between input events and anomalies in telemetry data is challenging in many cases. This paper proposes a machine learning approach to learn the causality correlation between input event sequences and telemetry data. To this aim, a Natural Language Processing(NLP) approach is employed to create a concept space model to distinguish between normal and abnormal test sequences. Based on a vectorized representation of each input sequence, the concept space indicates whether the sequence will cause a system failure. Since the meaning of fault is not established in system status Telemetry-based fault detection, the suggested technique first detects periods of time when a software system status encounters aberrant situations (Bug-Zones). An extensive study on a real-world database acquired by a telecommunication operator and an open-source microservice software demonstrates that our approach achieves 71% and 90% accuracy as a Bug-Zones predictor.

Industrial Control Systems are found often in industrial sectors and critical infrastructures to monitor and control industrial processes. Recently, the security of industrial control systems has gained much attention as these systems now exhibit an increased interaction with the Internet. In fact, classical SCADA systems are already lacking with security problems, and with the increased interconnectivity to the Internet, they are now exposed to new types of threats and cyber-attacks. Intrusion detection technology is one of the most important security solutions used today in industrial control systems to detect potential attacks and malicious activities. This paper summarizes previous work for Intrusion Detection Systems approaches in Industrial Control Systems and highlights challenges and opportunities in implementing such solutions. We believe that such insights are valuable for further research in the industrial security context.

Efficient detection and instrumentation of function calls is fundamental for a variety of dynamic analysis techniques, including dynamic callgraph construction, control-flow integrity, and automatic vulnerability discovery. A common way of detecting calls at the machine code level is to look for CALL instructions. However, optimizing compilers frequently implement function tail calls with JMP instructions instead, and distinguishing an intra-procedural jump from a JMP-based function call is not straightforward. Despite the importance of making this distinction, prior research has not produced a reliable solution. In this paper, we address the problem of dynamic function call detection in real-time. We propose a heuristic-based approach named iCi to efficiently and automatically instrument calls, including conventional CALLs and JMP-based calls, at runtime. iCi does not rely on source code, debug information, symbol tables or static analysis. We show that iCi achieves an f-score of 0.95 in the worst case, regardless of optimization level. We open-source our implementation as well as the oracle we used for our evaluation.1

Fuzz testing consists in automatically generating and sending malicious inputs to an application in order to hopefully trigger a vulnerability. Fuzzing entails such questions as: Where to fuzz? Which parameter to fuzz? Where to observe its effects?

SUMMARY From experience in component‐based software engineering, it is known that the integration of high‐quality components may not yield high‐quality software systems. It is difficult to evaluate all possible interactions between the components in the system to uncover inter‐component misfunctions. The problem is even harder when the components are used without source code, specifications or formal models. Such components are called black boxes in literature. This paper presents an iterative approach of combining model learning and testing techniques for the formal analysis of a system of black‐box components. In the approach, individual components in the system are learned as finite state machines that (partially) model the behavioural structure of the components. The learned models are then used to derive tests for refining the partial models and/or finding integration faults in the system. The approach has been applied on case studies that have produced encouraging results. Copyright © 2013 John Wiley & Sons, Ltd.

Browsers have become an interface to perform a plethora of activities. This situation led to the integration of various software components in browsers, including interpreters for many web-friendly scripting languages e.g. JavaScript. In this article, we propose an application of genetic programming to the area of fuzzing the interpreters by generating codes that may trigger crashes and thereby indicating the presence of some hidden vulnerabilities. Based on our previous work on smart fuzzing with genetic approaches, we present here elements for an extension of the concept to fuzz browser interpreters.

This paper proposes a functional testing approach for safety properties formalized as hybrid automata. We first propose a formalism inspired from the concept of operational profile to specify test requirements for hybrid automata. We propose an associated parametric adequacy criterion that measures to what extent a given test suite satisfies these requirements. We also develop a set of hypothesis under which the proposed criterion can be evaluated when testing from a black box system when time is discretized and some signals of the automaton are not observable on the concrete system under test. We finally present the HyATT tool prototype that was developed to implement the proposed approach, and report practical feedback of applying it on a case study.

Fuzzing is one of the most popular test-based software vulnerability detection techniques. It consists in running the target application with dedicated inputs in order to exhibit potential failures that could be exploited by a malicious user. In this paper we propose a global approach for fuzzing, addressing the main challenges to be faced in an industrial context: large-size applications, without source code access, and with a partial knowledge of the input specifications. This approach integrates several successive steps, and we mostly focus here on an important one which relies on binary-level dynamic taint analysis. We summarize the main problems to be addressed in this step, and we detail the solution we implemented to solve them.

We present an approach to detect web injection vulnerabilities by generating test inputs using a combination of model inference and evolutionary fuzzing. Model inference is used to obtain a knowledge about the application behavior. Based on this understanding, inputs are generated using genetic algorithm (GA). GA uses the learned formal model to automatically generate inputs with better fitness values towards triggering an instance of the given vulnerability.

Nowadays, one of the most effective ways to identify software vulnerabilities by testing is the use of fuzzing, whereby the robustness of software is tested against invalid inputs that play on implementation limits or data boundaries. A high number of random combinations of such inputs are sent to the system through its interfaces. Although fuzzing is a fast technique which detects real errors, its efficiency should be improved. Indeed, the main drawbacks of fuzz testing are its poor coverage which involves missing many errors, and the quality of tests. Enhancing fuzzing with advanced approaches such as: data tainting and coverage analysis would improve its efficiency and make it smarter. This paper will present an idea on how these techniques when combined give better error detection by iteratively guiding executions and generating the most pertinent test cases able to trigger potential vulnerabilities and maximize the coverage of testing.

Security policy testing is a practical way to ensure security policies are correctly implemented in information or networking systems with a certain level of confidence. In this paper, we adapt model based testing techniques for formal models of security policies, and propose a two stage approach to produce test cases from a security policy specified in Or-BAC, i.e., test purpose generation from Or-BAC rules, and test case generation from test purposes.

Although many of the software engineering activities can now be model-supported, the model is often missing in software development. We are interested in retrieving state- machine models from black-box software components. We assume that the details of the development process of such components (third-party software or COTS) are not available. To adequately support software engineering activities, we need to learn more complex models than simple automata. Our model is an extension of finite state machines that incorporates the notions of predicates and parameters on transitions. We argue that such a model can offer a suitable trade-off between expressivity of the model and complexity of model learning. We have been able to extend polynomial learning algorithms to extract such models in an incremental testing approach. In turn, the models can be used to derive tests or for component documentation.

We investigate the problem of configuration verification for the extended FSM (EFSM) model. This is an extension of the FSM state identification problem. Specifically, given a configuration ("state vector") and an arbitrary set of configurations, determine an input sequence such that the EFSM in the given configuration produces an output sequence different from that of the configurations in the given set or at least in a maximal proper subset. Such a sequence can be used in a test case to confirm the destination configuration of a particular EFSM transition. We demonstrate that this problem could be reduced to the EFSM traversal problem, so that the existing methods and tools developed in the context of model checking become applicable. We introduce notions of EFSM projections and products and, based on these notions, we develop a theoretical framework for determining configuration-confirming sequences. The proposed approach is illustrated on a realistic example.

The use of formal description techniques allows the partial automation of the design, the validation, and the implementation of communication protocols and distributed algorithms. In this paper, we present a methodology for validation of distributed algorithms and protocols, and our experiences of using the Estelle language, and a simulation and validation tool, called Veda, to simulate and validate complex distributed algorithms for the distributed implementation of multi-rendezvous. Some design errors in published distributed rendezvous algorithms were found. We obtain from these experiences heuristic guidelines for trouble shooting of distributed algorithms.

The development of a simulator, called Veda, is described. Veda is a software tool to help designers in protocol modeling and validation. It is oriented towards the rapid prototyping of distributed algorithms. Algorithms are described using an ISO (International Organisation for Standardization) formal description technique, called Estelle. The development of Veda and its internal structure is presented, emphasizing the use of Prolog as a software engineering tool. Typical uses of Veda are discussed.>