Abstract Large Language Models (LLMs) are increasingly being used in everyday software engineering tasks, particularly in automated code generation. Despite their widespread adoption, these models remain far from perfect, making systematic and fair evaluation essential to understand their strengths and limitations. In the context of code generation, existing benchmarks are limited: they often target a single programming language and rely primarily on unit test outcomes, while overlooking other critical dimensions such as the overall quality of the generated code and its closeness to a valid solution. To address these gaps, we introduce , an extensible benchmark framework that, unlike prior work, establishes a systematic structure built on diverse and well-defined metrics, representative workloads, varied prompt templates, and a robust experimental procedure. In practice, the code generated by the LLMs is evaluated along three complementary dimensions: functional correctness, proximity to valid solutions, and code quality, enabling a comprehensive assessment of performance. We use to evaluate four open-source and two proprietary models under three prompting strategies across five programming languages. We further complement this analysis with a study of common errors in the code and provide concrete examples, offering clearer insight into where LLMs tend to struggle. Our findings show that, while LLMs achieve promising results, they struggle with harder problems and, in the case of smaller models, with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.

Internet of Things (IoT) gateways play a critical role in interconnecting heterogeneous devices, local networks, and cloud infrastructures. Vulnerabilities affecting these devices pose significant security risks. Despite this relevance, existing studies primarily focus on generic IoT assets, leading to high false-positive rates, incomplete coverage, and limited insight into root causes of vulnerabilities in IoT gateways. This paper presents a comprehensive empirical study of security vulnerabilities in IoT gateway software, supported by a systematic methodology for asset identification and validation. This work offers the first curated dataset focused exclusively on IoT gateways, comprising 483 validated Common Vulnerabilities and Exposures (CVEs) obtained through iterative glossary refinement, automated vulnerability collection, and expert adjudication supported by explicit scope criteria and a full-consensus acceptance protocol. The dataset is mapped to the Common Weakness Enumeration (CWE)-1000 Research View, enabling hierarchical root-cause traversal for 80.3% of the vulnerabilities. The empirical analysis reveals a high-risk vulnerability profile, with most issues affecting operating-system-level components and scoring High or Critical severity. The results identify Improper Control of Resources (CWE-664), Improper Neutralization (CWE-707), and Improper Access Control (CWE-284) as the dominant root causes across IoT gateways. A qualitative decomposition of these three pillars reveals recurring concrete weaknesses (buffer overflows, OS command injection, and missing authentication for critical functions) that individually account for the majority of high-severity exploitation outcomes. These findings reveal systemic weaknesses in gateway software stacks and development practices, offering insights to improve the architectural resilience and secure design of IoT gateway platforms.

The Internet of Things (IoT) has rapidly expanded across multiple sectors, exposing significant opportunities but also raising important concerns. This rapid growth has raised concerns about the security of IoT devices and the protection of the large volumes of data they collect, transmit, store, and process. Numerous large-scale attacks on IoT systems underscore the need for security measures, as well as comprehensive security assessments and benchmarking methods to verify and validate these systems. We conduct a Systematic Literature Review (SLR) to analyze previous studies, methodologies, and tools used to assess and benchmark the security of IoT systems, and to identify critical challenges and gaps in the existing literature. As a result, we highlight that, due to their complexity, IoT systems lack a comprehensive security framework that covers all layers and their security concerns. Despite awareness of known vulnerabilities, there is a lack of best practices, tools, and techniques to prevent, detect, and mitigate threats effectively. The absence of standardized security benchmarks complicates the evaluation and comparison of the solutions. There is also limited alignment with emerging standards such as ISO/IEC 27402 and SESIP. Finally, it is noteworthy that IoT gateway security remains unexplored despite its critical role in IoT ecosystems. CCS Concepts: • Computer systems organization → Embedded systems ; Redundancy ; Robotics; • Networks → Network reliability.

Open vSwitch is a software implementation of a multilayer switch designed for virtualized environments. Its architecture includes components in both user and kernel space. Although Open vSwitch is considered to be a mature project and has been widely adopted, its robustness has never been publicly assessed. While previous works focused on performance, in this work, we investigate the robustness of a fundamental component of Open vSwitch, the kernel module. The approach is based on injecting faults into the control plane interface of the Open vSwitch kernel module - which is based on Netlink sockets. We systematically tested all Generic Netlink families implemented by Open vSwitch and their respective commands and attributes across four different Linux kernel versions. Results reveal a plethora of failures and clear indications of inconsistencies in the handling of faulty inputs.

Large Language Models (LLMs) have become increasingly popular for text-to-code generation, a task that involves converting natural language descriptions into code. While they have shown promising results, they are still flawed, especially when dealing with complex problems, making it crucial to have a deep understanding of their strengths and limitations. However, current evaluations are often limited, targeting only a single programming language, considering a small set of related models, and focusing heavily on execution-based metrics such as unit test pass rates. Moreover, they tend to overlook deeper issues like code quality, recurring mistakes, and underlying patterns in the generated code. To address these gaps and properly assess how effective LLMs are at generating quality code, we conduct a comprehensive evaluation of several LLMs across Python and C++ using a large, diverse dataset that spans multiple difficulty levels. Our evaluation combines execution-based and static analysis metrics, enabling us to assess both functional correctness and the structural quality of the code. Unlike prior work, our study also includes an in-depth analysis of the generated code, uncovering common mistakes and failure modes, allowing for a more complete and differentiated view of model capabilities. Results show that, despite their potential, open-source LLMs still struggle with complex tasks and make recurring systematic errors such as missing import statements and incorrect variable scoping.

Security is one of the most critical requirements for modern digital systems. As the paradigm shifts from attempting to develop fully secure systems to designing resilient strategies that detect, respond to, and recover from attacks, Intrusion Detection Systems (IDS) become indispensable. However, developing robust IDS that address sophisticated attacks—especially in scenarios such as Cloud services, IoT, edge computing, and microservices, remains a significant challenge. Among these, containerized services present unique security challenges due to their architecture, deployment methods, and reliance on shared resources. On the other hand, Machine Learning (ML) offers promising, but not yet fully understood, solutions to enable automated, scalable, and adaptive intrusion detection mechanisms. In this paper, we study the applicability of a ML-based approach to enhance intrusion detection in containerized services by training and testing various ML algorithms on system call data, a commonly used data type in intrusion detection. Furthermore, we propose a novel graph-based representation for system calls that preserves critical relationships and contextual information between system calls. With this improved representation, we achieve enhancements in intrusion detection performance, including an increase in detection rates by at least 193% for the tested vulnerabilities while maintaining false alarms at a safer threshold, below a mean of 0.4% to maximize attack identification while minimizing false alarms we also incorporate a post-processing phase using a sliding window technique. This work not only addresses the challenges of securing containerized environments but also provides a robust framework for leveraging machine learning to build next-generation IDS.

Moving Target Defense (MTD) consists of applying dynamic reconfiguration in the defensive side of the attack-defense cybersecurity game. Virtual Machine (VM) migration could be used as MTD against specific host-based attacks in the cloud computing environment by remapping the distribution of VMs in the existing physical hosts. This way, when the attacker’s VM is moved to a different machine, the attack has to be restarted. However, one significant gap here is how to select a proper VM migration-based MTD schedule to reach the desired levels of system protection. This paper develops a Stochastic Petri Net (SPN) model to address this issue. The model leverages empirical knowledge about the dynamics of the attack defense in a VM migration-enabled setup. First, we present the results of an experimental campaign to acquire knowledge about the system’s behavior. The experiments provide insights for the model design. Then, based on the model, we propose a tool named PyMTDEvaluator , which provides a graphical interface that serves as a wrapper for the simulation environment of the model. Finally, we exercise the tool using Multi-Criteria Decision-Making methods to aid the MTD policy selection. Hopefully, our results and methods will be helpful for system managers and cybersecurity professionals. • Comprehensive experimental results of VM migration as MTD against Memory DoS. • Availability and security Stochastic Petri Net of a system with VM migration as MTD. • An open-source simulation tool providing a graphical interface for the model. • Multi-Criteria Decision-Making to support MTD selection based on user preferences.

Microservice architectures offer scalability and flexibility, but due to their distributed nature and complex service structures, raise new security challenges, particularly in detecting DoS attacks. Although addressing these challenges calls for innovative attack detection approaches, developing effective solutions requires large-scale experiments and data collection to create representative datasets. This paper proposes a comprehensive framework to support research on the cybersecurity of microservice applications and the development of different methods to detect cyberattacks. The framework comprises two modules: (i) a data generation module that contains the components necessary to create datasets that reflect the behavior of microservices under attack and (ii) a model development and evaluation module suitable for different methods for detecting attacks on microservices. The framework is illustrated and validated by generating realistic high- and low-volume DoS attack data and developing models using supervised and unsupervised Machine Learning (ML) algorithms and a method based on Logic Scoring of Preference (LSP). The results indicate that supervised ML models have the best classification performance, especially with the XGBoost algorithm. Even though unsupervised ML and LSP models have worse performance, they can be used when the attack data are not available or are costly to generate.

Virtualization is drawing attention due to countless benefits, leaving Hypervisors with the paramount responsibility for performance, dependability, and security. However, while there are consolidated approaches to assessing the performance and dependability of virtualized systems, solutions to assess security are very limited. Key difficulties are evaluating the system in the presence of unknown attacks and vulnerabilities and comparing the security attributes of different systems and configurations when an intrusion occurs. In this paper, we propose a novel concept and approach of intrusion injection for virtualized environments, which consists of directly driving the system into the erroneous states that mimic the ones resulting from actual intrusions (in the same way errors are injected to mimic the effects of residual faults). We present a prototype capable of injecting erroneous states related to memory-corruption in the Xen Hypervisor to show that the concept and approach proposed here are feasible. The prototype is evaluated using publicly disclosed exploits across three different versions of Xen. Results show that our tool can inject erroneous states equivalent to those resulting from attacks that exploit existing vulnerabilities, even on versions where those vulnerabilities do not exist.

Microservice architectures allow the development of highly scalable, flexible, and manageable systems. However, such architectures raise new security problems and exacerbate the challenge of monitoring applications at runtime due to their high service granularity and distributed nature. Developing effective monitoring and security strategies is thus crucial to effectively detect potential attacks. This paper explores the applicability of Logic Scoring of Preference (LSP), a multi-criteria decision-making method to compute a score based on a set of preferences, for attack detection in microservice applications. We present an extensive experimental study and define a model based on LSP and application-level metrics to characterize the impact of DoS attacks. The output of the model is a unique score used to determine whether a microservice is under a DoS attack. The results of the experimental study show precision, recall, and f1-score rates of more than 80%, indicating that LSP could effectively characterize the application under attack, opening several possibilities for future work.

Online Failure Prediction (OFP) is a technique that attempts to predict incoming failures to mitigate their consequences. Machine Learning (ML) has been successfully used to create predictive models for OFP, but failures are rare, and thus failure data are typically not available. Fault injection has been accepted as a viable alternative to generate failure data. However, this raises several challenges, such as how to process the data to create and assess predictive models. The characteristics of fault injection campaigns (e.g., repeated/controlled experiments) and OFP (e.g., autocorrelation) require specific considerations. This work proposes a six-stage methodology for using failure data generated through fault injection to create accurate/representative models for OFP. It overviews the various phases, from generating and processing the data, to creating and assessing the performance of the models up to their deployment, while considering the intrinsic characteristics of the problem. As a case study, we apply the methodology to develop failure predictors for the Linux Operating System (OS). Results show that the proposed methodology led to accurate predictive models that could also generalize to failures that occur under different execution profiles, whilst using traditional techniques resulted in over-optimistic observations.

The malicious exploitation of security flaws by attackers can lead to a range of problems. While several techniques and tools allow detecting vulnerabilities during the Software Development Life Cycle (SDLC), most face the challenge of generating many false alarms while failing to detect vulnerabilities. This leads software development teams to waste considerable time in the analysis and to deploy potentially vulnerable code. In this context, we believe that complementary solutions are needed to help teams focusing the vulnerability detection and refactoring efforts on the code units that are more prone to have security issues. In this paper, we propose SCOLP (Security Characterization of Open-Source Functions using Logic Scoring of Preference (LSP)), an approach based on Multi-Criteria Decision Making (MCDM), aimed at categorizing code units (functions) based on the perceived proneness to have security issues. In practice, we use static information (e.g., Software Metrics (SMs) and memory management-related attributes) collected from the source code to feed Quality Models (QMs) that output a score that is then used to categorize the code units. To demonstrate SCOLP, we developed several QMs and applied them to the functions of a large open-source project developed using the C language (Linux Kernel). A preliminary validation with security experts was conducted to assess the accuracy of the categorization. Despite the subjectiveness of the process, results show that the output of SCOLP is aligned with the view of security experts. Our technique can be easily integrated into the SDLC without adding overhead to the development processes.

Online Failure Prediction (OFP) allows proactively taking countermeasures before a failure occurs, such as saving data or restarting a system. However, despite its potential contribution to improving dependability, OFP still presents key limitations. Besides the problem of choosing the optimal set of features, assessing predictive models is complex and common procedures for supporting comparison are not available. There is, in fact, little work on developing and assessing failure predictors for complex systems. In this aricle, we present two extensive case studies on distinct Operating Systems (OSs), Linux and Windows, showing that it is possible to create models that can predict different types of incoming failures, highlighting various important considerations such as the operational requirements of the target system. To drive the case studies, we define a well-structured framework for a fair and sound assessment and comparison of alternative predictive solutions. It includes scenarios for choosing the most adequate metrics for the assessment, comparing alternative models, and selecting the best predictor, while considering the need to tolerate perturbations in the data. In practice, we show that, by following a well-defined process, it is possible to develop accurate failure predictors and establish a ranking of the models under evaluation in different scenarios and OSs.

The use of Virtual Machine (VM) migration as support for software rejuvenation was introduced more than a decade ago. Since then, several works have validated this approach from experimental and theoretical perspectives. Recently, some works shed light on the possibility of using the same technique as Moving Target Defense (MTD). However, to date, no work evaluated the availability and security levels while applying VM migration for both rejuvenation and MTD (multipurpose VM migration). In this paper, we conduct a comprehensive evaluation using Stochastic Petri Net (SPN) models to tackle this challenge. The evaluation covers the steady-state system availability, expected MTD protection, and related metrics of a system under time-based multipurpose VM migration. Results show that the availability and security improvement due to VM migration deployment surpasses 50% in the best scenarios. However, there is a trade-off between availability and security metrics, meaning that improving one implies compromising the other.

Automated tools, namely Static Analysis Tools (SATs) and Penetration Testing Tools, are frequently used by developers to detect vulnerabilities. However, research and practice show that the effectiveness of those tools in large-scale projects is low, being prone to both false positives and false negatives. Thus, there is an urgent need for more effective techniques, which ultimately require representative field data for driving their design and testing. In this paper, we present a dataset of vulnerabilities from five large open-source C/C++ projects: Mozilla, Linux Kernel, Xen, httpd, and Glibc. For collecting the data, we designed an automated process grounded on vulnerabilities collected from the Common Vulnerability and Exposures (CVE) Details website. For each vulnerability, we retrieve the corresponding source code units from the project repository (including both vulnerable and fixed versions). We then compute a large set of Software Metrics (SMs) for those code units and run two SATs to collect security alerts (i.e., potential vulnerabilities and/or weaknesses). The dataset currently includes 5214 vulnerabilities. To demonstrate its usefulness, we explore the use of the dataset to train machine learning models to detect vulnerable C/C++ functions. Results clearly show that the dataset can be used in practice and is a key contribution for researchers working in software security.

The benchmarking of security tools is endeavored to determine which tools are more suitable to detect system vulnerabilities or intrusions. The analysis process is usually oversimplified by employing just a single metric out of the large set of those available. Accordingly, the decision may be biased by not considering relevant information provided by neglected metrics. This article proposes a novel approach to take into account several metrics, different scenarios, and the advice of multiple experts. The proposal relies on experts quantifying the relative importance of each pair of metrics towards the requirements of a given scenario. Their judgments are aggregated using group decision making techniques, and pondered according to the familiarity of experts with the metrics and scenario, to compute a set of weights accounting for the relative importance of each metric. Then, weight-based multi-criteria-decision-making techniques can be used to rank the benchmarked tools. The usefulness of this approach is showed by analyzing two different sets of vulnerability and intrusion detection tools from the perspective of multiple/single metrics and different scenarios.

Memory Denial of Service (DoS) attacks are easy-to-launch, hard to detect, and significantly impact their targets. In memory DoS, the attacker targets the memory of his Virtual Machine (VM) and, due to hardware isolation issues, the attack affects the co-resident VMs. Theoretically, we can deploy VM migration as Moving Target Defense (MTD) against memory DoS. However, the current literature lacks empirical evidence supporting this hypothesis. Moreover, there is a need to evaluate how the VM migration timing impacts the potential MTD protection. This practical experience report presents an experiment on VM migration-based MTD against memory DoS. We evaluate the impact of memory DoS attacks in the context of two applications running in co-hosted VMs: machine learning and OLTP. The results highlight that the memory DoS attacks lead to more than 70% reduction in the applications' performance. Nevertheless, timely VM migrations can significantly mitigate the attack effects in both considered applications.

This paper presents PyMTDEvaluator, a tool for evaluating the effectiveness of time-based Moving Target Defense (MTD) against availability attacks (e.g., Denial of Service - DoS, resource starvation attacks). PyMTDEvaluator is based on simulation runs of an extended deterministic Stochastic Petri Net (SPN) and offers a user-friendly interface where it is possible to analyze and compare MTD policies with different parameters. The SPN design relies on knowledge obtained from empirical observation. PyMTDEvaluator provides results such as probability of attack success, availability, and system capacity to support MTD design decision making. The tool allows analyzing and comparing several scenarios in the same evaluation, thus enabling the study of the pros and cons of different MTD deployment alternatives. PyMTDEvaluator aims to be part of the toolset for MTD policies design. It is also valuable for sensitivity analysis of MTD-enabled system parameters.

The need for improving the security level of Internet-of-Things (IoT) systems is growing. Users may refrain from using such systems if they realize that security measures are not in place. In this context, one of the most important IoT components has not received the necessary attention by the community: the gateway. IoT gateways play a central role in an IoT system as they solve heterogeneity issues. However, if compromised, gateways can be a source of security threats, and these threats become more relevant due to the gateway's central role. Gateways connect IoT devices and cloud services, and successful attacks on this component may exploit this fact. Using a well-known security requirements (SRs) engineering approach, this article evaluates and prioritizes SRs specifically for IoT gateways. The prioritization highlights a set of SRs that must be observed when evaluating and improving the gateway security level. Also, current solutions are detailed to support the enforcement of such SRs. Finally, a set of open challenges are discussed to highlight current research gaps that must be addressed to support SRs.

Moving Target Defense (MTD) is a defensive mechanism based on dynamic system reconfiguration to prevent or thwart cyberattacks. In the last years, considerable progress has been made regarding MTD approaches for virtualized environments, and Virtual Machine (VM) migration is the core of most of these approaches. However, VM migration produces system downtime, meaning that each MTD reconfiguration affects system availability. Therefore, a method for a combined evaluation of availability and security is of utmost importance for VM migration-based MTD design. In this paper, we propose a Stochastic Reward Net (SRN) for the probability of attack success and availability evaluation of an MTD based on VM migration scheduling. We study the MTD system under different conditions regarding 1) VM migration scheduling, 2) VM migration failure probability, and 3) attack success rate. Our results highlight the tradeoff between availability and security when applying MTD based on VM migration. The approach and results may provide inputs for designing and evaluating MTD policies based on VM migration.

The growing complexity of software calls for innovative solutions that support the deployment of reliable and secure software. Machine Learning (ML) has shown its applicability to various complex problems and is frequently used in the dependability domain, both for supporting systems design and verification activities. However, using ML is complex and highly dependent on the problem in hand, increasing the probability of mistakes that compromise the results. In this paper, we introduce Propheticus, a ML framework that can be used to create predictive models for reliable and secure software systems. Propheticus attempts to abstract the complexity of ML whilst being easy to use and accommodating the needs of the users. To demonstrate its use, we present two case studies (vulnerability prediction and online failure prediction) that show how it can considerably ease and expedite a thorough ML workflow.

The popularity of web applications and their world-wide use to support business critical operations raised the interest of hackers on exploiting security vulnerabilities to perform malicious operations. Fostering trust calls for assessment techniques that provide indicators about the quality of a web application from a security perspective. This paper studies the problem of using coding practices to characterize the trustworthiness of web applications from a security perspective. The hypothesis is that applying feasible security practices results in applications having a reduced number of unknown vulnerabilities, and can therefore be considered more trustworthy. The proposed approach is instantiated for the concrete case of input validation practices, and includes a Quality Model to compute trustworthiness scores that can be used to compare different applications or different code elements in the same application. Experimental results show that the higher scores are obtained for more secure code, suggesting that it can be used in practice to characterize trustworthiness, also providing guidance to compare and/or improve the security of web applications.

The use of cryptography is nowadays common in software systems, with cryptographic libraries widely available to software developers. As such, the likely weakest link in sensitive software has moved from cryptographic function implementations to the application code surrounding such functions. Ordinary developers usually lack knowledge in practical cryptography, and support from specialists is rare. Frequently, these difficulties are addressed by running static analysis tools to automatically detect cryptography misuse during coding and reviews. However, the effectiveness of such tools is not yet well understood. This article studies how well programmatic misuse of cryptography is detected by free static code analysis tools. The performance of such tools in detecting misuse is correlated to coding tasks and use cases commonly found in development efforts; also, cryptography misuse is classified in comprehensive categories, easily recognizable by software security practitioners. Our research shows that the coverage of public-key cryptography by static code analysis tools is full of blind spots, because tools prioritize only those misuses related to the most frequent coding tasks and use cases, while neglecting infrequent use cases. We found that, in addition to a relatively low recall in our tests, evaluated tools also have a small overlap regarding the misuses detected by all the evaluated tools, as well as an intersection of false alarms, suggesting lack of discrimination between specific misuses and corresponding good uses of cryptography. In spite of that, well-selected tools can be useful when developing cryptographic software, but support of experts is still required for solving complex cases.

Trustworthiness is a paramount concern for users and customers in the selection of a software solution, specially in the context of complex and dynamic environments, such as Cloud and IoT. However, assessing and benchmarking trustworthiness (worthiness of software for being trusted) is a challenging task, mainly due to the variety of application scenarios (e.g., businesscritical, safety-critical), the large number of determinative quality attributes (e.g., security, performance), and last, but foremost, due to the subjective notion of trust and trustworthiness. In this paper, we present trustworthiness as a measurable notion in relative terms based on security attributes and propose an approach for the assessment and benchmarking of software. The main goal is to build a trustworthiness assessment model based on software metrics (e.g., Cyclomatic Complexity, CountLine, CBO) that can be used as indicators of software security. To demonstrate the proposed approach, we assessed and ranked several files and functions of the Mozilla Firefox project based on their trustworthiness score and conducted a survey among several software security experts in order to validate the obtained rank. Results show that our approach is able to provide a sound ranking of the benchmarked software.

Summary HTML is being massively used as an interface to provide services to users. Web developers are producing and changing sites at a high pace while trying to support the latest HTML standards. In this context, it is common to find websites that do not comply with the standards and fail to be correctly processed by browsers. Considering this dynamic environment and the increasingly large diversity of browsers with frequent updates, the appearance of problems in web pages is a common, sometimes severe, and hard‐to‐track problem. In this short communication, we describe the initial design of an approach that will be used to obtain information regarding the characteristics of HTML documents on the Web and extract indicators of representative errors made by their developers. Preliminary results show nearly 90% of the pages analyzed having at least one type of error and the prevalence of a small number of error types.

Static analysis tools are recurrently used by developers to search for vulnerabilities in the source code of web applications. However, distinct tools provide different results depending on factors such as the complexity of the code under analysis and the application scenario; thus, missing some of the vulnerabilities while reporting false problems. Benchmarks can be used to assess and compare different systems or components, however, existing benchmarks have strong representativeness limitations, disregarding the specificities of the environment, where the tools under benchmarking will be used. In this paper, we propose a benchmark for assessing and comparing static analysis tools in terms of their capability to detect security vulnerabilities. The benchmark considers four real-world development scenarios, including workloads composed of real web applications with different goals and constraints, ranging from low budget to high-end applications. Our benchmark was implemented and assessed experimentally using a set of 134 WordPress plugins, which served as the basis for the evaluation of five free PHP static analysis tools. Results clearly show that the best solution depends on the deployment scenario and class of vulnerability being detected; therefore, highlighting the importance of these aspects in the design of the benchmark and of future static analysis tools.

Data privacy is a key challenge that is exacerbated by Big Data storage and analytics processing requirements. Big Data and Cloud Computing are related and allow the users to access data from any device, making data privacy essential as the data sets are exposed through the web. Organizations care about data privacy as it directly affects the confidence that clients have that their personal data are safe. This paper presents a data privacy approach - PRIVAaaS - and its inte-gration to the LEMONADE Web-based platform, developed to compose ETL (Extract, Transform, Load) process and Machine Learning workflows. The 3-level approach of PRIVAaaS, based on data anonymization policies, is implemented in a software toolkit that provides a set of libraries and tools which allows controlling and reducing data leakage in the context of Big Data processing.

The web services technology has been created to support communication between heterogeneous platforms. Despite its maturity, built upon more than a decade of experience, research and practice show that the technology still fails to connect web service client applications to servers, even when the programming languages involved are the same. This is especially troubling for service providers, as a failure in the inter-operation of web services may lead to disastrous consequences for the services involved, which frequently support businesses. In this paper, we present INTENSE, a service deployed as an on-line web application, designed to test the interoperability of a web service against specific client-side platforms. The tool is able to test the pre-runtime steps involving code generation and the end-to-end runtime communication, present in a web service interaction with a client. We used INTENSE to test a set of web services deployed on Glassfish and WildFly against the well-known Metro JAX-WS, JBossWS, and Axis2 client platforms, which disclosed severe interoperability issues.

The incorrect use of cryptography is a common source of critical software vulnerabilities. As developers lack knowledge in applied cryptography and support from experts is scarce, this situation is frequently addressed by adopting static code analysis tools to automatically detect cryptography misuse during coding and reviews, even if the effectiveness of such tools is far from being well understood. This paper proposes a method for benchmarking static code analysis tools for the detection of cryptography misuse, and evaluates the method in a case study, with the goal of selecting the most adequate tools for specific development contexts. Our method classifies cryptography misuse in nine categories recognized by developers (weak cryptography, poor key management, bad randomness, etc.) and provides the workload, metrics and procedure needed for a fair assessment and comparison of tools. We found that all evaluated tools together detected only 35% of cryptography misuses in our tests. Furthermore, none of the evaluated tools detected insecure elliptic curves, weak parameters in key agreement, and most insecure configurations for RSA and ECDSA. This suggests cryptography misuse is underestimated by tool builders. Despite that, we show that it is possible to benefit from an adequate tool selection during the development of cryptographic software.

Detecting software security vulnerabilities and distinguishing vulnerable from non-vulnerable code is anything but simple. Most of the time, vulnerabilities remain undisclosed until they are exposed, for instance, by an attack during the software operational phase. Software metrics are widely-used indicators of software quality, but the question is whether they can be used to distinguish vulnerable software units from the non-vulnerable ones during development. In this paper, we perform an exploratory study on software metrics, their interdependency, and their relation with security vulnerabilities. We aim at understanding: i) the correlation between software architectural characteristics, represented in the form of software metrics, and the number of vulnerabilities; and ii) which are the most informative and discriminative metrics that allow identifying vulnerable units of code. To achieve these goals, we use, respectively, correlation coefficients and heuristic search techniques. Our analysis is carried out on a dataset that includes software metrics and reported security vulnerabilities, exposed by security attacks, for all functions, classes, and files of five widely used projects. Results show: i) a strong correlation between several project-level metrics and the number of vulnerabilities, ii) the possibility of using a group of metrics, at both file and function levels, to distinguish vulnerable and non-vulnerable code with a high level of accuracy.

Defence-in-depth is a term often used in security literature to denote architectures in which multiple security protection systems are deployed to defend the valuable assets of an organization (e.g. the data and the services). In this paper we present an approach for analysing defence-in-depth, and illustrate the use of the approach with an empirical study in which we have assessed the detection capabilities of intrusion detection systems when deployed in diverse, two-version, parallel defence-in-depth configurations. The configurations have been assessed in settings that favour detection of attacks (reducing false negatives), as well as settings that favour legitimate traffic (reducing false positives).

An increasingly important requirement for certain classes of software-intensive systems is the ability to self-adapt their structure and behavior at run-time when reacting to changes that may occur to the system, its environment, or its goals. A major challenge related to self-adaptive software systems is the ability to provide assurances of their resilience when facing changes. Since in these systems, the components that act as controllers of a target system incorporate highly complex software, there is the need to analyze the impact that controller failures might have on the services delivered by the system. In this paper, we present a novel approach for evaluating the resilience of self-adaptive software systems by applying robustness testing techniques to the controller to uncover failures that can affect system resilience. The approach for evaluating resilience, which is based on probabilistic model checking, quantifies the probability of satisfaction of system properties when the target system is subject to controller failures. The feasibility of the proposed approach is evaluated in the context of an industrial middleware system used to monitor and manage highly populated networks of devices, which was implemented using the Rainbow framework for architecture-based self-adaptation.

The papers in this special issue on security and dependability of cloud systems and services. Service-based cloud computing systems are used nowadays in many business- and mission-critical scenarios. As the service-oriented paradigm increasingly spreads in a wide range of application fields, including big data, cloud storage, mobile cloud computing, and sensor cloud, there is a growing need for sound methodologies, algorithms and techniques for building services in which companies, organizations and citizens can trust and rely upon. Security and dependability are therefore becoming more and more relevant concerns for such systems, whose complexity, heterogeneity, and fast-changing dynamics bring difficult challenges to the research and industry communities.

This is the second part of the special issue on security and dependability of cloud systems and services, which was organized to solicit novel results in these important and closely related research areas. Indeed, security and dependability are increasingly important concerns for cloud systems and services, due to their spread in a large variety of application fields, including business- and mission-critical scenarios. This demands for innovative methodologies, algorithms and techniques for building services in which companies, organizations and citizens can trust and rely upon.

With the widespread adoption of virtualization, intrusion detection systems (IDSes) are increasingly being deployed in virtualized environments. When securing an environment, IT security officers are often faced with the question of how accurate deployed IDSes are at detecting attacks. To this end, metrics for assessing the attack detection accuracy of IDSes have been developed. However, these metrics are defined with respect to a fixed set of hardware resources available to the tested IDS. Therefore, IDSes deployed in virtualized environments featuring elasticity (i.e., on-demand allocation or deallocation of virtualized hardware resources during system operation) cannot be evaluated in an accurate manner using existing metrics. In this paper, we demonstrate the impact of elasticity on IDS attack detection accuracy. In addition, we propose a novel metric and measurement methodology for accurately quantifying the accuracy of IDSes deployed in virtualized environments featuring elasticity. We demonstrate their practical use through case studies involving commonly used IDSes.

The wide spreading of mobile devices, such as smartphones and tablets, and their advancing capabilities, ranging from taking photos to accessing banking accounts, make them an attractive target for attackers. This, together with the fact that users frequently store critical information in such devices and that many organizations allow employees to use their personal devices to access the enterprise information infrastructure and applications, makes security assessment a key need. This paper proposes an approach for assessing the security risk posed by user-defined configurations in Android devices. The approach is based on the analysis of the risk (impact and likelihood) of user misconfiguration to harm the device or the user. The impact and likelihood values are defined based on a Multiple-Criteria Decision Analysis (MCDA) performed on the inputs provided by a set of security experts. A case study considering the user-defined configurations of 561 Android devices is presented, showing that the majority of the users neglect important and basic security configurations and that the proposed approach can be used in practice to characterize the security risk level of such devices.

This emerging technology track focuses on key topics regarding dependability and security of software and services. Service-based systems are being used in business, safety, and mission-critical environments to achieve operational goals and possess special characteristics that bring in difficult challenges to the research and industry communities. Among these challenges, dependability and security have been widely identified as critical aspects that need to be addressed, especially when considering that many services are also nowadays being deployed on the web, used over unreliable networks, and potentially exposed to security threats. The goal of the Emerging Technology Track On Dependable and Secure Services is to bring together researchers and practitioners to present original research and industrial practice regarding techniques to improve the dependability and security of services. Services hold special characteristics, in particular their typically complex nature, high heterogeneity, and fast-changing dynamics. In such scenarios, infrastructure interdependencies, failure and recovery modeling and analysis, accidental threats and attack modeling and evaluation, testing approaches, testbeds, benchmarks, interoperability in presence of dependability and security guarantees, as well as techniques and tools to assess the impact of accidental and malicious threats, metrics for assessing dependability and security are among the crucial aspects to be addressed.

Research and practice show that the effectiveness of vulnerability detection tools depends on the concrete use scenario. Benchmarking can be used for selecting the most appropriate tool, helping assessing and comparing alternative solutions, but its effectiveness largely depends on the adequacy of the metrics. This paper studies the problem of selecting the metrics to be used in a benchmark for software vulnerability detection tools. First, a large set of metrics is gathered and analyzed according to the characteristics of a good metric for the vulnerability detection domain. Afterwards, the metrics are analyzed in the context of specific vulnerability detection scenarios to understand their effectiveness and to select the most adequate one for each scenario. Finally, an MCDA algorithm together with experts' judgment is applied to validate the conclusions. Results show that although some of the metrics traditionally used like precision and recall are adequate in some scenarios, others require alternative metrics that are seldom used in the benchmarking area.

Web Services are designed with the key goal of providing interoperable application-to-application interaction, regardless of the platforms involved. Although experience shows that interoperability is difficult to achieve, developers still have limited tools to assess the interoperability of their services and, to the best of our knowledge, none able to support end-to-end interoperability certification. In this paper, we lay the foundations of an interoperability certification process for Web services, which allows testing the interoperability level of a given Web service and also identifying possible interoperability issues. In practice, the process can be used by developers or providers to certify a given web service for interoperability, ensuring successful interaction with client-side platforms. We show the effectiveness of the process by conducting a large experimental evaluation to certify five different implementations of the services specified by the TPC-App benchmark, and about 2500 synthetic generated services.client-side platforms.

There is nowadays an increasing pressure to develop complex Web applications at a fast pace. The vast majority is built using frameworks based on third-party server-side plugins that allow developers to easily add new features. However, as many plugin developers have limited programming skills, there is a spread of security vulnerabilities related to their use. Best practices advise the use of systematic code review for assure security, but free tools do not support OOP, which is how most Web applications are currently developed. To address this problem we propose phpSAFE, a static code analyzer that identifies vulnerabilities in PHP plugins developed using OOP. We evaluate phpSAFE against two well-known tools using 35 plugins for a widely used CMS. Results show that phpSAFE clearly outperforms other tools, and that plugins are being shipped with a considerable number of vulnerabilities, which tends to increase over time.

This visionary track theme focuses on dependability and security of software and services. Service-based systems are being used in business and safety-critical environments to achieve operational goals and possess special characteristics that have bring difficult challenges to the research and industry communities. Among these challenges, dependability and security have been widely identified as critical aspects that need to be addressed, especially when considering that services are being deployed on the web, and used over unreliable networks to perform critical functions.

Selecting a vulnerability detection tool is a key problem that is frequently faced by developers of security-critical web services. Research and practice shows that state-of-the-art tools present low effectiveness both in terms of vulnerability coverage and false positive rates. The main problem is that such tools are typically limited in the detection approaches implemented, and are designed for being applied in very concrete scenarios. Thus, using the wrong tool may lead to the deployment of services with undetected vulnerabilities. This paper proposes a benchmarking approach to assess and compare the effectiveness of vulnerability detection tools in web services environments. This approach was used to define two concrete benchmarks for SQL Injection vulnerability detection tools. The first is based on a predefined set of web services, and the second allows the benchmark user to specify the workload that best portrays the specific characteristics of his environment. The two benchmarks are used to assess and compare several widely used tools, including four penetration testers, three static code analyzers, and one anomaly detector. Results show that the benchmarks accurately portray the effectiveness of vulnerability detection tools (in a relative manner) and suggest that the proposed benchmarking approach can be applied in the field.

Web Services are a set of technologies designed to support the invocation of remote services by client applications, with the key goal of providing interoperable application-to-application interaction while supporting vendor and platform independence. The goal of this work is to study the real level of interoperability provided by these technologies through a massive experimental campaign involving a wide set of very popular frameworks for web services, implemented using seven different programming languages. We have tested the inter-operation of eleven client-side framework subsystems with three of the most widely used server-side implementations, each one hosting thousands of different services. The results highlight numerous situations where the goal of interoperability between different frameworks is not met due to problems both on the client and the server side. Moreover, we have identified issues also affecting interactions between the client and server subsystems of the same framework.

Web services are supported by a set of protocols that have been designed with the main goal of providing interoperable communication to applications. In typical business-critical services environments the occurrence of interoperability issues can have disastrous consequences, including direct financial costs, reputation, and client fidelity losses. Despite this, experience suggests that interoperability is still quite difficult to achieve, since the heterogeneity of frameworks for providing web services is quite large. In addition, current tools have limited testing capabilities and, in many cases do not specialize in this problem. In this paper we present ITWS, an extensible Interoperability Testing tool for Web Services that is able to assess the interoperability of a web service, supported by any given framework. We have used ITWS to test the interoperability of a set of home-implemented TPC-App web services and a set of thousands of web services created in .NET C# against 11 client-side web service frameworks, including frameworks for mainstream programming languages. Numerous issues have been disclosed, showing the benefits of using ITWS and the importance of testing services for interoperability.

Hypervisors are becoming increasingly ubiquitous with the growing proliferation of virtualized data centers. As a result, attackers are exploring vectors to attack hypervisors, against which an attack may be executed via several attack vectors such as device drivers, virtual machine exit events, or hyper calls. Hyper calls enable intrusions in hypervisors through their hyper call interfaces. Despite the importance, there is very limited publicly available information on vulnerabilities of hyper call handlers and attacks triggering them, which significantly hinders advances towards monitoring and securing these interfaces. In this paper, we characterize the hyper call attack surface based on analyzing a set of vulnerabilities of hyper call handlers. We systematize and discuss the errors that caused the considered vulnerabilities, and activities for executing attacks triggering them. We also demonstrate attacks triggering the considered vulnerabilities and analyze their effects. Finally, we suggest an action plan for improving the security of hyper call interfaces.

Techniques to classify defects have been used for decades, providing relevant information on how to improve systems. Such techniques heavily rely on human experience and have been generalized to cover different types of systems at different maturity levels. However, their application to safety-critical systems development and operation phases neither is very common, or at least not spread publicly, nor disseminated in the industrial and academic worlds. This practical experience report presents the results and conclusions from applying a mature Orthogonal Defect Classification (ODC) to a large set of safety-critical issues. The work is based on the analysis of more than 240 real issues (defects) identified during all the lifecycle phases of 4 safety-critical systems in the aerospace and space domains. The outcomes reveal the challenges in properly classifying this specific type of issues with the broader ODC approach. The difficulties are identified and systematized and specific proposals for improvement are proposed.

Failure Prediction allows improving the dependability of computer systems, but its use is still uncommon due to scarcity of failure-related data that can be used for training, assessing and comparing alternative failure predictors. As failures are rare events and the characteristics of failure data varies from system to system, in this paper we propose the use of realistic software fault injection to facilitate the generation of failure data on a particular system installation. In practice, we propose a comprehensive experimental approach that allows generating failure data in short time and we study the applicability and limitations of such process in assessing and comparing alternative failure prediction algorithms. A case study is presented comparing four algorithms for predicting failures in a system based on a Windows OS. Results show that using fault injection allows to dramatically speed up the generation of failure data and that the proposed procedure can be used in practice.

This workshop focuses on dependability and security of software and services. Service-based systems are being used in business and safety-critical environments to achieve operational goals and possess special characteristics that have been bringing difficult challenges to the research and industry communities for several years now. Among such challenges, dependability and security have been widely identified as critical aspects that need to be addressed, especially when considering that many times services are being deployed on the web, and used over unreliable networks to perform critical functions.

In an attempt to support customization, many web applications allow the integration of third-party server-side plugins that offer diverse functionality, but also open an additional door for security vulnerabilities. In this paper we study the use of static code analysis tools to detect vulnerabilities in the plugins of the web application. The goal is twofold: 1) to study the effectiveness of static analysis on the detection of web application plugin vulnerabilities, and 2) to understand the potential impact of those plugins in the security of the core web application. We use two static code analyzers to evaluate a large number of plugins for a widely used Content Manage-ment System. Results show that many plugins that are current-ly deployed worldwide have dangerous Cross Site Scripting and SQL Injection vulnerabilities that can be easily exploited, and that even widely used static analysis tools may present disappointing vulnerability coverage and false positive rates.

Most web applications have critical bugs (faults) affecting their security, which makes them vulnerable to attacks by hackers and organized crime. To prevent these security problems from occurring it is of utmost importance to understand the typical software faults. This paper contributes to this body of knowledge by presenting a field study on two of the most widely spread and critical web application vulnerabilities: SQL Injection and XSS. It analyzes the source code of security patches of widely used Web applications written in weak and strong typed languages. Results show that only a small subset of software fault types, affecting a restricted collection of statements, is related to security. To understand how these vulnerabilities are really exploited by hackers, this paper also presents an analysis of the source code of the scripts used to attack them. The outcomes of this study can be used to train software developers and code inspectors in the detection of such faults and are also the foundation for the research of realistic vulnerability and attack injectors that can be used to assess security mechanisms, such as intrusion detection systems, vulnerability scanners, and static code analyzers.

In this paper we propose a methodology and a prototype tool to evaluate web application security mechanisms. The methodology is based on the idea that injecting realistic vulnerabilities in a web application and attacking them automatically can be used to support the assessment of existing security mechanisms and tools in custom setup scenarios. To provide true to life results, the proposed vulnerability and attack injection methodology relies on the study of a large number of vulnerabilities in real web applications. In addition to the generic methodology, the paper describes the implementation of the Vulnerability & Attack Injector Tool (VAIT) that allows the automation of the entire process. We used this tool to run a set of experiments that demonstrate the feasibility and the effectiveness of the proposed methodology. The experiments include the evaluation of coverage and false positives of an intrusion detection system for SQL Injection attacks and the assessment of the effectiveness of two top commercial web application vulnerability scanners. Results show that the injection of vulnerabilities and attacks is indeed an effective way to evaluate security mechanisms and to point out not only their weaknesses but also ways for their improvement.

Developing robust web services is a difficult task. Field studies show that a large number of web services are deployed with robustness problems (i.e., presenting unexpected behaviors in the presence of invalid inputs). Although several techniques for the identification of robustness problems have been proposed in the past, there is no practical approach to automatically fix those problems. This paper proposes a mechanism that automatically fixes robustness problems in web services. The approach consists of using robustness testing to detect robustness issues and then mitigate those issues by applying inputs verification based on well-defined parameter domains, including domain dependencies between different parameters. This integrated and fully automated methodology has been used to improve three different implementations of the TPC-App web services and several services publicly available on the Internet. Results show that the proposed approach can be easily used to improve the robustness of web services code.

Fault injection is a well-known technology that enables assessing dependability attributes of computer systems. Many works on fault injection have been developed in the past, and fault injection has been used in different application domains. This fast abstract briefly revises previous applications of fault injection, especially for embedded systems, and puts forward ideas on its future use, both in terms of application areas and business markets.

Satellite simulators are used to support the tasks of space mission analysis and satellite verification and operation, having high dependability requirements. When used for different missions or different phases of the same mission, robustness and scalability are two particularly important properties. This practical experience report presents the results of a robustness and scalability evaluation of two open-source Runtime Libraries implementing HLA, a standard widely used in the context of simulators development. Results show that the tested implementations present a large number of robustness problems and that scalability is quite limited.

Real-time Data Warehouses (DWs) must be able to deal with continuous updates while ensuring 24/7 availability. To improve their performance, distributing data using round-robin algorithms on clusters of shared-nothing machines is normally used. This paper proposes a solution for distributed DW databases that ensures its continuous availability and deals with frequent data loading requirements, while adding small performance overhead. We use a data striping and replication architecture to distribute portions of each fact table among pairs of slave nodes, where each slave node is an exact replica of its partner. This allows balancing query execution and replacing any defective node, ensuring the system's continuous availability. The size of each portion in a given node depends on its individual features, namely performance benchmark measures and dedicated database RAM. The estimated cost for executing each query workload in each slave node is also used for balancing query performance. We include experiments using the TPC-H decision support benchmark to evaluate the scalability of the proposed solution and show that it outperforms standard round-robin distributed DW setups.

Developers often rely on penetration testing tools to detect vulnerabilities in web services, although frequently without really knowing their effectiveness. In fact, the lack of information on the internal state of the tested services and the complexity and variability of the responses analyzed, limits the effectiveness of such technique, highlighting the importance of evaluating and improving existing tools. The goal of this paper is to investigate if attack signatures and interface monitoring can be an effective mean to assess and improve the performance of penetration testing tools in web services environments. In practice, attacks performed by such tools are signed and the interfaces between the target application and external resources are monitored (e.g., between services and a database server), allowing gathering additional information on existing vulnerabilities. A prototype was implemented focusing on SQL injection vulnerabilities. The experimental evaluation results clearly show that the proposed approach can be used in real scenarios.

Security is one of the biggest concerns of database administrators. Most marketed software products announce a variety of features and mechanisms designed to improve security. However, that same variety largely complicates the process of selecting the adequate set of software products (i.e., a software package) for a given installation. In this paper we propose an approach that can be used to fairly compare alternative software packages regarding security capabilities in database environments. We focus specifically on the two main software systems required for a new installation: the Operating System and the Database Management System (DBMS). We carefully explain and discuss our method, which is based on the idea of evaluating the characteristics of software packages against a comprehensive list of security concerns that are universally accepted as vital to any database installation. We created an actual benchmark, and used it to assess seven software packages composed by four different DBMS engines and two different operating systems. Results show that alternative software packages allow fulfilling different security concerns and that the proposed benchmark is quite effective in identifying the main differences regarding the capabilities of the systems evaluated.

Benchmarking the security of web applications is complex and, although there are many proposals of metrics, no consensual quantitative security metric has been proposed so far. Static analysis is an effective approach for detecting vulnerabilities, but the complexity of applications and the large variety of vulnerabilities prevent any single tool from being foolproof. In this application paper we investigate the hypothesis of combining the output of multiple static code analyzers to define metrics for comparing the trustworthiness of web applications. Various experiments, including a benchmarking campaign over seven distinct open source web forums, show that the raw number of vulnerabilities reported by a set of tools allows rough trustworthiness comparison. We also study the use of normalization and false positive rate estimation to calibrate the output of each tool. Results show that calibration allows computing a very accurate metric that can be used to easily and automatically compare different applications.

Technological evolution has redefined many business models. Many decision makers are now required to act near real-time, instead of periodically, given the latest transactional information. Decision-making occurs much more frequently and considers the latest business data. Since data warehouses (DWs) are the core of business intelligence, decision support systems need to deal with 24/7 real-time requirements. Thus, the ability to deal with continuous data loading and decision support availability simultaneously is critical, for producing continuous actionable knowledge. The main challenge in this context is to efficiently manage the DW's refreshment, when data sources change, to recapture consistency and accuracy with those sources, while maintaining OLAP availability and database performance. This paper proposes a simple, fast and efficient solution based on database replication and temporary tables to change a traditional enterprise DW into a real-time DW, enabling continuous data loading and OLAP availability on a 24/7 schedule. Experimental evaluations using a real-world DW and the TPC-H decision support benchmark show its advantages and analyze its impact in OLAP performance.

Data Warehouses (DWs) are the enterprise's most valuable asset in what concerns critical business information, making them an appealing target for attackers. Packaged database encryption solutions are considered the best solution to protect sensitive data. However, given the volume of data typically processed by DW queries, the existing encryption solutions heavily increase storage space and introduce very large overheads in query response time, due to decryption costs. In many cases, this performance degradation makes encryption unfeasible for use in DWs. In this paper we propose a transparent data masking solution for numerical values in DWs based on the mathematical modulus operator, which can be used without changing user application and DBMS source code. Our solution provides strong data security while introducing small overheads in both storage space and database performance. Several experimental evaluations using the TPC-H decision support benchmark and a real-world DW are included. The results show the overall efficiency of our proposal, demonstrating that it is a valid alternative to existing standard encryption routines for enforcing data confidentiality in DWs.

Effort estimation in software development projects is far from being an easy task. In fact, despite the several effort estimation techniques available in the literature and the need for companies to perform such task in a daily basis, most small and medium-sized companies still suffer from the problem of incorrect estimations that often result in losing the contract bid or in failure during project execution. In this paper we present and discuss the implementation of a software effort estimation process in a medium-sized company, currently recognized as CMMI Level 5. The paper contextualizes the problem and the company, introduces the estimation techniques used, and presents some preliminary results, showing that software effort estimation can be successfully applied in medium-sized companies at low cost, allowing the reduction of project uncertainty and increasing the probability of success during bidding and execution.

Software reuse is the practice of using existing artifacts (code, architecture, requirements, etc.) in new projects. The advantages of using previously developed software in new projects are easily understood. However, reusing artifacts is usually done in an ad-hoc and incipient way, requiring an important effort of adaptation, so developers frequently prefer to develop components from scratch. In this paper we present a strategy that is being adopted by Critical Software, a medium-sized company, to promote software reuse. This strategy starts by assuming that the success of software reuse is dependent on the ability of measuring its advantages. We have thus proposed the use of the Goal-Question-Metric (GQM) technique, extended with Data Warehousing data model design concepts to extract a set of reuse-specific metrics for measuring the gains of reuse. We show that it is very easy to measure the productivity improvement due to code reuse, by simply measuring or estimating the efforts of developing a component for reuse, integrating it a new artifact, and developing this artifact, built with reusing the component.

Data Warehouses (DWs) store the golden nuggets of the business, which makes them an appealing target. To ensure data privacy, encryption solutions have been used and proven efficient in their security purpose. However, they introduce massive storage space and performance overheads, making them unfeasible for DWs. We propose a data masking technique for protecting sensitive business data in DWs that balances security strength with database performance, using a formula based on the mathematical modular operator. Our solution manages apparent randomness and distribution of the masked values, while introducing small storage space and query execution time overheads. It also enables a false data injection method for misleading attackers and increasing the overall security strength. It can be easily implemented in any DataBase Management System (DBMS) and transparently used, without changes to application source code. Experimental evaluations using a real-world DW and TPC-H decision support benchmark implemented in leading commercial DBMS Oracle llg and Microsoft SQL Server 2008 demonstrate its overall effectiveness. Results show substantial savings of its implementation costs when compared with state of the art data privacy solutions provided by those DBMS and that it outperforms those solutions in both data querying and insertion of new data.

Vulnerability detection tools are frequently considered the silver-bullet for detecting vulnerabilities in web services. However, research shows that the effectiveness of most of those tools is very low and that using the wrong tool may lead to the deployment of services with undetected vulnerabilities. In this paper we propose a benchmarking approach to assess and compare the effectiveness of vulnerability detection tools in web services environments. This approach was used to define a concrete benchmark for SQL Injection vulnerability detection tools. This benchmark is demonstrated by a real example of benchmarking several widely used tools, including four penetration-testers, three static code analyzers, and one anomaly detector. Results show that the benchmark accurately portrays the effectiveness of vulnerability detection tools and suggest that the proposed approach can be applied in the field.

System administrators frequently rely on intrusion detection tools to protect their systems against SQL Injection, one of the most dangerous security threats in database-centric web applications. However, the real effectiveness of those tools is usually unknown, which may lead administrators to put an unjustifiable level of trust in the tools they use. In this paper we present an experimental evaluation of the effectiveness of five SQL Injection detection tools that operate at different system levels: Application, Database and Network. To test the tools in a realistic scenario, Vulnerability and Attack Injection is applied in a setup based on three web applications of different sizes and complexities. Results show that the assessed tools have a very low effectiveness and only perform well under specific circumstances, which highlight the limitations of current intrusion detection tools in detecting SQL Injection attacks. Based on experimental observations we underline the strengths and weaknesses of the tools assessed.

Web applications are a fundamental pillar of today's globalized world. Society depends and relies on them for business and daily life. However, web applications are under constant attack by hackers that exploit their vulnerabilities to access valuable assets and disrupt business. Many studies and reports on web application security problems analyze the victim's perspective by detailing the vulnerabilities publicly disclosed. In this paper we present a field study on the attacker's perspective by looking at over 300 real exploits used by hackers to attack web applications. Results show that SQL injection and Remote File Inclusion are the two most frequently used exploits and that hackers prefer easier rather than complicated attack techniques. Exploit and vulnerability data are also correlated to show that, although there are many types of vulnerabilities out there, only few are interesting enough for attackers to obtain what they want the most: root shell access and admin passwords.

This paper presents a field study on real errors found in space software requirements documents. The goal is to understand and characterize the most frequent types of requirement problems in this critical application domain. To classify the software requirement errors analyzed we initially used a well-known existing taxonomy that was later extended in order to allow a more thorough analysis. The results of the study show a high rate of requirement errors (9.5 errors per each 100 requirements), which is surprising if we consider that the focus of the work is critical embedded software. Besides the characterization of the most frequent types of errors, the paper also proposes a set of operators that define how to inject realistic errors in requirement documents. This may be used in several scenarios, including: evaluating and training reviewers, estimating the number of requirement errors in real specifications, defining checklists for quick requirement verification, and defining benchmarks for requirements specifications.

Predicting failures at runtime is one of the most promising techniques to increase the availability of computer systems. However, failure prediction algorithms are still far from providing satisfactory results. In particular, the identification of the variables that show symptoms of incoming failures is a difficult problem. In this paper we propose an approach for identifying the most adequate variables for failure prediction. Realistic software faults are injected to accelerate the occurrence of system failures and thus generate a large amount of failure related data that is used to select, among hundreds of system variables, a small set that exhibits a clear correlation with failures. The proposed approach was experimentally evaluated using two configurations based on Windows XP. Results show that the proposed approach is quite effective and easy to use and that the injection of software faults is a powerful tool for improving the state of the art on failure prediction.

Business critical applications are increasingly being deployed as web services that access database systems, and must provide secure operations to its clients. Although the open web environment emphasizes the need for security, several studies show that web services are still being deployed with command injection vulnerabilities. This paper proposes a learning-based approach to secure web services against SQL and XPath Injection attacks. Our approach is able to transparently learn valid request patterns (learning phase) and then detect and abort potentially harmful requests (protection phase). When it is not possible to have a complete learning phase, a set of heuristics can be used to accept/discard doubtful cases. Our mechanism was applied to secure TPC-App services and open source services. It showed to be extremely effective in stopping all tested attacks, while introducing a negligible performance impact.

Self-adaptive systems are widely recognized as the future of computer systems. Due to their dynamic and evolving nature, the characterization of self-adaptation and resilience attributes is of upmost importance. The problem is that nowadays there is no practical way to characterize self-adaptation capabilities or to compare alternative solutions concerning resilience. In this paper we discuss the problem of resilience benchmarking of self-adaptive systems. We start by identifying a set of key challenges and then propose a research roadmap to tackle those challenges.

Testing web services for robustness is an effective way of disclosing software bugs. However, when executing robustness tests, a very large amount of service responses has to be manually classified to distinguish regular responses from responses that indicate robustness problems. Besides requiring a large amount of time and effort, this complex classification process can easily lead to errors resulting from the human intervention in such a laborious task. Text classification algorithms have been applied successfully in many contexts (e.g., spam identification, text categorization, etc) and are considered a powerful tool for the successful automation of several classification-based tasks. In this paper we present a study on the applicability of five widely used text classification algorithms in the context of web services robustness testing. In practice, we assess the effectiveness of Support Vector Machines, Naïve Bayes, Large Linear Classification, K-nearest neighbor (Ibk), and Hyperpipes in classifying web services responses. Results indicate that these algorithms can be effectively used to automate the identification of robustness issues while reducing human intervention. However, in all mechanisms there are cases of misclassified responses, which means that there is space for improvement.

Quantifying security is very hard and, although there are many proposals of security metrics in the literature, no consensual quantitative security metric has been proposed so far. A key difficulty is that security is, usually, more influenced by what is unknown about a system than by what is known about it. In this paper we present the idea of trust-based metrics, which are based on the idea of quantifying and exposing the trustworthiness relationship between a system and its owner. We defend that they represent a powerful alternative to traditional security metrics and are much easier to obtain. As an instantiation, we propose minimum untrustworthiness as a low-cost high-reward trust-based metric that can be easily used to assess and compare security aspects. We discuss what does it express, show how it can be computed and what are its advantages. Finally, we present preliminary work on the definition of an untrustworthiness benchmark for database configurations.

In this paper we propose a methodology to inject realistic attacks in Web applications. The methodology is based on the idea that by injecting realistic vulnerabilities in a Web application and attacking them automatically we can assess existing security mechanisms. To provide true to life results, this methodology relies on field studies of a large number of vulnerabilities in Web applications. The paper also describes a set of tools implementing the proposed methodology. They allow the automation of the entire process, including gathering results and analysis. We used these tools to conduct a set of experiments to demonstrate the feasibility and effectiveness of the proposed methodology. The experiments include the evaluation of coverage and false positives of an intrusion detection system for SQL injection and the assessment of the effectiveness of two Web application vulnerability scanners. Results show that the injection of vulnerabilities and attacks is an effective way to evaluate security mechanisms and tools.

The Student Forum at DSN provides an opportunity for students currently working in the area of dependable and secure computing to present and discuss their research objectives, approach and preliminary results.

Although Web services are becoming business-critical components, they are often deployed with critical software bugs that can be maliciously explored. Web vulnerability scanners allow detecting security vulnerabilities in Web services by stressing the service from the point of view of an attacker. However, research and practice show that different scanners have different performance on vulnerabilities detection. In this paper we present an experimental evaluation of security vulnerabilities in 300 publicly available Web services. Four well known vulnerability scanners have been used to identify security flaws in Web services implementations. A large number of vulnerabilities has been observed, which confirms that many services are deployed without proper security testing. Additionally, the differences in the vulnerabilities detected and the high number of false-positives (35% and 40% in two cases) and low coverage (less than 20% for two of the scanners) observed highlight the limitations of Web vulnerability scanners on detecting security vulnerabilities in Web services.

Many organizations have to manage an increasingly large number of software projects. In many cases, these projects are outsourced to different companies or developed across several departments. This creates a problem because it is increasingly difficult for a project manager, responsible for several projects, to have an accurate view of all activities underway, being able to act proactively before problems occur. In this paper we present an approach and corresponding implementation for effectively tracking and managing multiple projects across an organization, offering an integrated view of the state of projects being implemented. This approach allows to effectively monitoring risks, progress, budget, deliveries, and other critical aspects, allowing responding in real-time. The system was implemented for the European Space Agency, being now successfully used in a number of projects. This paper also presents an accurate view on how software development and tracking takes place in the scope of mission-critical systems.

Developing robust web services is a difficult task. Field studies show that a large number of web services are deployed with robustness problems (i.e., presenting unexpected behaviors in the presence of invalid inputs). Several techniques for the identification of robustness problems have been proposed in the past. This paper proposes a mechanism that automatically fixes the problems detected. The approach consists of using robustness testing to detect robustness issues and then mitigate those issues by applying inputs verification based on well-defined parameter domains, including domain dependencies between different parameters. This integrated and fully automatable methodology has been used to improve three different implementations of the TPC-App web services. Results show that this tool can be easily used by developers to improve the robustness of web services implementations.

This paper presents a field study on Web security vulnerabilities from the programming language type system perspective. Security patches reported for a set of 11 widely used Web applications written in strongly typed languages (Java, C#, VB.NET) were analyzed in order to understand the fault types that are responsible for the vulnerabilities observed (SQL injection and XSS). The results are analyzed and compared with a similar work on Web applications written using a weakly typed language (PHP). This comparison points out that some of the types of defects that lead to vulnerabilities are programming language independent, while others are strongly related to the language used. Strongly typed languages do reduce the frequency of vulnerabilities, as expected, but there still is a considerable number of vulnerabilities observed in the field. The characterization of those vulnerabilities shows that they are caused by a small number of fault types. This result is relevant to train programmers and code inspectors in the manual detection of such faults, and to improve static code analyzers to automatically detect the most frequent vulnerable program structures found in the field.

Web services are becoming business-critical components that must provide a non-vulnerable interface to the client applications. However, previous research and practice show that many web services are deployed with critical vulnerabilities. SQL injection vulnerabilities are particularly relevant, as Web services frequently access a relational database using SQL commands. Penetration testing and static code analysis are two well-know techniques often used for the detection of security vulnerabilities. In this work we compare how effective these two techniques are on the detection of SQL injection vulnerabilities in Web services code. To understand the strengths and limitations of these techniques, we used several commercial and open source tools to detect vulnerabilities in a set of vulnerable services. Results suggest that, in general, static code analyzers are able to detect more SQL injection vulnerabilities than penetration testing tools. Another key observation is that tools implementing the same detection approach frequently detect different vulnerabilities. Finally, many tools provide a low coverage and a high false positives rate, making them a bad option for programmers.

Database management systems (DBMS), the central component of many computers applications, are typically immersed in very complex environments. Protecting the DBMS from security attacks requires evaluating a long list of complex configuration characteristics that may impact, in a variety of ways, the applications and people that interact with the database system. Effectively, understanding the impact of different configuration alternatives in terms of security is one of the most difficult problems faced by database administrators nowadays (DBA). In this paper we propose a benchmark that allows DBAs to assess and compare database configurations. The benchmark provides a trust-based security metric, named minimum untrustworthiness, that expresses the minimum level of distrust the DBA should have in a given configuration regarding its ability to prevent attacks. The practical application of the benchmark in four real large database installations shows that it is quite easy to use and is, in fact, a powerful tool for DBAs to make informed security decisions, by taking into account the specifics needs of the environment being managed.

Current business critical environments increasingly rely on SOA standards to execute business operations. These operations are frequently based on Web service compositions that use several Web services over the internet and have to fulfill specific timing constraints. In these environments, an operation that does not conclude in due time may have a high cost as it can easily turn into service abandonment with financial and prestige losses to the service provider. In fact, at certain points, carrying on with the execution of an operation may be useless as a timely response will be impossible to obtain. This paper proposes a time-aware programming model for Web services that provides transparent timing failure detection. The paper illustrates the proposed model using a set of services specified by the TPC-App performance benchmark.

Web applications are typically developed with hard time constraints and are often deployed with critical software bugs, making them vulnerable to attacks. The classification and knowledge of the typical software bugs that lead to security vulnerabilities is of utmost importance. This paper presents a field study analyzing 655 security patches of six widely used web applications. Results are compared against other field studies on general software faults (i.e., faults not specifically related to security), showing that only a small subset of software fault types is related to security. Furthermore, the detailed analysis of the code of the patches has shown that web application vulnerabilities result from software bugs affecting a restricted collection of statements. A detailed analysis of the conditions/locations where each fault was observed in our field study is presented allowing future definition of realistic fault models that cause security vulnerabilities in web applications, which is the key element to design a realistic attack injector.

Database management systems (DBMS) have a long tradition in high security. Several mechanisms needed to protect data have been proposed/consolidated in the database arena. However, the effectiveness of those mechanisms is very dependent on the actual configuration chosen by the database administrator. Tuning a large database is quite complex and achieving high security is a very difficult task that requires a lot of expertise and continuous and proactive work. In this paper we analyze the security best practices behind the many configuration options available in several well-known DBMS. These security best practices are then generalized in order to be applicable to practically any DBMS available today. Finally, we use these best practices to define a set of configuration tests, which have been successfully used to evaluate four real database installations based in four well-known and widely used DBMS.

Writing secure web applications is a complex task. In fact, a vast majority of web applications are likely to have security vulnerabilities that can be exploited using simple tools like a common web browser. This represents a great danger as the attacks may have disastrous consequences to organizations, harming their assets and reputation. To mitigate these vulnerabilities, security code inspections and penetration tests must be conducted by well-trained teams during the development of the application. However, effective code inspections and testing takes time and cost a lot of money, even before any business revenue. Furthermore, software quality assurance teams typically lack the knowledge required to effectively detect security problems. In this paper we propose an approach to quickly and effectively train security assurance teams in the context of web application development. The approach combines a novel Vulnerability Injection Technique with relevant guidance information about the most common security vulnerabilities to provide a realistic training scenario. Our experimental results show that a short training period is sufficient to clearly improve the ability of security assurance teams to detect vulnerabilities during both code inspections and penetration tests.

This paper presents an approach to assess security of web servers. This method can be used to compare the security features of different web servers installations and to determine how secure a given web server configuration is. The assessment is done by applying a set of tests designed to check if the system under evaluation fulfils a set of security practices defined by an extensive field study. This work targets the most typical issues related to web servers ranging from classic web servers misconfiguration to the absence of a secure network infrastructure and of well-defined security policies to respond to security incidents. The effectiveness and usefulness of the proposed approach is illustrated through the security assessment and comparison of five different real web servers.

Web-services are supported by a complex software infrastructure that must provide a robust service to the client applications. This practical experience report presents a practical approach for the evaluation of the robustness of Web-services infrastructures. A set of robustness tests (i.e., invalid web-services call parameters) is applied during Web-services execution in order to reveal possible robustness problems in the Web-services code and in the application server infrastructure. The approach is illustrated using two different implementations of the Web-services specified by the TPC-App performance benchmark running on top of the JBoss application server. The proposed approach is generic and can be used to evaluate the robustness of Web-services implementations (relevant for programmers) and application server infrastructures (relevant for administrators and system integrators).

Web-services are supported by a complex software infrastructure that must ensure high performance and availability to the client applications. Web services industry holds a well established platform for performance benchmarking (e.g., TPC-App and SPEC jAppServer2004 benchmarks). In addition, several studies have been published recently by main vendors focusing web services performance. However, as peak performance evaluation has been the main focus, the characterization of the impact of faults in such systems has been largely disregarded. This paper proposes an approach for the evaluation and comparison of performance and recovery time in web services infrastructures. This approach is based on fault injection and is illustrated through a concrete example of benchmarking three alternative software solutions for web services deployment.

Web applications are typically developed with hard time constraints and are often deployed with security vulnerabilities. Automatic web vulnerability scanners can help to locate these vulnerabilities and are popular tools among developers of web applications. Their purpose is to stress the application from the attacker's point of view by issuing a huge amount of interaction within it. Two of the most widely spread and dangerous vulnerabilities in web applications are SQL injection and cross site scripting (XSS), because of the damage they may cause to the victim business. Trusting the results of web vulnerability scanning tools is of utmost importance. Without a clear idea on the coverage and false positive rate of these tools, it is difficult to judge the relevance of the results they provide. Furthermore, it is difficult, if not impossible, to compare key figures of merit of web vulnerability scanners. In this paper we propose a method to evaluate and benchmark automatic web vulnerability scanners using software fault injection techniques. The most common types of software faults are injected in the web application code which is then checked by the scanners. The results are compared by analyzing coverage of vulnerability detection and false positives. Three leading commercial scanning tools are evaluated and the results show that in general the coverage is low and the percentage of false positives is very high.

This paper proposes an approach for the evaluation of the robustness of web services, which are complex software components that must provide a robust interface to the client applications. However, although web services are becoming business-critical components, there is no practical way to assess the robustness of the code or to compare alternative implementations concerning robustness. The approach proposed is based on a set of robustness tests (i.e., invalid web services call parameters) that is applied in order to discover both programming and design errors. The web services are classified based on the failures observed during the execution of the tests. The approach is illustrated by evaluating several web services publicly available in the Internet and two different implementations of the web services specified by the standard TPC-App performance benchmark. The proposed approach is useful for both web services providers (to assess the robustness of their web services code) and consumers (to select the web services that best fit their requirements).

Database management systems (DBMS) represent the ultimate layer in preventing malicious data access or corruption and implement several security mechanisms to protect data. However these mechanisms cannot always stop malicious users from accessing data by exploiting system vulnerabilities. The aim of this paper is to propose an intrusion detection mechanism for DBMS to fill this gap. Our approach consists of a comprehensive representation of user database utilization profiles to perform concurrent intrusion detection. Prior to the detection it is necessary to define and learn these utilization profiles. Profiles are defined using a three level abstraction and learned directly from monitoring the database utilization in real conditions. The proposed mechanism is generic and can be easily implemented in commercial and open-source DBMS

On time data management is becoming a key difficulty faced by organizations. In spite of the importance of timeliness requirements in database applications, commercial DBMS do not assure the detection of the cases when a transaction takes longer than the expected/desired time. This paper discusses the problem of timing failure detection in database applications and proposes a transaction programming approach to help developers in programming database applications with time constraints

One of the main problems faced by organizations is the protection of their data against unauthorized access or corruption due to malicious actions. Database management systems (DBMS) constitute the kernel of the information systems used today to support the daily operations of most organizations and represent the ultimate layer in preventing unauthorized access to data stored in information systems. Nevertheless, in spite of the key role played by the DBMS in the overall data security, no practical way has been proposed so far to characterize the security in such systems or to compare alternative solutions concerning security features. This paper proposes an approach to characterize the security mechanisms in database systems and database applications, according to a set of security classes. The proposed approach is generic and can be applied to both DBMS (relevant for system integrators) and real database installations (relevant for database administrators and end-users).

A major difficulty faced by organizations is the protection of data against malicious access or corruption. Database management systems (DBMS) are a key component in the information infrastructure of most organizations and represent the ultimate layer in preventing unauthorized data accesses. Several mechanisms needed to protect data, such as authentication, user privileges, encryption, and auditing, have been implemented in commercial DBMS. However, typical database security mechanisms are not able to detect and handle many data security attacks. In fact, malicious transactions executed by unauthorized users that may gain access to the database by exploring system vulnerabilities and unauthorized database transactions executed by authorized users cannot be detected and stopped by typical security mechanisms. In this paper we propose a new mechanism for the detection of malicious transactions in DBMS. The paper presents a practical example of the implementation of the proposed mechanism in the Oracle 10g DBMS and evaluates the mechanism using the TPC-C benchmark.

Databases play a central role in the information infrastructure of most organizations. The characterization of DBMS (database management systems) dependability is then of utmost importance. Existing performance benchmarks for transactional and database areas include two major components: a workload and a set of performance measures. The definition of a benchmark to characterize dependability needs a new component - the faultload. Operator faults represent a major cause of failures in large DBMS. This paper proposes three approaches for the definition of portable faultloads based on operator faults to benchmark the dependability of DBMS and shows a benchmarking example of a commercial (Oracle) and an open source (PostgreSQL) database.

Two important questions on experimental dependability evaluation remain largely unanswered: 1) how to analyze the usually large amount of raw data produced in dependability evaluation experiments and 2) how to compare results from different experiments or results from similar experiments across different systems. These problems are also common to other dependability evaluation techniques such as the ones based on simulation, or even to the analysis of field data on computer faults. We propose the use of data warehousing technologies to store raw results from different experiments/setups in a common multidimensional structure where raw data can be analyzed and shared world wide by means of web-enabled OLAP (On-Line Analytical Processing) tools. This paper describes how to use the proposed approach in a concrete example of dependability evaluation experiment.

On-Line Transaction Processing (OLTP) systems constitute the kernel of the information systems used today to support the daily operations of most organizations. Although these systems comprise the best examples of complex business-critical systems, no practical way has been proposed so far to characterize the impact of faults in such systems or to compare alternative solutions concerning dependability features. This paper presents a practical example of benchmarking key dependability features of four different transactional systems using a first proposal of dependability benchmark for OLTP application environments. This dependability benchmark is an extension to the TPC-C standard performance benchmark, and specifies the measures and all the steps required to evaluate both the performance and dependability features of OLTP systems. Two different versions of the Oracle transactional engine running over two different operating systems were evaluated and compared. The results show that dependability benchmarking can be successfully applied to OLTP application environments.

A major cause of failures in large database management systems (DBMS) is operator faults. Although most of the complex DBMS have comprehensive recovery mechanisms, the effectiveness of these mechanisms is difficult to characterize. On the other hand, the tuning of a large database is very complex and database administrators tend to concentrate on performance tuning and disregard the recovery mechanisms. Above all, database administrators seldom have feedback on how good a given configuration is concerning recovery. This paper proposes an experimental approach to characterize both the performance and the recoverability in DBMS. Our approach is presented through a concrete example of benchmarking the performance and recovery of an Oracle DBMS running the standard TPC-C benchmark, extended to include two new elements: a fault load based on operator faults and measures related to recoverability. A classification of operator faults in DBMS is proposed. The paper ends with the discussion of the results and the proposal of guidelines to help database administrators in finding the balance between performance and recovery tuning.

The characterization of database management system (DBMS) recovery mechanisms and the comparison of recovery features of different DBMS require a practical approach to benchmark the effectiveness of recovery in the presence of faults. Existing performance benchmarks for transactional and database areas include two major components: a workload and a set of performance measures. The definition of a benchmark to characterize DBMS recovery needs a new component the faultload. A major cause of failures in large DBMS is operator faults, which make them an excellent starting point for the definition of a generic faultload. This paper proposes the steps for the definition of generic faultloads based on operator faults for DBMS recovery benchmarking. A classification for operator faults in DBMS is proposed and a comparative analysis among three commercially DBMS is presented. The paper ends with a practical example of the use of operator faults to benchmark different configurations of the recovery mechanisms of the Oracle 8i DBMS.

This paper presents an experimental study on the emulation of software faults by fault injection. In a first experiment, a set of real software faults has been compared with faults injected by a SWIFI tool (Xception) to evaluate the accuracy of the injected faults. Results revealed the limitations of Xception (and other SWIFI tools) in the emulation of different classes of software faults (about 44% of the software faults cannot be emulated). The use of field data about real faults was discussed and software metrics were suggested as an alternative to guide the injection process when field data is nor available. In a second experiment, a set of rules for the injection of errors meant to emulate classes of software faults was evaluated. The fault triggers used seem to be the cause for the observed strong impact of the faults in the target system and in the program results. The results also show the influence in the fault emulation of aspects such as code size, complexity of data structures, and recursive versus sequential execution.