The performance assessment of blockchain applications holds significant challenges due to their decentralized architecture, immutable smart contracts, distributed ledgers, and operational costs such as gas fees. Existing blockchain benchmarks often either fail to fully capture blockchain-specific behaviors or offer limited configurability and metric reporting. In this paper, we present a new and comprehensive benchmark designed explicitly for blockchain applications, named bBench. Building on established principles from traditional benchmarking and by specializing them in the blockchain context and supported by customized blockchain tools (i.e., Hyperledger Caliper, web3.eth, and node-os-utils), bBench characterizes blockchain application performance in four dimensions: network performance, resource utilization, storage usage, and operational cost. We demonstrate the effectiveness of our benchmark through a case study involving 12 smart contract applications with varying performance demands, some of which hold known vulnerabilities. The results show the benchmark’s ability to quantify performance deviations across different applications, as well as those caused by the activation of specific vulnerabilities.

Blockchain has recently become very popular due to its use in cryptocurrencies and potential application in various domains (e.g., retail, healthcare, and insurance). The smart contract is a key part of blockchain systems and specifies an agreement between transaction participants. Nowadays, smart contracts are being deployed to carry residual faults, including severe vulnerabilities that lead to different types of failures at runtime. Fault detection tools can be used to detect faults that may then be removed from the code before deployment. However, in the case of smart contracts, the common opinion is that tools are immature and ineffective. In this work, we carry out a fault injection campaign to empirically analyze the runtime impact that realistic faults present in smart contracts may have on the reliability of blockchain systems. We pay particular attention to the faults that elude popular smart contract verification tools and show if and in which ways the faults lead the blockchain system to fail at runtime. We map the observations to the fault detection capabilities of three state-of-the-art fault detection tools, namely Mythril, Slither, and Securify. The results show that the tools individually have poor detection capabilities (e.g., Securify with 6.4% accuracy and Mythril with 60% accuracy) or tend to generate false alerts (i.e., only 1.74% of Slither's alerts are correct). The results also show several elusive faults responsible for severe blockchain failures, such as A_MCV, which impacts the integrity of the ledger, and I_MVMSV, which causes gas depletion, just to name a few.

Abstract Smart contracts have accelerated the adoption of blockchain technology across various domains by enabling coded agreements between transaction participants. However, increased software defects and vulnerabilities in smart contracts, driven by developer inexperience with languages like Solidity and a lack of effective detection tools, pose significant risks. Given the high value of assets managed on blockchain (e.g., cryptocurrencies), these vulnerabilities can lead to severe consequences. Researchers and practitioners have proposed numerous smart contract design patterns to mitigate certain faults or vulnerabilities. Despite these efforts, it remains unclear which types of defects these patterns target and how effectively they address the wide range of existing smart contract security vulnerabilities. In this paper, we review the state of the art in smart contract design patterns, categorizing them and analyzing their effectiveness in mitigating known security vulnerabilities. Our findings reveal that only five patterns directly aim to prevent security vulnerabilities, collectively addressing just 6 out of 94 security issues identified by OpenSCV (a state-of-the-art vulnerability taxonomy), highlighting the need for further research on smart contract security design patterns.

Context: Identifying security bugs in software is critical to minimize vulnerability windows. Traditionally, bug reports are submitted through issue trackers and manually analyzed, which is time-consuming. Challenges such as data scarcity and imbalance generally hinder the development of effective machine learning models that could be used to automate this task. Generative Pre-trained Transformer (GPT) models do not require training and are less affected by the imbalance problem. Therefore, they have gained popularity for various text-based classification tasks, apparently becoming a natural highly promising solution for this problem. Objective: This paper explores the potential of using GPT models to identify security bug reports from the perspective of a user of this type of models. We aim to assess their classification performance in this task compared to traditional machine learning (ML) methods, while also investigating how different factors, such as the prompt used and datasets’ characteristics, affect their results. Methods: We evaluate the performance of four state-of-the-art GPT models (i.e., GPT4All-Falcon, Wizard, Instruct, OpenOrca) on the task of security bug report identification. We use three different prompts for each GPT model and compare the results with traditional ML models. The empirical results are based on using bug report data from seven projects (i.e., Ambari, Camel, Derby, Wicket, Nova, OpenStack, and Ubuntu). Results: GPT models show noticeable difficulties in identifying security bug reports, with performance levels generally lower than traditional ML models. The effectiveness of the GPT models is quite variable, depending on the specific model and prompt used, as well as the particular dataset. Conclusion: Although GPT models are nowadays used in many types of tasks, including classification, their current performance in security bug report identification is surprisingly insufficient and inferior to traditional ML models. Further research is needed to address the challenges identified in this paper in order to effectively apply GPT models to this particular domain.

Microservice architectures offer scalability and flexibility, but due to their distributed nature and complex service structures, raise new security challenges, particularly in detecting DoS attacks. Although addressing these challenges calls for innovative attack detection approaches, developing effective solutions requires large-scale experiments and data collection to create representative datasets. This paper proposes a comprehensive framework to support research on the cybersecurity of microservice applications and the development of different methods to detect cyberattacks. The framework comprises two modules: (i) a data generation module that contains the components necessary to create datasets that reflect the behavior of microservices under attack and (ii) a model development and evaluation module suitable for different methods for detecting attacks on microservices. The framework is illustrated and validated by generating realistic high- and low-volume DoS attack data and developing models using supervised and unsupervised Machine Learning (ML) algorithms and a method based on Logic Scoring of Preference (LSP). The results indicate that supervised ML models have the best classification performance, especially with the XGBoost algorithm. Even though unsupervised ML and LSP models have worse performance, they can be used when the attack data are not available or are costly to generate.

Abstract Smart contracts are nowadays at the core of most blockchain systems. Like all computer programs, smart contracts are subject to the presence of residual faults, including severe security vulnerabilities. However, the key distinction lies in how these vulnerabilities are addressed. In smart contracts, when a vulnerability is identified, the affected contract must be terminated within the blockchain, as due to the immutable nature of blockchains, it is impossible to patch a contract once deployed. In this context, research efforts have been focused on proactively preventing the deployment of smart contracts containing vulnerabilities, mainly through the development of vulnerability detection tools. Along with these efforts, several heterogeneous vulnerability classification schemes appeared (e.g., most notably DASP and SWC). At the time of writing, these are mostly outdated initiatives, even though new smart contract vulnerabilities are consistently uncovered. In this paper, we propose OpenSCV, a new and Open hierarchical taxonomy for Smart Contract vulnerabilities, which is open to community contributions and matches the current state of the practice while being prepared to handle future modifications and evolution. The taxonomy was built based on the analysis of the existing research on vulnerability classification, community-maintained classification schemes, and research on smart contract vulnerability detection. We show how OpenSCV covers the announced detection ability of the current vulnerability detection tools and highlight its usefulness in smart contract vulnerability research. To validate OpenSCV, we performed an expert-based analysis wherein we invited multiple experts engaged in smart contract security research to participate in a questionnaire. The feedback from these experts indicated that the categories in OpenSCV are representative, clear, easily understandable, comprehensive, and highly useful. Regarding the vulnerabilities, the experts confirmed that they are easily understandable.

Microservice architectures allow the development of highly scalable, flexible, and manageable systems. However, such architectures raise new security problems and exacerbate the challenge of monitoring applications at runtime due to their high service granularity and distributed nature. Developing effective monitoring and security strategies is thus crucial to effectively detect potential attacks. This paper explores the applicability of Logic Scoring of Preference (LSP), a multi-criteria decision-making method to compute a score based on a set of preferences, for attack detection in microservice applications. We present an extensive experimental study and define a model based on LSP and application-level metrics to characterize the impact of DoS attacks. The output of the model is a unique score used to determine whether a microservice is under a DoS attack. The results of the experimental study show precision, recall, and f1-score rates of more than 80%, indicating that LSP could effectively characterize the application under attack, opening several possibilities for future work.

Identifying security vulnerabilities in issue reports is a complex and time-sensitive task that when carried out effectively and in a timely manner can prevent attackers from exploiting software systems. While it is possible to address this using machine learning, the heavy imbalance of the datasets involved requires a meticulous use of rebalancing methods to achieve reasonably effective models. In this paper we analyze the effectiveness of different data rebalancing methods (e.g., oversampling and undersampling) applied to the classification of security issue reports using machine learning techniques. Our results using the Ubuntu dataset show that oversampling is an overall better strategy for rebalancing, SVMSMOTE and Random Undersampling are the individual methods that show the best performance. We also found that variations in the proportion of the minority class have little effect on the difference in the effectiveness of the best methods. Overall, our results are useful for creating more effective machine learning models for the automatic identification of security bug reports.

Blockchain has become popular due to its use in cryptocurrencies and potential to support different business-critical services (e.g., financial services, retail). The smart contract is at the center of blockchain systems and is a coded specification of an agreement between interacting partners in a transaction. Like other software artifacts, smart contracts are prone to carry residual faults. As many contracts are being used to handle financial transactions, huge losses may occur if a vulnerability is exploited. Also, a faulty contract cannot be corrected once it has been deployed on the blockchain, it can only be terminated and a new one must be deployed, which aggravates the cost of deploying contracts with faults and marks the reputation of the provider. Smart contract verification tools have been emerging, but limited knowledge is available regarding their real effectiveness. In this paper, we define a smart contract defect classification scheme based on the Orthogonal Defect Classification and apply it to a contract dataset, which has been extracted from multiple sources and holds different types of defects. We use the dataset to evaluate three state of the art verification tools regarding their fault detection performance. Results show the relatively low effectiveness of the tools and their complementarity.

Businesses are increasingly deploying their services on the web, in the form of web applications, SOAP services, message-based services, and, more recently, REST services. Although the movement towards REST is widely recognized, there is not much concrete information regarding the technical features being used in the field, such as typical data formats, how HTTP verbs are being used, or typical URI structures, just to name a few. In this paper, we go through the Alexa.com top 4000 most popular sites to identify precisely 500 websites claiming to provide a REST web service API. We analyze these 500 APIs for key technical features, degree of compliance with REST architectural principles (e.g., resource addressability), and for adherence to best practices (e.g., API versioning). We observed several trends (e.g., widespread JSON support, software-generated documentation), but, at the same time, high diversity in services, including differences in adherence to best practices, with only 0.8 percent of services strictly complying with all REST principles. Our results can help practitioners evolve guidelines and standards for designing higher quality services and also understand deficiencies in currently deployed services. Researchers may also benefit from the identification of key research areas, contributing to the deployment of more reliable services.

NoSQL databases are increasingly used for storing and managing data in business-critical Big Data systems. The presence of software defects (i.e., bugs) in these databases can bring in severe consequences to the NoSQL services being offered, such as data loss or service unavailability. Thus, it is essential to understand the types of defects that frequently affect these databases, allowing developers take action in an informed manner (e.g., redirect testing efforts). In this paper, we use Orthogonal Defect Classification (ODC) to classify a total of 4096 software defects from three of the most popular NoSQL databases: MongoDB, Cassandra, and HBase. The results show great similarity for the defects across the three different NoSQL systems and, at the same time, show the differences and heterogeneity regarding research carried out in other domains and types of applications, emphasizing the need for possessing such information. Our results expose the defect distributions in NoSQL databases, provide a foundation for selecting representative defects for NoSQL systems, and, overall, can be useful for developers for verifying and building more reliable NoSQL database systems.

Recent studies have shown that programmers' cognitive load during typical code development activities can be assessed using wearable and low intrusive devices that capture peripheral physiological responses driven by the autonomic nervous system. In particular, measures such as heart rate variability (HRV) and pupillography can be acquired by nonintrusive devices and provide accurate indication of programmers' cognitive load and attention level in code related tasks, which are known elements of human error that potentially lead to software faults. This paper presents an experimental study designed to evaluate the possibility of using HRV and pupillography together with eye tracking to identify and annotate specific code lines (or even finer grain lexical tokens) of the program under development (or under inspection) with information on the cognitive load of the programmer while dealing with such lines of code. The experimental data is discussed in the paper to assess different alternatives for using code annotations representing programmers' cognitive load while producing or reading code. In particular, we propose the use of biofeedback code highlighting techniques to provide online programmer's warnings for potentially problematic code lines that may need a second look at (to remove possible bugs), and biofeedback-driven software testing to optimize testing effort, focusing the tests on code areas with higher bug probability.

The use of cryptography is nowadays common in software systems, with cryptographic libraries widely available to software developers. As such, the likely weakest link in sensitive software has moved from cryptographic function implementations to the application code surrounding such functions. Ordinary developers usually lack knowledge in practical cryptography, and support from specialists is rare. Frequently, these difficulties are addressed by running static analysis tools to automatically detect cryptography misuse during coding and reviews. However, the effectiveness of such tools is not yet well understood. This article studies how well programmatic misuse of cryptography is detected by free static code analysis tools. The performance of such tools in detecting misuse is correlated to coding tasks and use cases commonly found in development efforts; also, cryptography misuse is classified in comprehensive categories, easily recognizable by software security practitioners. Our research shows that the coverage of public-key cryptography by static code analysis tools is full of blind spots, because tools prioritize only those misuses related to the most frequent coding tasks and use cases, while neglecting infrequent use cases. We found that, in addition to a relatively low recall in our tests, evaluated tools also have a small overlap regarding the misuses detected by all the evaluated tools, as well as an intersection of false alarms, suggesting lack of discrimination between specific misuses and corresponding good uses of cryptography. In spite of that, well-selected tools can be useful when developing cryptographic software, but support of experts is still required for solving complex cases.

Summary HTML is being massively used as an interface to provide services to users. Web developers are producing and changing sites at a high pace while trying to support the latest HTML standards. In this context, it is common to find websites that do not comply with the standards and fail to be correctly processed by browsers. Considering this dynamic environment and the increasingly large diversity of browsers with frequent updates, the appearance of problems in web pages is a common, sometimes severe, and hard‐to‐track problem. In this short communication, we describe the initial design of an approach that will be used to obtain information regarding the characteristics of HTML documents on the Web and extract indicators of representative errors made by their developers. Preliminary results show nearly 90% of the pages analyzed having at least one type of error and the prevalence of a small number of error types.

The web services technology has been created to support communication between heterogeneous platforms. Despite its maturity, built upon more than a decade of experience, research and practice show that the technology still fails to connect web service client applications to servers, even when the programming languages involved are the same. This is especially troubling for service providers, as a failure in the inter-operation of web services may lead to disastrous consequences for the services involved, which frequently support businesses. In this paper, we present INTENSE, a service deployed as an on-line web application, designed to test the interoperability of a web service against specific client-side platforms. The tool is able to test the pre-runtime steps involving code generation and the end-to-end runtime communication, present in a web service interaction with a client. We used INTENSE to test a set of web services deployed on Glassfish and WildFly against the well-known Metro JAX-WS, JBossWS, and Axis2 client platforms, which disclosed severe interoperability issues.

The incorrect use of cryptography is a common source of critical software vulnerabilities. As developers lack knowledge in applied cryptography and support from experts is scarce, this situation is frequently addressed by adopting static code analysis tools to automatically detect cryptography misuse during coding and reviews, even if the effectiveness of such tools is far from being well understood. This paper proposes a method for benchmarking static code analysis tools for the detection of cryptography misuse, and evaluates the method in a case study, with the goal of selecting the most adequate tools for specific development contexts. Our method classifies cryptography misuse in nine categories recognized by developers (weak cryptography, poor key management, bad randomness, etc.) and provides the workload, metrics and procedure needed for a fair assessment and comparison of tools. We found that all evaluated tools together detected only 35% of cryptography misuses in our tests. Furthermore, none of the evaluated tools detected insecure elliptic curves, weak parameters in key agreement, and most insecure configurations for RSA and ECDSA. This suggests cryptography misuse is underestimated by tool builders. Despite that, we show that it is possible to benefit from an adequate tool selection during the development of cryptographic software.

An increasingly important requirement for certain classes of software-intensive systems is the ability to self-adapt their structure and behavior at run-time when reacting to changes that may occur to the system, its environment, or its goals. A major challenge related to self-adaptive software systems is the ability to provide assurances of their resilience when facing changes. Since in these systems, the components that act as controllers of a target system incorporate highly complex software, there is the need to analyze the impact that controller failures might have on the services delivered by the system. In this paper, we present a novel approach for evaluating the resilience of self-adaptive software systems by applying robustness testing techniques to the controller to uncover failures that can affect system resilience. The approach for evaluating resilience, which is based on probabilistic model checking, quantifies the probability of satisfaction of system properties when the target system is subject to controller failures. The feasibility of the proposed approach is evaluated in the context of an industrial middleware system used to monitor and manage highly populated networks of devices, which was implemented using the Rainbow framework for architecture-based self-adaptation.

Reliable communication is nowadays pervasively supported by TCP, which is poorly adapted for message-based communications, because it offers a streaming channel with no mechanisms to encapsulate messages. Moreover, TCP does not tolerate connection crashes. Thus, whenever reliable message-based communication is needed, developers either use heavy-weight middleware, like Java Message Service (JMS), or develop their own custom error-prone solutions for recovering from crashes. In this paper, we introduce two TCP-based design patterns that address these limitations, and facilitate the development of light-weight and reliable message-based applications. Our design solutions are modular, in the sense that they build on top of each other.

This emerging technology track focuses on key topics regarding dependability and security of software and services. Service-based systems are being used in business, safety, and mission-critical environments to achieve operational goals and possess special characteristics that bring in difficult challenges to the research and industry communities. Among these challenges, dependability and security have been widely identified as critical aspects that need to be addressed, especially when considering that many services are also nowadays being deployed on the web, used over unreliable networks, and potentially exposed to security threats. The goal of the Emerging Technology Track On Dependable and Secure Services is to bring together researchers and practitioners to present original research and industrial practice regarding techniques to improve the dependability and security of services. Services hold special characteristics, in particular their typically complex nature, high heterogeneity, and fast-changing dynamics. In such scenarios, infrastructure interdependencies, failure and recovery modeling and analysis, accidental threats and attack modeling and evaluation, testing approaches, testbeds, benchmarks, interoperability in presence of dependability and security guarantees, as well as techniques and tools to assess the impact of accidental and malicious threats, metrics for assessing dependability and security are among the crucial aspects to be addressed.

Web Services are designed with the key goal of providing interoperable application-to-application interaction, regardless of the platforms involved. Although experience shows that interoperability is difficult to achieve, developers still have limited tools to assess the interoperability of their services and, to the best of our knowledge, none able to support end-to-end interoperability certification. In this paper, we lay the foundations of an interoperability certification process for Web services, which allows testing the interoperability level of a given Web service and also identifying possible interoperability issues. In practice, the process can be used by developers or providers to certify a given web service for interoperability, ensuring successful interaction with client-side platforms. We show the effectiveness of the process by conducting a large experimental evaluation to certify five different implementations of the services specified by the TPC-App benchmark, and about 2500 synthetic generated services.client-side platforms.

Data is part of our everyday life and an essential asset in numerous businesses and organizations. The quality of the data, i.e., the degree to which the data characteristics fulfill requirements, can have a tremendous impact on the businesses themselves, the companies, or even in human lives. In fact, research and industry reports show that huge amounts of capital are spent to improve the quality of the data being used in many systems, sometimes even only to understand the quality of the information in use. Considering the variety of dimensions, characteristics, business views, or simply the specificities of the systems being evaluated, understanding how to measure data quality can be an extremely difficult task. In this paper we survey the state of the art in classification of poor data, including the definition of dimensions and specific data problems, we identify frequently used dimensions and map data quality problems to the identified dimensions. The huge variety of terms and definitions found suggests that further standardization efforts are required. Also, data quality research on Big Data appears to be in its initial steps, leaving open space for further research.

This visionary track theme focuses on dependability and security of software and services. Service-based systems are being used in business and safety-critical environments to achieve operational goals and possess special characteristics that have bring difficult challenges to the research and industry communities. Among these challenges, dependability and security have been widely identified as critical aspects that need to be addressed, especially when considering that services are being deployed on the web, and used over unreliable networks to perform critical functions.

Web Services are a set of technologies designed to support the invocation of remote services by client applications, with the key goal of providing interoperable application-to-application interaction while supporting vendor and platform independence. The goal of this work is to study the real level of interoperability provided by these technologies through a massive experimental campaign involving a wide set of very popular frameworks for web services, implemented using seven different programming languages. We have tested the inter-operation of eleven client-side framework subsystems with three of the most widely used server-side implementations, each one hosting thousands of different services. The results highlight numerous situations where the goal of interoperability between different frameworks is not met due to problems both on the client and the server side. Moreover, we have identified issues also affecting interactions between the client and server subsystems of the same framework.

Web services are supported by a set of protocols that have been designed with the main goal of providing interoperable communication to applications. In typical business-critical services environments the occurrence of interoperability issues can have disastrous consequences, including direct financial costs, reputation, and client fidelity losses. Despite this, experience suggests that interoperability is still quite difficult to achieve, since the heterogeneity of frameworks for providing web services is quite large. In addition, current tools have limited testing capabilities and, in many cases do not specialize in this problem. In this paper we present ITWS, an extensible Interoperability Testing tool for Web Services that is able to assess the interoperability of a web service, supported by any given framework. We have used ITWS to test the interoperability of a set of home-implemented TPC-App web services and a set of thousands of web services created in .NET C# against 11 client-side web service frameworks, including frameworks for mainstream programming languages. Numerous issues have been disclosed, showing the benefits of using ITWS and the importance of testing services for interoperability.

This workshop focuses on dependability and security of software and services. Service-based systems are being used in business and safety-critical environments to achieve operational goals and possess special characteristics that have been bringing difficult challenges to the research and industry communities for several years now. Among such challenges, dependability and security have been widely identified as critical aspects that need to be addressed, especially when considering that many times services are being deployed on the web, and used over unreliable networks to perform critical functions.

Developing robust web services is a difficult task. Field studies show that a large number of web services are deployed with robustness problems (i.e., presenting unexpected behaviors in the presence of invalid inputs). Although several techniques for the identification of robustness problems have been proposed in the past, there is no practical approach to automatically fix those problems. This paper proposes a mechanism that automatically fixes robustness problems in web services. The approach consists of using robustness testing to detect robustness issues and then mitigate those issues by applying inputs verification based on well-defined parameter domains, including domain dependencies between different parameters. This integrated and fully automated methodology has been used to improve three different implementations of the TPC-App web services and several services publicly available on the Internet. Results show that the proposed approach can be easily used to improve the robustness of web services code.

Business critical applications are increasingly being deployed as web services that access database systems, and must provide secure operations to its clients. Although the open web environment emphasizes the need for security, several studies show that web services are still being deployed with command injection vulnerabilities. This paper proposes a learning-based approach to secure web services against SQL and XPath Injection attacks. Our approach is able to transparently learn valid request patterns (learning phase) and then detect and abort potentially harmful requests (protection phase). When it is not possible to have a complete learning phase, a set of heuristics can be used to accept/discard doubtful cases. Our mechanism was applied to secure TPC-App services and open source services. It showed to be extremely effective in stopping all tested attacks, while introducing a negligible performance impact.

Testing web services for robustness is an effective way of disclosing software bugs. However, when executing robustness tests, a very large amount of service responses has to be manually classified to distinguish regular responses from responses that indicate robustness problems. Besides requiring a large amount of time and effort, this complex classification process can easily lead to errors resulting from the human intervention in such a laborious task. Text classification algorithms have been applied successfully in many contexts (e.g., spam identification, text categorization, etc) and are considered a powerful tool for the successful automation of several classification-based tasks. In this paper we present a study on the applicability of five widely used text classification algorithms in the context of web services robustness testing. In practice, we assess the effectiveness of Support Vector Machines, Naïve Bayes, Large Linear Classification, K-nearest neighbor (Ibk), and Hyperpipes in classifying web services responses. Results indicate that these algorithms can be effectively used to automate the identification of robustness issues while reducing human intervention. However, in all mechanisms there are cases of misclassified responses, which means that there is space for improvement.

Developing robust web services is a difficult task. Field studies show that a large number of web services are deployed with robustness problems (i.e., presenting unexpected behaviors in the presence of invalid inputs). Several techniques for the identification of robustness problems have been proposed in the past. This paper proposes a mechanism that automatically fixes the problems detected. The approach consists of using robustness testing to detect robustness issues and then mitigate those issues by applying inputs verification based on well-defined parameter domains, including domain dependencies between different parameters. This integrated and fully automatable methodology has been used to improve three different implementations of the TPC-App web services. Results show that this tool can be easily used by developers to improve the robustness of web services implementations.

Current business critical environments increasingly rely on SOA standards to execute business operations. These operations are frequently based on Web service compositions that use several Web services over the internet and have to fulfill specific timing constraints. In these environments, an operation that does not conclude in due time may have a high cost as it can easily turn into service abandonment with financial and prestige losses to the service provider. In fact, at certain points, carrying on with the execution of an operation may be useless as a timely response will be impossible to obtain. This paper proposes a time-aware programming model for Web services that provides transparent timing failure detection. The paper illustrates the proposed model using a set of services specified by the TPC-App performance benchmark.

Web-services are supported by a complex software infrastructure that must provide a robust service to the client applications. This practical experience report presents a practical approach for the evaluation of the robustness of Web-services infrastructures. A set of robustness tests (i.e., invalid web-services call parameters) is applied during Web-services execution in order to reveal possible robustness problems in the Web-services code and in the application server infrastructure. The approach is illustrated using two different implementations of the Web-services specified by the TPC-App performance benchmark running on top of the JBoss application server. The proposed approach is generic and can be used to evaluate the robustness of Web-services implementations (relevant for programmers) and application server infrastructures (relevant for administrators and system integrators).

Web-services are supported by a complex software infrastructure that must ensure high performance and availability to the client applications. Web services industry holds a well established platform for performance benchmarking (e.g., TPC-App and SPEC jAppServer2004 benchmarks). In addition, several studies have been published recently by main vendors focusing web services performance. However, as peak performance evaluation has been the main focus, the characterization of the impact of faults in such systems has been largely disregarded. This paper proposes an approach for the evaluation and comparison of performance and recovery time in web services infrastructures. This approach is based on fault injection and is illustrated through a concrete example of benchmarking three alternative software solutions for web services deployment.

This paper proposes an approach for the evaluation of the robustness of web services, which are complex software components that must provide a robust interface to the client applications. However, although web services are becoming business-critical components, there is no practical way to assess the robustness of the code or to compare alternative implementations concerning robustness. The approach proposed is based on a set of robustness tests (i.e., invalid web services call parameters) that is applied in order to discover both programming and design errors. The web services are classified based on the failures observed during the execution of the tests. The approach is illustrated by evaluating several web services publicly available in the Internet and two different implementations of the web services specified by the standard TPC-App performance benchmark. The proposed approach is useful for both web services providers (to assess the robustness of their web services code) and consumers (to select the web services that best fit their requirements).