Developers of highly configurable performance-intensive software systems often use in-house performance-oriented "regression testing" to ensure that their modifications do not adversely affect their software's performance across its large configuration space. Unfortunately, time and resource constraints can limit in-house testing to a relatively small number of possible configurations, followed by unreliable extrapolation from these results to the entire configuration space. As a result, many performance bottlenecks escape detection until systems are fielded. In our earlier work, we improved the situation outlined above by developing an initial quality assurance process called "main effects screening". This process 1) executes formally designed experiments to identify an appropriate subset of configurations on which to base the performance-oriented regression testing, 2) executes benchmarks on this subset whenever the software changes, and 3) provides tool support for executing these actions on in-the-field and in-house computing resources. Our initial process had several limitations, however, since it was manually configured (which was tedious and error-prone) and relied on strong and untested assumptions for its accuracy (which made its use unacceptably risky in practice). This paper presents a new quality assurance process called "reliable effects screening" that provides three significant improvements to our earlier work. First, it allows developers to economically verify key assumptions during process execution. Second, it integrates several model-driven engineering tools to make process configuration and execution much easier and less error prone. Third, we evaluate this process via several feasibility studies of three large, widely used performance-intensive software frameworks. Our results indicate that reliable effects screening can detect performance degradation in large-scale systems more reliably and with significantly less resources than conventional techniques

Product-line architectures (PLAs) are an emerging paradigm for developing software families for distributed real-time and embedded (DRE) systems by customizing reusable artifacts, rather than hand-crafting software from scratch. To reduce the effort of developing software PLAs and product variants for DRE systems, developers are applying general-purpose -- ideally standard -- middleware platforms whose reusable services and mechanisms support a range of application quality of service (QoS) requirements, such as low latency and jitter. The generality and flexibility of standard middleware, however, often results in excessive time/space overhead for DRE systems, due to lack of optimizations tailored to meet the specific QoS requirements of different product variants in a PLA.This paper provides the following contributions to the study of middleware specialization techniques for PLA-based DRE systems. First, we identify key dimensions of generality in standard middleware stemming from framework implementations, deployment platforms, and middleware standards. Second, we illustrate how context-specific specialization techniques can be automated and used to tailor standard middleware to better meet the QoS needs of different PLA product variants. Third, we quantify the benefits of applying automated tools to specialize a standard Realtime CORBA middleware implementation. When applied together, these middleware specializations improved our application product variant throughput by ~65%, average- and worst-case end-to-end latency measures by ~43% and ~45%, respectively, and predictability by a factor of two over an already optimized middleware implementation, with little or no effect on portability, standard middleware APIs, or application software implementations, and interoperability.

Developers of highly configurable performance-intensive software systems often use a type of in-house performance-oriented "regression testing" to ensure that their modifications have not adversely affected their software's performance across its large configuration space. Unfortunately, time and resource constraints often limit developers to in-house testing of a small number of configurations and unreliable extrapolation from these results to the entire configuration space, which allows many performance bottlenecks and sources of QoS degradation to escape detection until systems are fielded. To improve performance assessment of evolving systems across large configuration spaces, we have developed a distributed continuous quality assurance (DCQA) process called main effects screening that uses in-the-field resources to execute formally designed experiments to help reduce the configuration space, thereby allowing developers to perform more targeted in-house QA. We have evaluated this process via several feasibility studies on several large, widely-used performance-intensive software systems. Our results indicate that main effects screening can detect key sources of performance degradation in large-scale systems with significantly less effort than conventional techniques.

This paper provides two contributions to R&D on model-driven development (MDD) techniques that help codify the impact of middleware configurations on end-to-end distributed real-time and embedded (DRE) system quality of service (QoS). First, we describe how MDD techniques can help select middleware configuration parameters that satisfy key functional and QoS requirements of DRE systems. Second, we apply our MDD techniques to empirically evaluate the end-to-end QoS of representative DRE systems in the avionics and industrial manufacturing domains. Our results show how MDD techniques significantly enhance conventional ad hoc processes used by developers to configure middleware that meets the QoS needs of DRE systems.

The report summarizes the results of the Workshop on Middleware Benchmarking held during OOPSLA 2003. The goal of the workshop was to help advance the current practice of gathering performance characteristics of middleware implementations through benchmarking. The participants of the workshop have focused on identifying requirements of and obstacles to middleware benchmarking and forming a position on the related issues. Selected requirements and obstacles are presented, together with guidelines to adhere to when benchmarking, open issues of current practice, and perspectives on further research. Copyright © 2005 John Wiley & Sons, Ltd.

End-to-end middleware predictability is essential to support quality of service (QoS) capabilities needed by distributed real-time and embedded (DRE) applications. Real-time CORBA is a middleware standard that allows DRE applications to allocate, schedule, and control the QoS of CPU, memory, and networking resources. Existing real-time CORBA solutions are implemented in C++, which is generally more complicated and error-prone to program than Java. The real-time specification for Java (RTSJ) provides extensions that enable Java to be used for developing DRE systems. Real-time CORBA does not currently leverage key RTSJ features, such as scoped memory and real-time threads. Thus, integration of real-time CORBA and RTSJ is essential to ensure the predictability required for Java-based DRE applications. We provide the following contributions to the study of middleware for DRE applications. First we analyze the architecture of ZEN, our implementation of real-time CORBA, identifying sources for the application of RTSJ features. Second, we describe how RTSJ features, such as scoped memory and real-time threads, can be associated with key ORB components to enhance the predictability of DRE applications using realtime CORBA and the RTSJ. Third, we perform preliminary qualitative and quantitative analysis of predictability enhancements arising from our application of RTSJ features. Our results show that use of RTSJ features can considerably improve the predictability of DRE applications written using Real-time CORBA and real-time Java.

Commercial off-the-shelf (COTS) middleware is now widely used to develop distributed real-time and embedded (DRE) systems. DRE systems are themselves increasingly combined to form "systems of systems" that have diverse quality of service (QoS) requirements. Earlier generations of COTS middleware, such as Object Request Brokers (ORBs) based on the CORBA 2.x standard, do not facilitate the separation of QoS policies from application functionality, which makes it hard to configure and validate complex DRE applications. The new generation of component middleware, such as the CORBA component model (CCM) based on the CORBA 3.0 standard, addresses the limitations of earlier generation middleware by establishing standards for implementing, packaging, assembling, and deploying component implementations. There has been little systematic empirical study of the performance characteristics of component middleware implementations in the context of DRE systems. This paper therefore provides three contributions to the study of CCM for DRE systems. First, we describe the challenges involved in benchmarking different CORBA component model (CCM) implementations. Second, we describe key criteria for comparing different CCM implementations using key black-box and white-box metrics. Third, we describe the design of our CCMPerf benchmarking suite to illustrate test categories that evaluate aspects of CCM implementation to determine their suitability for the DRE domain. We demonstrate CCMPerf by using it to collect metrics from a CCM implementation designed for DRE applications.