Service-oriented architecture (SOA) techniques are being increasingly used for developing critical applications, especially network-centric systems. While the SOA paradigm provides flexibility and agility to better respond to changing business requirements, the task of assessing the reliability of SOA-based systems is challenging, especially for composite services. However, deriving high confidence reliability estimates for mission- critical systems can require huge costs and time. This paper presents a reliability assessment and prediction model for SOA-based systems. The services are assumed to be realized with reuse and logical composition of components. The model uses AI reasoning techniques on dynamically collected failure data of each service and its components as one of the evidences together with results from random testing. Memory-Based Reasoning technique and Bayesian Belief Networks are used as reasoning tools to guide the prediction analysis. The least tested and "high usage " input subdomains are identified and necessary remedial actions are taken depending on the predicted results from the proposed model. The model is illustrated using a simulated case study based on a real-time dataset from the NASA software repository.

Due to the dynamic nature, such as runtime composition and evaluation, it is critical for an SOA system to consider its data provenance, which concerns security, reliability, and integrity of data as it is routed in the system. In a traditional software system, one can focus on the software itself to determine the security, reliability, and integrity; however, in an SOA system, one also needs to consider the origins and routes of data and their impact, i.e., data provenance. This paper first analyzes the unique natures and characteristics of data provenance in an SOA system. Then it proposes a new framework to classify data provenance and collect data in SOA systems. Finally, this paper uses an example to illustrate these concepts and techniques

Service-oriented computing is making strides due to acceptance by government and major computer and software companies, however there are several issues that we need to address. SOA is related to a number of traditional professional areas such as business models, programming languages, model construction, verification, software architecture and design, software reusability, databases, ontology, autonomic computing, grid computing, and computer networks.

The architecture of SOA-based applications is different from traditional software architecture where the architecture is mainly static. The architecture of an SOA-based application is dynamic, i.e., the application may be composed at runtime using existing services. Thus SOA has provided a new direction for software architecture study, where the architecture is determined at runtime and architecture can be dynamically changed at runtime to meet the new software requirements. This paper proposes an architecture classification scheme for SOA-based applications. Using this classification, several well-known SOA-based applications are reviewed including the architectures proposed and adopted by major computer companies and standard organizations. The architecture classification provides a unified way to evaluate a variety of architectures for SOA-based applications.

This paper proposes a rule-based expert system model to assist developers in enhancing the quality of embedded software. The expert system supports semi-automated source code enhancements from multiple perspectives including optimization, reconfiguration to achieve quality of service (QoS) trade-offs, conformance to coding standards, etc., by defining a rule base that is composed of modularized and extensible rule sets. The rule sets are composed of re-usable rules; the rules capture the in-depth knowledge of different software experts. The expert system also adopts a technique to support automated fact generation. This allows the expert system to support processing programs developed in different programming languages

Web services and service-oriented architecture are emerging technologies that are changing the way we develop and use computer software. Due to the standardization of Web services related description languages and protocols, as well as the open platforms, for the same Web service specification, many different implementations can be offered from different service providers. This paper presents an adaptive group testing technique that can test large number Web services simultaneously and effectively. Based on a case study, experiments are performed to validate the correctness and effectiveness of the technique.

This paper presents an integrated development process for Web services. The key differences with the traditional software development are that this new process involves collaboration and cooperation among all parties involved: developers, brokers, and clients. The process defines a new way of developing trustworthy Web services based on the existing Internet infrastructure. The paper serves as a roadmap to research on Web services specification, discovery, ontology, composition, re-composition, testing, reliability assessing, ranking, and collaboration and cooperation among all parities involved in Web services research, development, and application.

Web services are emerging technologies that are changing the way we develop and use computer systems and software. Current Web services testing techniques are unable to assure the desired level of trustworthiness, which presents a barrier to WS applications in mission and business critical environments. This paper presents a framework that assures the trustworthiness of Web services. New assurance techniques are developed within the framework, including specification verification via completeness and consistency checking, specification refinement, distributed Web services development, test case generation, and automated Web services testing. Traditional test case generation methods only generate positive test cases that verify the functionality of software. The Swiss cheese test case generation method proposed in this paper is designed to perform both positive and negative testing that also reveal the vulnerability of Web services. This integrated development process is implemented in a case study. The experimental evaluation demonstrates the effectiveness of this approach. It also reveals that the Swiss cheese negative testing detects even more faults than positive testing and thus significantly reduces the vulnerability of Web services.

Web services emerge as a new paradigm for distributed computing. Model checking is an important verification method to ensure the trustworthiness of composite WS. Boolean abstraction and counterexample driven refinement are major techniques for model checking software and WS. In most of the literature, the refinement is governed by the precision of the abstraction. In this paper, we present an innovative technique to distribute the precision information among proof slices, which can be selectively reused by future proofs and hence improve the performance by reducing excessive invocations of theorem provers. Moreover, the reuse approach is flexible for virtually arbitrary future extension. Our theoretical framework subsumes several existing abstraction-based model checking techniques, e.g., lazy abstraction. Besides the correctness and termination proofs, we also conducted theoretical analysis on the performance of the proof slicing algorithm.

Web services form a new distributed computing paradigm. Collaborative verification and validation are important when Web services from different vendors are integrated together to carry out a coherent task. This paper presents a new approach to verify Web services by model checking the process model of OWL-S (Web ontology language for Web services) and to validate them by the test cases automatically generated in the model checking process. We extend the BLAST, a model checker that handles control flow model naturally, to handle the concurrency in OWL-S. We also propose enhancement in OWL-S and PDDL (Planning Domain Definition Language) to facilitate the automated test case generation. Experiments on realistic examples are provided to illustrate the process.

Process-control systems usually deal with changeable environments so that high adaptability of these systems is required. After a process-control system has been implemented and deployed, it is not an easy task to adapt the system to satisfy new requirements. Considering this problem, we propose the use of relational program architectures to design process-control systems for high adaptability. With the relational program architecture, a process-control system is designed to be the composition of several Independently Developable End-user Assessable Logical (IDEAL) components through pre-defined composition patterns. Each IDEAL component is associated with several end-user visible properties and can be solved and validated in its restricted "view" of the system, i.e., it can be designed and implemented independently, and can be tested or verified by the end-user independently. In addition, the system level properties (safety, stability, and reliability) can be inferred from the individual IDEAL components mathematically. The relational program architecture can provide not only the ultra-high dependability assurance of a process-control system, but can also make the system highly adaptable. We propose several principles, by which we can accurately determine which components of the system need to be adapted and how to adapt them after the system's requirement has changed. These principles can be evolved to automated tool support.

Many critical application systems require reconfigurability in order to adapt under changing environment. To achieve system reconfigurability, it is possible to use adaptation mechanisms that are external to components or to use parameterized components. This paper focuses on making components reconfigurable in terms of QoS tradeoffs. We propose a new component parameterization technique which can transform individual components to reconfigurable ones such that they can support various QoS tradeoffs to achieve overall system QoS requirements. The parameterized components can be used to develop both statically and dynamically reconfigurable systems. The proposed technique is rule based, which provides flexibility and extensibility to enhance and improve the technique. A tool set has been developed based on the proposed technique to achieve semi-automated component parameterization.

Recently service-oriented architecture (SOA) has received significant attention and one reason is that it is potentially survivable as services are located, bound, and executed at runtime over the Internet. However, this is not enough for dependable computing because the system must also be able to reconfigure once a system failure or overload is detected, and this reconfiguration must be done in real-time at runtime with minimum disruption to the current operation. This work presents reconfiguration requirements for building dependable SOA, and proposes a dynamic reconfiguration framework based on distributed monitoring, synchronization, and runtime verification with distributed agents.

Verification pattern (VP) is a new technique to test embedded systems rapidly, and it has been used to test industrial safety-critical embedded systems successfully. The key concept of this approach is to classify system scenarios into patterns, and use the same code template to test all the scenarios of the same pattern. In this way, testing effort can be greatly reduced. This paper extends VPs so that they can fully interoperate with a formalized scenario model ACDATE. In this way, various static and dynamic analyses can be performed on system scenarios as well as on system patterns. Furthermore, this paper provides a mapping from system scenarios into temporal logic expressions. In this way, a practitioner can specify system constraints in scenarios, and follow the mapping to obtain the temporal logic expressions easily to perform formal model checking. This paper also provides an OO framework to support automated test script development from VPs. In this way, VPs can be used in an integrated process where both semi-formal analyses and formal techniques can be used together to develop mission-critical embedded applications.

Quality and risk concerns currently limit the application of commercial off-the-shelf (COTS) software components to non-critical applications. Software metrics can quantify factors contributing to the overall quality of a component-based system, and models for tradeoffs between cost and various aspects of quality can guide quality and risk management by identifying and eliminating sources of risk. This paper discusses metrics and models that can be used to alleviate quality concerns for COTS-based systems, enabling the use of COTS components in a broader range of applications.

This paper presents a temporal model for the coverage achieved by software testing. The proposed model, which is applicable at any level of the testing hierarchy, can determine the value of test coverage at any given time, as well as predicting future values. The model is comprised of two main components: coverage functions, and the coverage matrix. The coverage functions represent the coverage of a single entity as a function of time and reflect the test environment through their stochastic parameters. The coverage matrix utilizes the coverage functions to depict the coverage attained for each entity by each test within the test suite. A normalized sum of the elements of the coverage matrix is used to represent the overall coverage achieved by the test suite, as a function of time. The application of the model to multi-phase testing is illustrated In the application section, test coverage values from Y2K compliance testing are used to verify model predictions.

This paper presents a process to develop adaptive object-oriented scenario-based test frameworks for testing embedded systems. Embedded systems often require rigorous testing due to the mission-critical nature of their applications, and they are often developed as a family of products. The process uses techniques such as design-for-change, design patterns, scenarios, ripple effect analysis, and regression testing. This paper then uses an example to illustrate this process by applying it to test a mobile phone system, and the framework constructed can facilitate generation of numerous test cases quickly with minimum effort, and it can also accommodate many changes suggested by another party without changing the overall structure of the framework.

The goal of accurate software measurement data analysis is to increase the understanding and improvement of software development process together with increased product quality and reliability. Several techniques have been proposed to enhance the reliability prediction of software systems using the stored measurement data, but no single method has proved to be completely effective. One of the critical parameters for software prediction systems is the size of the measurement data set, with large data sets providing better reliability estimates. In this paper, we propose a software defect classification method that allows defect data from multiple projects and multiple independent vendors to be combined together to obtain large data sets. We also show that once a sufficient amount of information has been collected, the memory-based reasoning technique can be applied to projects that are not in the analysis set to predict their reliabilities and guide their testing process. Finally, the result of applying this approach to the analysis of defect data generated from fault-injection simulation is presented.

To assure the quality of software by running test cases and evaluating the results is one of the difficult parts of the entire software development project. The difficulty usually comes from the lack of appropriate supporting tools and the complexity of the software. In the past ad hoc supporting tools were made for each project and test results were usually not used across projects. This conventional way of test and evaluation (T&E) is time consuming, and the most important decision "When is this software ready to ship?" is left to the engineers depending on their experiences. Our objective is to build a knowledge-based T&E environment such that tests cases, test results, object snapshots and other information are accumulated in a database. These longitudinal data can be automatically tracked and analyzed to provide decision support information. As a result, test results can be reviewed repeatedly and software quality can be assured by analyzing these data from various perspectives.

Several methods have been explored for assuring the reliability of mission critical systems (MCS), but no single method has proved to be completely effective. This paper presents an approach for quantifying the confidence in the probability that a program is free of specific classes of defects. The method uses memory-based reasoning techniques to admit a variety of data from a variety of projects for the purpose of assessing new systems. Once a sufficient amount of information has been collected, the statistical results can be applied to programs that are not in the analysis set to predict their reliabilities and guide the testing process. The approach is applied to the analysis of Y2K defects based on defect data generated using fault-injection simulation.

Regression testing has been a popular quality-assurance technique. Most regression testing techniques are based on code or software design. This paper proposes a scenario-based functional regression testing, which is based on end-to-end (E2E) integration test scenarios. The test scenarios are first represented in a template model that embodies both test dependency and traceability. By using test dependency information, one can obtain a test slicing algorithm to detect the scenarios that are affected and thus they are candidates for regression testing. By using traceability information, one can find affected components and their associated test scenarios and test cases for regression testing. With the same dependency and traceability information one can use the ripple effect analysis to identify all affected, including directly or indirectly, scenarios and thus the set of test cases can be selected for regression testing. This paper also provides several alternative test-case selection approaches and a hybrid approach to meet various requirements. A web-based tool has been developed to support these regression testing tasks.

The growing reliance on commercial-off-the-shelf (COTS) components for developing large-scale projects introduces a new paradigm in software engineering, which requires the design of new software development and business processes. Large scale component reuse leads to savings in development resources, enabling these resources to be applied to areas such as quality improvement. These savings come at the price of integration difficulties, performance constraints, and incompatibility of components from multiple vendors. Relying on COTS components also increases the system's vulnerability to risks arising from third-party development, which can negatively affect the quality of the system, as well as causing expenses not incurred in traditional software development. We aim to alleviate such concerns by using software metrics to accurately quantify factors contributing to the overall quality of a component-based system, guiding quality and risk management by identifying and eliminating sources of risk.

Integration testing has always been a challenge especially if the system under test is large with many subsystems and interfaces. This paper proposes an approach to design End-to-End (E2E) integration testing, including test scenario specification, test case generation and tool support. Test scenarios are specified as thin threads, each of which represents a single function from an end user's point of view. Thin threads can be organized hierarchically into a tree with each branch consisting of a set of related thin threads representing a set of related functionality. A test engineer can use thin-thread trees to generate test cases systematically, as well as carry out other related tasks such as risk analysis and assignment, regression testing, ripple effect analysis. A prototype tool has been developed to support E2E testing in a distributed environment on the J2EE platform.

Testing is the primary means for quality assurance for enterprise systems, and integration testing is often the most time consuming and expensive part of testing. Recently Department of Defense proposed an End-to-End (E2E) integration testing process to address the challenge of testing large integrated information systems. The E2E testing activities include test thin-thread tree construction, condition tree specification, test configuration management, risk analysis, regression testing, ripple effect analysis, test scenario/case generation, rest result analysis, statistical analysis, and project management. A tool has been developed to support this E2E testing on J2EE using EJB and XML with Cloudscape relational database management system. This web-based tool also allows distributed collaboration and remote project management by the E2E testing participants including project managers, contractors, designers and testers.

Real time simulation is an advanced mode of simulation in which the simulator components are designed to show essentially the same timing behavior that the simulation targets do. Distributed real time simulation is a field in its infancy but its practice is under increasing demands. In recent years the author and his collaborators have been establishing a new approach called the distributed time triggered simulation (DTS) scheme which is conceptually simple and easy to use but widely applicable. The concept was initiated in the course of developing a new-generation object oriented real time programming scheme called the time triggered message triggered object (TMO) programming scheme. Some fundamental issues inherent in distributed real time simulation and major design principles and implementation techniques for resolving those issues within the DTS framework are presented. This technical foundation has been identified through multiple major DTS experiments conducted over the years. Some issues that require further research to realize the full potentials of the DTS scheme are also discussed.

The emergence of new technology and global DoD commitments are transforming what was recently a relatively centralized IT and decision-making infrastructure. The new challenges facing DoD missions and IT applications require DOD's traditional needs for the IT enterprise to be transformed into a powerful infrastructure of new-generation technology. The new technology must provide capabilities to collect, process and disseminate an uninterrupted flow of secure information, logistics and knowledge that will support dispersed and mobile services to users anywhere, and at any time. The goal is to provide such an improved infrastructure. GIG will allow unfettered collaboration among DoD services and users with high levels of agility. To successfully realize the GIG infrastructure, a comprehensive plan is needed to achieve strategic business objectives through the effective exploitation of the DoD IT portfolio management and cost-effective acquisition of new technology. Fulfilling the stated mission requires achieving the following principal objectives, and addressing their challenges: (a) management and control of assets, (b) quantitative and qualitative evaluation of assets and their investment, and (c) distributed infrastructure analysis. These are the orthogonal dimensions that capture the challenges faced.

The last several years have seen a tremendous proliferation of applications utilizing resources from a distributed system, such as the Internet. Attractive applications include real-time command and control systems, distributed multimedia information systems, e-business infrastructures, telemedicine and manufacturing. Decentralization of applications offers enhanced capabilities such as fast access to remote data and virtually unlimited storage capacity, but significantly complicates the resource management required for quality assurance. Supporting the diverse quality of service (QoS) requirements of distributed applications is a critical and challenging problem involving database, networking and data security technologies. The paper presents a distributed system resource management infrastructure.

Most software reliability methods have been developed to predict the reliability of a program using only data gathered during the resting and validation of a specific program. Hence, the confidence that can be attained in the reliability estimate is limited since practical resource constraints can result in a statistically small sample set. One exception is the Orthogonal Defect Classification (ODC) method, which uses data gathered from several projects to track the reliability of a new program, Combining ODC with root-cause analysis can be useful in many applications where it is important to know the reliability of a program for a specific type of a fault. By focusing on specific classes of defects, it becomes possible to (a) construct a detailed model of the defect and (b) use data from a large number of programs. In this paper, we develop one such approach and demonstrate its application to modeling Y2K defects.

The construction and maintenance of large, high-quality software projects is a complex, error-prone and difficult process. Tools employing software database metrics can play an important role in the efficient execution and management of such large projects. In this paper, we present a generic framework to address this problem. This framework incorporates database and knowledge-base tools, a formal set of software test and evaluation metrics, and a suite of advanced analytic techniques for extracting information and knowledge from available data. The proposed combination of critical metrics and analytic tools can enable highly efficient and cost-effective management of large and complex software projects. The framework has the potential for greatly reducing venture risks and enhancing production quality in the domain of large-scale software project management.

We propose a dynamic bandwidth management scheme for asynchronous LANs that uses the concept of time division multiple access (TDMA). The scheme is to ensure quality of service (QoS) for multimedia applications. Significant performance improvement is observed through experimental results. In particular, the transmission rates for multimedia hosts improve significantly with low jitter variations in media streams. We also propose a framework for graceful degradation of play-out quality of multimedia objects in case the LAN's total capacity is not sufficient to meet the overall demand.

The concept of software architecture has recently emerged as a new way to improve our ability to effectively construct large scale software systems. However, there is no formal architecture specification language available to model and analyze temporal properties of complex real-time systems. In this paper, an object-oriented logic-based architecture specification language for real-time systems is discussed. Representation of the temporal properties and timing constraints, and their integration with the language to model real-time concurrent systems is given. Architecture based specification languages enable the construction of large system architectures and provide a means of testing and validation. In general, checking the timing constraints of real-time systems is done by applying model checking to the constraint expressed as a formula in temporal logic. The complexity of such a formal method depends on the size of the representation of the system. It is possible that this size could increase exponentially when the system consists of several concurrently executing real-time processes. This means that the complexity of the algorithm will be exponential in the number of processes of the system and thus the size of the system becomes a limiting factor. Such a problem has been defined in the literature as "state explosion problem". We propose a method of incremental verification of architectural specifications for real-time systems. The method has a lower complexity in a sense that it does not work on the whole state space, but only on a subset of it that is relevant to the property to be verified.

The authors present an approach to manage evolutionary changes that take place over time in an object-oriented software metrics database schema. The framework is based on the entity-relation (E-R) model and uses a recursive graph structure, known as R-graph, to support the views of software quality and risk. The fundamental mechanism for abstracting views is the introduction of two semantic operators for the R-graph. These operators are proposed based on graph-based predicates and Petri-net based predicate formalism for view abstraction.

A metrics based project management system (MPMS) integrates software metrics data, management queries based on the software metrics data, as well as their responses in a single location. Given the variations in projects, queries, metrics and physical databases to be used, MPMSes should be flexible from a user point of view, and also generic from a designer point of view. Two important steps in the design of an MPMS involve the determination of typical queries that may be posed by users, as well as the selection of the physical database for storing metrics and other project related knowledge. The paper discusses the relative advantages and drawbacks of three selected physical database models, namely the relational data model, object oriented data model, and graph data model, based on a variety of classes of metrics based queries. We conclude that the choice of data model varies according to the class of queries, and has strong implications on the ease and efficiency with which metrics based queries may be posed and executed.

We present a framework for managing large software projects using metrics. The collection and analysis of metrics data serve as a consistent mechanism for risk identification and quality management. The crux of this mechanism is the temporal modeling of metrics data. We present a comprehensive set of temporal operators which serve as the building block for the formal specification of users' views of risk and quality. The notion of temporal inheritance can be used to define an object oriented query language.

This paper presents a discussion of significant issues in selection of a standardized set of the "best" software metrics to support a software reuse program. This discussion illustrates the difficulty in selection of a standardized set of reuse metrics because the "best" reuse metrics are determined by unique characteristics of each reuse application. An example of the selection of a single set of reuse metrics for a specific management situation is also presented.

Discusses issues related to the design and and usage of database management systems consisting of massive amounts of data in parallel environments. The issues include the placement of the data in the memory, file systems, concurrent access to data, effects on query processing, the implications of specific machine architectures, and the peculiarities of specific parallel systems. Since not all parameters are currently amenable to rigorous analysis, results of performance studies are highlighted wherever they are deemed appropriate. >

Discusses the current state-of-the-art issues in multimedia database management systems. The most important aspect of data management is the semantic modeling and indexing of multimedia information. We discuss various schemes to represent temporal synchronization requirements and highlight current research challenges facing the multimedia database community.>

An important objective of software test programs is to identity, "high-risk" components. This paper focuses on one method which can be applied to identify high-risk software components, the use of a classification tree with an established software metrics set. The selected examples of high-risk software components are those modules which are most likely to induce errors in the target operational system, and those software components which will require the most effort in the development process. The associated metrics are software reliability and productivity. This paper describes the methodology utilized by the US Army in the application of classification trees for analysis of software metrics data. A detailed example is provided with a step-by-step procedure for construction of a classification tree for software metrics analysis.>

The neural network described performs classification of software metrics. It is a three-layer, error back-propagation network. Using historical data, the neural network learns the relationship between certain metrics and a particular classification. The neural network selects the classification which best fits the input metrics. The capability of neural networks to classify nonlinearly separable problem spaces gives them an advantage over tree-based and linear network-based classification methods. When applied to actual software metrics, the neural network correctly classified 100% of the data presented.>