The convergence of technologies is happening across various aspects, such as communication, computing, medicine, and transportation. The smartphone is a perfect example of convergence, packing features, such as a camera, GPS, artificial intelligence, and Internet connectivity into one sleek device. Autonomous driving is another good example. In a time of rapidly converging technologies, reliability engineering must take into account the potential for cyber threats, the need for cyber trust, the importance of cyber security, and the criticality of cyber resilience. In this way, reliability engineers can ensure the confidentiality, integrity, and availability of computer systems and networks in the face of evolving threats and changing technologies. In this article, we introduce the challenges and current progress of reliability engineering in emerging technologies, including practices and applications of cyber trust and security, AI-empowered autonomous driving systems, modern mobile networks, blockchains and distributed ledger technologies, prognostic and health management, integrated circuit and hardware, and enterprise cybersecurity and threat hunting.

Summary In this paper, we report on applying combinatorial testing to Internet of Things (IoT) home automation hub systems. We detail how to create a dedicated input parameter model of an IoT home automation hub system for use with combinatorial test case generation strategies. Further, we developed an automated test execution framework and two test oracles for evaluation purposes. We applied and evaluated our proposed methodological approach to a real‐world IoT system and analysed the obtained results of various combinatorial test sets with different properties generated based on the derived input model. Additionally, we compare these results to a random testing approach. Our empirical testing evaluations revealed multiple errors in the tested devices and also showed that all considered approaches performed nearly equally well.

System primitives allow formalisms, reasoning, simulations, and reliability and security risk-tradeoffs to be formulated and argued. In this work, six core primitives belonging to most distributed systems are presented. These primitives apply well to systems with large amounts of data, scalability concerns, heterogeneity concerns, temporal concerns, and elements of unknown pedigree with possible nefarious intent. These primitives form the basic building blocks for a Network of 'Things' (NoT), including the Internet of Things (IoT). This keynote offers an underlying and foundational science to IoT. To my knowledge, the ideas and the manner in which the science underlying IoT is presented here is unique. Further, this talk reflects my personal viewpoints and not those of NIST.

Reliability engineering have been longstanding, well understood, well defined, and practiced by a small community when one looks at the entirety of engineering. Of more recent, the hardware, component, and system principles of reliability and reliability engineering have gradually moved into the software arena. Today, terms like software reliability, software reliability engineering, software fault tolerance, and software resilience are common vernacular and main stream. And we are now able to achieve levels of ultra-reliability in software systems of uncompromising criticality, but unfortunately at extreme costs, and slowly.

We, as individuals, as well as governments, corporations, and institutions, form a networked society and we are increasingly dependent on that network. The very fabric of our everyday life and business utilizes this networked connectivity, particularly critical infrastructures such as the electric power grid, oil and gas pipeline and distribution systems, telecommunications, transportation, and water treatment systems. All of these applications, and many more, are relied upon daily and need to be "trustworthy." By trustworthiness, we mean many qualities which include, but are not limited to, availability, assurance of information delivered, reliability, security, survivability, recoverability, confidentiality, integrity and other "ilities." Basically, when you pick up the phone, you expect to hear a dial tone; when you turn on a light switch, you expect the light to turn on; and when you send an e-mail, you expect it to reach its intended recipient with message intact and without others reading it en route. Since all of our critical infrastructures are themselves network-centric systems, this trustworthiness is both difficult to achieve and critical to maintain. One panel session cannot begin to capture all of the aspects of trustworthy computing; however, this panel is composed of experts who will explore many of the fundamental aspects of this increasingly important topic, particularly focused on critical research needs.

The ubiquity of information systems has made correct and reliable operation of critical systems indispensable. The trustworthiness of digital systems is increasingly dependent on the trustworthiness of the software. While hardware trustworthiness is by no means a solved problem, system-wide problems are increasingly blamed on poorly tested, defective software. System trustworthiness is therefore a combination of several key software attributes: reliability, safety, security, availability, performance, fault-tolerance, and privacy. Some of these attributes can be directly measured, some cannot. For example performance and availability can be numerically measured; safety and security cannot. Further, several of these attributes may conflict, such as security and performance. Therefore to demonstrate that the software of a system can be trusted, it requires a combination of qualitative arguments concerning the level achieved for some attributes in combination with the numerical (quantitative) scores measured for others. In order to understand the trustworthiness and security of a software system, we first need to understand its reliability and fault tolerance

Formal electronic conferences (FEC) refers to online meetings for a geographically distributed group of people that are regulated by a rigorous set of rules. FEC technologies enable organizations to replace face-to-face business meetings with trustworthy virtual online meetings. In This work we present a Robert's rules of order (RRO)-compatible, motion-driven discussion-thread-centered meta-model, which is capable of uniformly modeling formal electronic conference activities. A tailored computerized mechanism, the collaboration description language (CODL) and its runtime environment, is also developed to formalize the model. The CODL virtual machine adds a layer of encapsulation that decouples FEC applications from underlying platforms: therefore, the development of FEC applications will become more reliable, efficient, and secure. Our preliminary experience with this meta-model is also reported.

When software is built from components, nonfunctional properties such as security, reliability, fault-tolerance, performance, availability, safety, etc. are not necessarily composed. The problem stems from our inability to know a priori, for example, that the security of a system composed of two components can be determined from knowledge about the security of each. This is because the security of the composite is based on more than just the security of the individual components. There are numerous reasons for this. The article considers only the factors of component performance and calendar time. It is concluded that no properties are easy to compose and some are much harder than others.

There are many definitions as to what COTS software is. The author uses a very simple definition: COTS software is software functionality obtained from a third party and that is used on an "as is" basis. Examples here include operating system utilities, class libraries, databases, word processing applications, and browser plug-ins. Software assurance problems become magnified for systems that are heavily comprised from COTS software because most of the prevalent quality improvement techniques cannot be applied by the user of the acquired software. These techniques are of course available to the software publisher, but whether the publisher opts to use them cannot be independently verified by the user. That is alarming and could be highly problematic for organizations that opt to buy versus build software systems. In summary, the lack of software quality and publisher responsibility are the greatest concerns that the author sees facing the software industry. The upside, however, is that certification and IV&V can bring assurance back to generic software.

News from the Information Technology (IT) community is mixed. Project backlogs, high employment and personnel shortages are claimed on one side; project cancellations, unproductive use of personnel time and continued poor quality software are claimed on the other. Which side is component-based software on? The drive to use components such as COTS components comes from claims of instant productivity gains, shorter time to market, lower cost, management mandates and a ‘parts’ philosophy. On the other side are difficulties in composing systems from unreliable components, the ‘ility’ factors, the traditional white-box software processes, short-changed testing, software component support limits and software malleability. If components such as COTS are used a lot, then finger-pointing grows more common as we move toward black-box component integration. Some of the defensive options include component certification, strengthened testing, and adding wrappers and middleware. All add to maintenance cost. If we use components such as COTS, our ability to maintain the resulting composite systems is reduced to the level of ‘unmaintainable’. This leads us to think in terms of disposable information systems. Overall, I observe as we move toward component-based software, quality will be even more important for maintainability. Copyright © 1999 John Wiley & Sons, Ltd.

Software certification processes have become so intertwined with development processes that true product certification, which should demonstrate that the software will behave appropriately, is almost never employed. This deficiency calls for a new generation of certification methodologies. The paper presents a methodology for measuring whether software will do what we want. If the software will, it deserves to be labeled as being of high quality (or high assurance). The methodology we provide is applicable to all types of software, covering the spectrum from safety critical control software to games.

The author has investigated a fault injection-based technique for undermining the ability of software components to produce undesirable outputs into the state of the system. Undesirable outputs are any class of outputs that a component must not release into the state of the system given its current environment. Software components are said to be "failure-tolerant" if they release desirable outputs regardless of the programmer faults, potential malicious input data directed against the component, and other non-malicious but corrupted input data. The technology assesses the failure tolerance of software components after simulated program state corruptions are injected into the components as they execute. Based on the types of outputs that result from fault injection, the technique knows where "recovery assertions" (which act somewhat like antibodies do in an organism) should be injected into software components to ensure desirable system outputs; the second part of the approach then suggests what the assertions should be.

Multi-version programming is employed in fault-tolerant computer systems in order to provide protection against common-mode failure in software. Multi-version programming involves building diverse software implementations of critical functions. The premise of building diverse versions is that the likelihood of a programming error in one version causing a failure in an identical manner as an error in another version is reduced. Skeptics of multi-version programming have correctly pointed out that common-mode failures between redundant diverse versions can reduce the return on investment in creating diverse versions. To date, other than using historical data from other projects, there has been no way to estimate the potential for a given multi-version programming system to suffer a common-mode failure. This paper presents an algorithm and software analysis prototype to reduce the uncertainty of whether software flaws in diverse versions can result in common-mode failure. The analysis uses software fault-injection techniques to subject one or more versions to anomalous behavior. From this, we can predict how the software will behave if real faults exist in the multiple versions.

With demands for increased software reuse, a market for buying and selling generic software functionality is emerging. This generic software functionality is usually referred to as Commercial-off-the-shelf (COTS) software. Generic functionality has greater commercial viability than specialized functionality, and can be used in diverse application domains, including systems with high-assurance requirements. A problem arises, however, in that the quality of COTS software cannot be guaranteed past what the developing organization certifies, which is usually little. This puts the onus on the buyer to determine whether the quality of the COTS software is sufficient for their application. This panel will discuss various means for qualifying the reliability of COTS and third-party software. This panel may also discuss some of the legal and certification issues that are associated with trying to standardize on a fixed set of assessment techniques to reduce this concern. 1071.9458197 $10.00

We model the relationship between static software product measures and a dynamic quality measure, testability. To our knowledge, this is the first time a dynamic quality measure has been modeled using static software product measures. We first give an overview of testability analysis and discriminant modeling. Using static software product measures collected from a real time avionics software system, we develop two discriminant models and classify the component program modules as having low or high testability. The independent variables are principal components derived from the observed software product measures. One model is used to evaluate the quality of fit and one is used to assess classification performance. We show that for this study, the quality of fit and classification performance of the discriminant modeling methodology are excellent and yield a potentially useful insight into the relationship between static software measures and testability.

The paper turns the concept of input distributions on its head to exploit inverse input distributions. Although such distributions are not always true mathematical inverses, they do capture an intuitive property: inputs that have high frequencies in the original distribution will have low frequencies in the inverse distribution, and vice versa. We can use the inverse distribution in several different quality checks during development. We provide a fault based (fault injection) method to determine minimum time to failure and mean time to failure for software systems under normal operational and non normal operational conditions (meaning rare but legal events). In our calculations, we consider how various programmer faults, design errors, and incoming hardware failures are expected to impact the observability of the software system.

Assertions that are placed at each statement in a program can automatically monitor the internal computations of a program execution. However, the advantages of universal assertions come at a cost. A program with such extensive internal instrumentation will be slower than the same program without the instrumentation. Some of the assertions may be redundant. The task of instrumenting the code with correct assertions at each location is burdensome, and there is no guarantee that the assertions themselves will be correct. We advocate a middle ground between no assertions at all (the most common practice) and the theoretical ideal of assertions at every location. Our compromise is to place assertions only at locations where traditional testing is unlikely to uncover software faults. One type of testability measurement, sensitivity analysis, identifies locations where testing is unlikely to be effective.>

The authors present a software engineering technique called sensitivity analysis that aims at producing software that is less likely to hide faults. When sensitivity analysis indicates that faults are not likely to hide from testing, they can more comfortably rely on the results of testing. Sensitivity analysis has been automated, and preliminary results from the automated analysis suggest that the technique will have significant advantages for the development, validation, and regulation of medical devices.>

Most of the effort that goes into improving the quality of software paradoxically does not lead to quantitative, measurable quality. Software developers and quality-assurance organizations spend a great deal of effort preventing, detecting, and removing “defects”—parts of software responsible for operational failure. But software quality can be measured only by statistical parameters like hazard rate and mean time to failure, measures whose connection with defects and with the development process is little understood.

Software testability is the tendency of code to reveal existing faults during random testing. This paper proposes to take software testability predictions into account throughout the development process. These predictions can be made from formal specifications, design documents, and the code itself. The insight provided by software testability is valuable during design, coding, testing and quality assurance. The authors believe that software testability analysis can play a crucial role in quantifying the likelihood that faults are not hiding, after the testing does not result in any failures for the current version.>

Abstract ‘Propagation, infection, and execution analysis’ (termed PIE) is used for predicting where faults can more easily hide in software. To make such predictions, programs are dynamically executed with test cases, and information concerning the test cases is collected into a histogram, each bin of which represents a single test case. The score in a bin predicts the likelihood that the test case will reveal a fault through the production of a failure (if a fault exists in the set of program locations that the test case executes). Preliminary experiments using program mutations suggest that the histogram technique presented in this paper can rank test cases according to their fault revealing ability.

Formulas for estimating the probability of failure when testing reveals no errors are introduced. These formulas incorporate random testing results, information about the input distribution; and prior assumptions about the probability of failure of the software. The formulas are not restricted to equally likely input distributions, and the probability of failure estimate can be adjusted when assumptions about the input distribution change. The formulas are based on a discrete sample space statistical model of software and include Bayesian prior assumptions. Reusable software and software in life-critical applications are particularly appropriate candidates for this type of analysis.>

A dynamic technique called PIE (propagation, infection, and execution) is presented for statistically estimating three program characteristics that affect a program's computational behavior: (1) the probability that a particular section of a program is executed, (2) the probability that the particular section affects the data state, and (3) the probability that a data state produced by that section has an effect on program output. These three characteristics can be used to predict whether faults are likely to be uncovered by software testing.>