Context: In metamorphic testing (MT), a set of metamorphic relations (MRs) are identified to verify whether or not a trained deep neural network (DNN) can produce consistent performance when specific transformations are applied to its input. Most DNNs trained with existing methods often perform poorly with respect to MRs, thereby indicating that these DNNs are not robust. Objective: To improve DNN’s performance in the context of MT, a set of defined MRs is used to generate training inputs to retrain a DNN model. Our main objective is to develop a method to balance a DNN’s accuracy and robustness with less time consumption and having the capability to cater to multiple MRs. Methods: In this paper, we introduce our regularization-based method (known as MT-Boost), which uses reinforcement learning to search for the best way of using MRs to generate inputs and express them as loss function regularizers. When developing MT-Boost, we transform the robustness-improving problem into a reinforcement-learning agent’s training problem. Results: MT-Boost is evaluated on eight DNN models with four popular datasets. MT-Boost achieves the largest robustness improvement for each model and maintains relatively high accuracy performance when compared with seven other baseline methods. Our sensitivity analysis also shows the high stability performance of MT-Boost across four reinforcement-learning algorithms and other hyperparameters. Conclusion: Experimental results show that MT-Boost is effective and efficient for improving DNN’s robustness.

Although many of today’s object detectors (ODs) are fairly powerful and advanced, most of them still suffer from high detection failure rates. To address this issue, we have developed an innovative, multiple-object detection validation method using a complex-network-community-based relevance metric. This metric aims to measure the relevance of multiple objects in the same OD output, based on our observation that a faulty OD output generally includes objects that are irrelevant or unrelated to each other. To verify the effectiveness of our method, we formulated four research questions, and performed an experiment with statistical analyses to address these questions. Our experiment provides strong support that our method (particularly the relevance metric) is highly effective at helping human testers in identifying faulty OD outputs.

Software defect prediction (SDP) techniques play a crucial role in identifying defective code regions and improving testing efficiency. Over recent decades, a plethora of SDP approaches has emerged, with machine learning (ML) models being the most widely employed. Despite their superior predictive performance, their black-box nature and uncertainties make it challenging for developers to trust their predictions. To address this issue, we propose a novel trustworthiness score, the adjusted trust score (ATS), which helps determine when to rely on classifier predictions. Furthermore, we employ ATS to develop a reject option for SDP models. Comprehensive experiments on 32 benchmark datasets and six prevalent ML classifiers reveal that high (low) ATS values successfully yield high precision in identifying correct (or incorrect) predictions. ATS also demonstrates superiority over its counterparts, as evidenced by the Wilcoxon signed-rank test. Furthermore, a comparative analysis of prediction performance, with and without a reject option, confirms the feasibility of designing a reject option for SDP models utilizing ATS. Our work highlights that ATS can assist developers in better comprehending the strengths and weaknesses of SDP models. Therefore, it is an essential component for guaranteeing trust from developers and deserves further investigation.

Metamorphic Relations (MRs) play a key role in determining the fault detection capability of Metamorphic Testing (MT). As human judgement is required for MR identification, systematic MR generation has long been an important research area in MT. Additionally, due to the extra program executions required for follow-up test cases, some concerns have been raised about MT cost-effectiveness. Consequently, the reduction in testing costs associated with MT has become another important issue to be addressed. MR composition can address both of these problems. This technique can automatically generate new MRs by composing existing ones, thereby reducing the number of follow-up test cases. Despite this advantage, previous studies on MR composition have empirically shown that some composite MRs have lower fault detection capability than their corresponding component MRs. To investigate this issue, we performed theoretical and empirical analyses to identify what characteristics component MRs should possess so that their corresponding composite MR has at least the same fault detection capability as the component MRs do. We have also derived a convenient, but effective guideline so that the fault detection capability of MT will most likely not be reduced after composition.

Mandelbug-caused software failures are significant threats to system availability, especially in the context of mission-critical and safety-critical systems. However, there is still no systematic method for keeping the software free from Mandelbugs before release. To guarantee the availability of systems suffering from Mandelbugs, environmental-diversity-based fault tolerance techniques have been proposed to recover from the failures caused by them. In this article, we develop and study an analytic model to assess the availability of systems that utilize a sequence of environmental-diversity-based recovery methods. Improving over previous relevant studies, the availability formula we obtain in this article works for any number of recovery methods the system is equipped with; it is also independent on both the nature of those recovery methods and the order of their utilization. In addition, we consider the problem of how to arrange the set of available recovery methods to achieve the largest system availability. Based on the results of our analysis, we develop an open-source tool, called OPENS, which assists in the calculation of the optimal system availability. We validate the effectiveness of the proposed modeling approach in two ways, namely by comparing our results with those obtained for specific systems considered in relevant studies and by conducting numerical analyses for more general scenarios of its application.

The Internet world is moving toward a scenario where users and applications have very diverse service expectation, making the current best-effort model inadequate and limiting. To be able to design high-availability service systems, it is essential to consider not only the actual failure and recovery behavior of the service infrastructure, but also the behavioral aspects of its user and their subjective perceptions and reactions in the wake of failure events. In this paper, we propose to use Markov regenerative process (MRGP) models to study the availability of Internet-based services perceived by a Web user on two different online service scenarios: (1) single-user-single-host and (2) single-user-multiple-host. The MRGP models capture the interactions between the service facility and the user. We also detect its parameter bottlenecks by applying the formal sensitivity analysis technique. The trends of the users' perceived unavailability are analyzed with the changed different parameter values, and the necessity of the sophisticated MRGP modeling is evidenced by the comparisons with the corresponding continuous time Markov chain (CTMC) models, which show that the popular convenient CTMC models tend to overestimate user-perceived service unavailability. Finally, controlled experiments are carried out on a real Web service to demonstrate the proposed approach.

Software failures caused by data race bugs have always been major concerns in parallel and distributed systems, despite significant efforts spent in software testing. Due to their nondeterministic and hard-to-reproduce features, when evaluating systems' operational reliability, a rather long period of experimental execution time is expected to be spent on observing failures caused by data race conditions. To address this problem, in this paper, we make two contributions. First, this paper proposes stress testing with influencing factors, in which the system runs under certain workloads for a long time with controlled stress conditions to accelerate the occurrence of data race failures. Second, it explores and formulates mathematical relationship models between data races' statistical characteristics of time to failure (TTF) or mean TTF (MTTF) and the influencing factors. Such relationship models are used for TTF/MTTF extrapolation under different operational conditions and are essential to reduce systems' reliability evaluation time. The proposed method is empirically evaluated on six applications suffering from failures caused by real-world data race bugs. Through analysis of the experimental results, we obtain several important findings: First, the reduction in the manifestation time to data race failures achieved by controlling the influencing factors is statistically significant. Second, Power model is the best-fitting model of the relationship between the MTTF and the influencing factors. Third, Power Weibull distribution is the best-fitting probability distribution between the TTF and the influencing factors. Finally, the TTF/MTTF can be accurately estimated with the approach proposed in this paper.

Path planning algorithms play critical roles in the systems of robots and unmanned aerial vehicles (UAVs). However, it is always difficult to verify the correctness of the implementations for such algorithms because the "planning oracles", the expected planning results, are usually hard to be obtained for complicate planning tasks. To improve software reliability, in this paper, we present a testing technique for verifying the implementations of graph searching based path planning algorithms deployed on robots and UAVs. Our approach is based on the technique of Metamorphic Testing, which has been shown considerable effectiveness in alleviating the absence of Oracle problems. According to the characteristics of graph searching based path planning problem, we present a framework to systematically design metamorphic relations. Based on the framework, six categories of metamorphic relations are proposed. We conduct the empirical analysis on 21 implements of three different path planning algorithms applied in a released business software project. The experimental results show that our approach can effectively detect dormant faults.

Datarace is a common problem on shared-memory parallel computers, including multicores. Due to its dependence on the thread scheduling scheme of its execution environment, the time to a datarace failure is usually very long. How to accelerate the occurrence of a datarace failure and further estimate the mean time to failure (MTTF) is an important topic to be studied. In this paper, the influencing factors for failures triggered by datarace bugs are explored and their influences on the time to datarace failure including the relationship with the MTTF are empirically studied. Experiments are conducted on real datarace suffering programs to verify the factors and their influences. Empirical results show that the influencing factors do have influences on the time to datarace failure of the subjects. They can be used to accelerate the occurrence of datarace failures and accurately estimate the MTTF.

When combining several services into a composite service, it is non-trivial to determine, prior to service deployment, performance and reliability values of the composite service. Moreover, once the service is deployed, it is often the case that during operation it fails to meet its service-level agreement (SLA) and one needs to detect what has gone wrong (i.e., performance/reliability bottlenecks). To study these issues, we develop a Semi-Markov Process (SMP) formulation of composite services with failures and restarts. By explicitly including failure states into the SMP representation of a service, we can compute both its performance and reliability using a single SMP. We can also detect its performance and reliability bottlenecks by applying the formal sensitivity analysis technique. We demonstrate our approach by choosing a representative example that is validated using experiments on real Web services.