Abstract Previous studies have demonstrated the usefulness of employing automated static analysis tools (ASAT) and techniques to detect security bugs in software systems. However, these studies are usually focused on analyzing the effectiveness of the tools using open‐source tools based on C/C++ source code. The choice for making an appropriate decision on the most suitable tool for bug detection in Java code software remains a relatively unexplored domain. To address this deficiency, this study empirically evaluates eight widely used ASATs, namely, Findbug, PMD, YASCA, LAPSE+, JLint, Bandera, ESC/Java, and Java Pathfinder using the Juliet Test Suite (Test Suite v1.2). Additionally, we assessed the performance of the detection capabilities for the aforementioned bug detection tools using robust performance measures such as precision, recall, Youden index, and the OWASP web benchmark evaluation (WBE). The experimental results show that the tools obtain precision values ranging from 83% to 90.7% based on the studied datasets. Specifically, the Java Pathfinder achieves the best precision score of 90.7%, followed by YASCA and Bandera with a precision score of 88.7% and 83%, respectively. Similarly, Bandera, ESC/Java, and Java Pathfinder obtain a Youden index of 0.8, which indicates the effectiveness of the tools in detecting security bugs in Java source code.

This extended abstract presents an approach to enhance the Fixed-Sized-Candidate-Set Adaptive Random Testing (FSCS-ART) sampling strategy. SWFC-ART, the proposed approach, stores the previously-executed, non-failure-causing test cases into a Hierarchical Navigable Small World Graph (HNSWG) data structure and uses an efficient and consistent Nearest Neighbor Search (NNS) mechanism, especially for high-dimensional input domains. Our experiments show that SWFC-ART reduces the computational overhead of FSCS-ART from quadratic to log-linear order while retaining the failure-detection effectiveness of FSCS-ART.

Abstract Context The use of a subset of recently completed and exemplary data, namely, Bellwether moving window (BMW) has proven successful to result in improved accuracy in software effort estimation (SEE). These outcomes were achieved based on the theory that estimation outcome of a future event depends on previous events. Thus, the existence of a BMW yield improved prediction accuracy for new project estimation. However, the conclusion instability problem across learners still threatens the reliability of SEE for new projects. Such instability concerns are attributed to the data subset considered for the training and validation needs of learners. Objective To investigate whether the use of BMWs together with an effort classification scheme can minimize the conclusion instability problem across learners. Method We apply a Bellwether method comprising of three operators, namely, SORT+CLUSTER, GENERATE_TPM, and APPLY to sample the BMW from a pool of chronological projects from the Maxwell and International Software Benchmarking Standards Group (ISBSG) datasets. The sampled BMW is benchmarked against the entire collection of preprocessed projects, namely, growing portfolio to evaluate prediction and classification accuracy across a set of learners–ElasticNet regression, deep neural networks, and automatically transformed linear model. Results (1) BMW exists in the studied projects and (2) training the learners with a BMW of average window size 28.5%–75.5% of the growing portfolio (not older than 3 years) relatively minimizes the conclusion instability of prediction results. Conclusion When BMWs are available, we recommend their use for estimating the effort for a new project to minimize the conclusion instability problem.

Summary Web vulnerability scanners (WVSs) are tools that can detect security vulnerabilities in web services. Although both commercial and open‐source WVSs exist, their vulnerability detection capability and performance vary. In this article, we report on a comparative study to determine the vulnerability detection capabilities of eight WVSs (both open and commercial) using two vulnerable web applications: WebGoat and Damn vulnerable web application. The eight WVSs studied were: Acunetix; HP WebInspect; IBM AppScan; OWASP ZAP; Skipfish; Arachni; Vega; and Iron WASP. The performance was evaluated using multiple evaluation metrics: precision; recall; Youden index; OWASP web benchmark evaluation; and the web application security scanner evaluation criteria. The experimental results show that, while the commercial scanners are effective in detecting security vulnerabilities, some open‐source scanners (such as ZAP and Skipfish) can also be effective. In summary, this study recommends improving the vulnerability detection capabilities of both the open‐source and commercial scanners to enhance code coverage and the detection rate, and to reduce the number of false‐positives.

Vulnerability Prediction Models (VPMs) aims to identify vulnerable and non-vulnerable components in large software systems. Consequently, VPMs presents three major drawbacks (i) finding an effective method to identify a representative set of features from which to construct an effective model. (ii) the way the features are utilized in the machine learning setup (iii) making an implicit assumption that parameter optimization would not change the outcome of VPMs. To address these limitations, we investigate the significant effect of the Bellwether analysis on VPMs. Specifically, we first develop a Bellwether algorithm to identify and select an exemplary subset of data to be considered as the Bellwether to yield improved prediction accuracy against the growing portfolio benchmark. Next, we build a machine learning approach with different parameter settings to show the improvement of performance of VPMs. The prediction results of the suggested models were assessed in terms of precision, recall, F-measure, and other statistical measures. The preliminary result shows the Bellwether approach outperforms the benchmark technique across the applications studied with F-measure values ranging from 51.1%-98.5%.

Software vulnerability analysis is one of the critical issues in the software industry, and vulnerability classification plays a major role in this analysis. A typical vulnerability classification model usually involves a stage of term selection, in which the relevant terms are identified via feature selection. It also involves a stage of term weighting, in which document weights for the selected terms are computed, and a stage for classifier learning. Generally, the term frequency-inverse document frequency (TF-IDF) is the most widely used term-weighting method. However, empirical evidence shows that the TF-IDF is plagued with issues pertaining to its effectiveness. This paper introduces a new approach for vulnerability classification, which is based on term frequency and inverse gravity moment (TF-IGM). The proposed method is validated by empirical experiments using three machine learning algorithms on ten publicly available vulnerability datasets. The result shows that TF-IGM outperforms the benchmark method across the applications studied.

Mirror Adaptive random testing (MART) is an overhead reduction strategy for adaptive random testing methods. Theoretically speaking, MART's advantage over ordinary ARTs is determined by the mirroring scheme selected. Incidentally, an inherent problem with MART relates to the difficulty in the choice of a scheme for any testing task. This is because a higher scheme (larger mirror domains) does not necessarily guarantee efficient utilization of testing resources due to lack of diversity of mirror generated test cases. The culprit has been identified as the mapping functions used as substitutes to complex ART methods. In this paper, we present a new method for generating diversified mirror test cases by randomly displacing the mirror partitions upon which the mapping functions of MART operates. The result of simulations and experiments conducted shows remarkable improvement over MART's effectiveness and efficiency across MART schemes, especially where program failures are unrelated to one or more input parameters.

Finding an effective method for testing object-oriented software (OOS) has proven elusive in the software community due to the rapid development of object-oriented programming (OOP) technology. Although significant progress has been made by previous studies, challenges still exist in relation to the object distance measurement of OOS using Adaptive Random Testing (ART). This is partly due to the unique features of OOS such as encapsulation, inheritance and polymorphism. In a previous work, we proposed a new similarity metric called the Object and Method Invocation Sequence Similarity (OMISS) metric to facilitate multi-class level testing using ART. In this paper, we broaden the set of models in the metric (OMISS) by considering the method parameter and adding the weight in the metric to develop a new distance metric to improve unit testing of OOS. We used the new distance metric to calculate the distance between the set of objects and the distance between the method sequences of the test cases. Additionally, we integrate the new metric in unit testing with ART and applied it to six open source subject programs. The experimental result shows that the proposed method with method parameter considered in this study is better than previous methods without the method parameter in the case of the single method. Our finding further shows that the proposed unit testing approach is a promising direction for assisting software engineers who seek to improve the failure-detection effectiveness of OOS testing.

An effective method for finding the relevant number (window size) and the elapsed time (window age) of recently completed projects has proven elusive in software effort estimation.Although these two parameters significantly affect the prediction accuracy, there is no effective method to stratify and sample chronological projects to improve prediction performance of software effort estimation models.Exemplary projects (Bellwether) representing the training set have been empirically validated to improve the prediction accuracy in the domain of software defect prediction.However, the concept of Bellwether and its effect have not been empirically proven in software effort estimation as a method of selecting exemplary/relevant projects with defined window size and age.In view of this, we introduce a novel method for selecting relevant and recently completed projects referred to as Bellwether moving window for improving the software effort prediction accuracy.We first sort and cluster a pool of N projects and apply statistical stratification based on Markov chain modeling to select the Bellwether moving window.We evaluate the proposed approach using the baseline Automatically Transformed Linear Model on the ISBSG dataset.Results show that (1) Bellwether effect exist in software effort estimation dataset, (2) the Bellwether moving window with a window size of 82 to 84 projects and window age of 1.5 to 2 years resulted in an improved prediction accuracy than the traditional approach.