The ideal principle of software testing is that test results ought to be deterministic: a test failure indicates the presence of a software bug, while a test success suggests the absence of a bug. Nevertheless, flaky tests break the principle. Flaky tests yield inconsistent results when executed repeatedly under the same conditions. The most straightforward approach runs the tests multiple times to predict flaky tests whereas it is highly time-consuming. Many researchers have proposed efficient approaches to reduce the cost, e.g., recent approaches leverage machine learning techniques for the prediction of flaky tests. However, traditional machine learning primarily focuses on predicting specific instances, which is not conducive to identify flaky tests across an entire project. Therefore, we propose Flakyrank, a ranking framework based on augmented learning to rank to predict flaky tests. The insight is that learning to rank, as compared with traditional machine learning, not only concentrates on individual samples but also optimizes the overall ranking. Since flaky tests constitute a small proportion of the dataset (i.e., approximately 3.6% of the total tests), we utilize generative adversarial networks to generate some synthetic flaky tests to augment the dataset. Based on the augmented dataset, FLAKYRANK treats predicting flaky tests as an information retrieval task, where newly detected flaky tests and test cases serve as queries and documents, respectively. For each newly detected flaky test (i.e., query), FLAKYRANK combines multiple relevant features into a learning to rank model to predict flaky tests candidate tests. We conduct large-scale experiments on different learning to rank models, and the results show that FLAKYRANK with the LambdaMART algorithm yields the best performance. In addition, the experimental results on 23 benchmark projects show that FLAKYRANK outperforms the state-of-the-art predictors.

A test suite is indispensable for fault localization by providing useful execution information of its test cases for locating suspicious statements of being faulty. There exists a type of test cases known as coincidental correctness (CC) test cases, which executes the faulty statement whereas produces the anticipated output. The existing studies have shown CC test cases harmfully impact fault localization effectiveness. Therefore, it is crucial to detect CC test cases to mitigate the adverse impact of CC test cases on fault localization.To address this issue, we propose ContraCC: a CC test cases detection method using contrastive learning. The insight of ContraCC is that the internal structural information of source test case execution data should be beneficial for CC detection whereas there is a lack of suitable representation methods. Inspired by the insight, ContraCC uses contrastive learning to learn new differentiated representations as test case vectors, which differentiate between similar and dissimilar pairs of test cases by maximizing their similarity within the same class and minimizing it between different classes. Based on the contrastive learning representations (i.e., test case vectors), ContraCC adopts multi-layer perceptron for binary classification to detect CC in downstream tasks. To evaluate the effectiveness of ContraCC, we conduct large-scale experiments on widely-used benchmarks by comparing ContraCC with five state-of-the-art CC test cases detection methods and applying ContraCC for fault localization. The experimental results show that ContraCC outperforms four state-of-the-art methods (e.g., from 10% to 84% improvement in Top-N on the best-performing baseline NeuralCCD) and significantly improves fault localization effectiveness (e.g., 24% improvement on the best-performing baseline Dstar).