The emergence of Autonomous Vehicles (AVs) has spurred research into testing the resilience of their perception systems, i.e., ensuring that they are not susceptible to critical misjudgements. It is important that these systems are tested not only with respect to other vehicles on the road, but also with respect to objects placed on the roadside. Trash bins, billboards, and greenery are examples of such objects, typically positioned according to guidelines developed for the human visual system, which may not align perfectly with the needs of AVs. Existing tests, however, usually focus on adversarial objects with conspicuous shapes or patches, which are ultimately unrealistic due to their unnatural appearance and reliance on white-box knowledge. In this work, we introduce a black-box attack on AV perception systems that creates realistic adversarial scenarios (i.e., satisfying road design guidelines) by manipulating the positions of common roadside objects and without resorting to "unnatural" adversarial patches. In particular, we propose TrashFuzz, a fuzzing algorithm that finds scenarios in which the placement of these objects leads to substantial AV misperceptions -- such as mistaking a traffic light's colour -- with the overall goal of causing traffic-law violations. To ensure realism, these scenarios must satisfy several rules encoding regulatory guidelines governing the placement of objects on public streets. We implemented and evaluated these attacks on the Apollo autonomous driving system, finding that TrashFuzz induced violations of 15 out of 24 traffic laws.

Garbage Collection (GC) in the Java Virtual Machine (JVM) serves as an automatic memory management mechanism, efficiently reclaiming unused memory space in different production scenarios. To optimize JVM performance, developers typically fine-tune the garbage collector by identifying an optimal set of GC configurations for specific scenarios. Despite the sophisticated design of garbage collectors, they still have the potential for bugs in different settings, and these bugs can result in more severe consequences. Hence, comprehensive testing of these garbage collectors is imperative before their release. Code coverage criteria are typically employed to assess the comprehensiveness of a test suite. However, traditional code coverage metrics, such as branch coverage, are hardly applicable for GC testing due to their inherent concurrency. Additionally, existing JVM testing techniques do not adequately consider the characteristics of garbage collectors, making it difficult to test these garbage collectors sufficiently. In this article, we make the first effort to design coverage criteria against garbage collectors based on the events of GC called Event-Coverage . Its key insight is to measure the diversity of GC executions for testing purposes by assessing the range of GC events these executions cover. Furthermore, we design a new testing method for maximizing Event-Coverage called GCFuzz . GCFuzz conducts an exhaustive investigation of the memory state space of GC and thoroughly explores the memory state under various GC configurations. To enhance GCFuzz’s efficiency in achieving higher Event-Coverage, we have further designed a coverage-driven strategy for preserving candidate seed programs and selecting GC configurations. Extensive evaluations demonstrate a positive correlation between Event-Coverage and the bug-revealing efficiency. Moreover, GCFuzz outperforms state-of-the-art techniques in detecting unique GC-related inconsistencies and achieving higher Event-Coverage. Remarkably, GCFuzz has identified 20 previously undetected GC bugs, with 15 of them already confirmed or fixed by developers.

Large language models (LLMs) typically provide multilingual support and demonstrate remarkable capabilities in solving tasks described in different languages. However, LLMs can exhibit linguistic discrimination due to the uneven distribution of training data across languages. That is, LLMs struggle to maintain consistency when handling the same task in different languages, compromising both safety equity and knowledge equity. In this paper, we first systematically evaluate the linguistic discrimination of LLMs from two aspects: safety and quality, using a form of metamorphic testing. The metamorphic relationship we examine is that LLMs are expected to deliver outputs with similar semantics when prompted with inputs that have the same meaning. We conduct this evaluation with two datasets based on four representative LLMs. The results show that LLMs exhibit stronger human alignment capabilities with queries in English, French, Russian, and Spanish compared to queries in Bengali, Georgian, Nepali and Maithili. Moreover, for queries in English, Danish, Czech and Slovenian, LLMs tend to produce responses with a higher quality compared to the other languages. Upon these findings, we propose LDFighter, a similarity-based voting method, to mitigate the linguistic discrimination in LLMs. We comprehensively evaluate LDFighter against a spectrum of queries including benign, harmful, and adversarial prompts. The results show that LDFighter significantly reduces jailbreak success rates and improves response quality. All code, data, and the technical appendix are publicly available at: \url{https://github.com/dgl-prc/ldfighter}.

Abstract Smart contracts are distributed self‐enforcing programs which execute on top of blockchain networks. They have the potential to revolutionize many industries and have already been adopted for applications such as distributed finance and crowdfunding. Because smart contracts are immutable once they are deployed, it is important to identify and eliminate code vulnerabilities in smart contracts systematically. In this work, we propose sFuzz2.0, a storage‐access‐pattern guided adaptive fuzzer based on sFuzz. sFuzz2.0 is motivated by the fact that certain vulnerabilities only manifest in the presence of certain function call sequences (as well as particular arguments). Given that there are exponentially many function call sequences, sFuzz randomly generates sequences without guidance. As a result, the probability of discovering those vulnerabilities is negligible. sFuzz2.0 tackles the problem with two approaches, that is, by generating function call sequences that trigger different storage‐access patterns passively (i.e., by prioritizing seeds which cover new patterns) or actively (i.e., by actively seeking out different patterns). The experiment results suggest that the passive strategy outperforms sFuzz by achieving better code coverage (i.e., 37.53%) and discovering more vulnerabilities (i.e., 20.49%).

MLIR (Multi-Level Intermediate Representation) compiler infrastructure has gained popularity in recent years to support the construction of many compilers. Instead of designing a new IR with a single abstraction for each domain, MLIR compiler infrastructure provides systematic passes to support a wide range of functionalities for benefiting multiple domains together and introduces dialects to support different levels of abstraction in MLIR. Due to its fundamental role in compiler community, ensuring its quality is very critical. In this work, we propose MLIRSmith, the first fuzzing technique for MLIR compiler infrastructure. MLIRSmith employs a two-phase strategy to generate valid and diverse MLIR programs, which first constructs diverse program templates guided by extended MLIR syntax rules and then generates valid MLIR programs through template instantiation guided by our designed context-sensitive grammar. After applying MLIRSmith to the latest revision of MLIR compiler infrastructure, we detected 53 previously unknown bugs, among which 49/38 have been confirmed/fixed by developers. We also transform the high-level programs generated by NNSmith (a high-level program generator for deep learning compilers) to MLIR programs for indirectly fuzzing MLIR compiler infrastructure. During the same testing time, MLIRSmith largely outperforms such an indirect technique by detecting 328.57% more bugs and covering 194.67%/225.87% more lines/branches in MLIR compiler infrastructure.

Question answering (QA) software uses information retrieval and natural language processing techniques to automatically answer questions posed by humans in a natural language. Like other AI-based software, QA software may contain bugs. To automatically test QA software without human labeling, previous work extracts facts from question answer pairs and generates new questions to detect QA software bugs. Nevertheless, the generated questions could be ambiguous, confusing, or with chaotic syntax, which are unanswerable for QA software. As a result, a relatively large proportion of the reported bugs are false positives. In this work, we proposed QAQA, a sentence-level mutation based metamorphic testing technique for QA software. To eliminate false positives and achieve precise automatic testing, QAQA leverages five Metamorphic Relations (MRs) as well as semantics-guided search and enhanced test oracles. Our evaluation on three QA datasets demonstrates that QAQA outperforms the state-of-the-art in both quantity (8,133 vs. 6,601 bugs) and quality (97.67% vs. 49% true positive rate) of the reported bugs. Moreover, the test inputs generated by QAQA successfully reduce MR violation rate from 44.29% to 20.51% when being adopted in fine-tuning the QA software under test.

Concurrency bugs are notoriously hard to identify and fix. A systematic way of avoiding concurrency bugs is to design and implement a locking policy that consistently guards all shared variables. Concurrency bugs thus can be viewed as the result of an illy-designed or poorly implemented locking policy. The trouble is that the locking policy is often not documented, which makes debugging concurrency bugs clueless. We argue that it is too late to debug concurrency bugs after programming is done and we instead detect and fix them while they are being implemented. In this work, we propose an approach named IFIX which flags potential concurrency bugs and recommends fixes while the bugs are introduced. The key idea is to automatically conjecture what the intended locking policy is based on static analysis and recommend fixes accordingly. The recommended fixes are present to the programmer promptly and the user feedback (i.e., whether the certain recommendation is selected) is used to refine the conjectured locking policy and consequently future fixes. IFIX is evaluated on 43 concurrent programs, and through a user study with 30 programmers. The experiment results and user feedback show that IFIX is efficient, accurate and user-friendly.