VLDB 2026 Research / reviewers in the wild / expert
Jieke Shi
dblp:246/7302
· DBLP profile ↗
28ranked-venue papers
6as first author
25since 2021 · last 2026
0000-0002-0799-5018ORCID · verified
Domains — the database's venue-derived domains; a paper can count in several
Software engineering, systems software and programming languages · 20 · 4 first-author · 20 since 2021Artificial intelligence and machine learning · 2 · 2 since 2021Security and privacy · 2 · 2 since 2021Systems, architecture and hardware · 1 · 1 since 2021Computer networks · 1Databases, data management, data science and information retrieval · 1 · 1 since 2021Graphics, computer vision, multimedia, augmented reality and games · 1 · 1 first-authorHuman-computer interaction and ubiquitous computing · 1 · 1 first-author
| Year | Publication | Venue | Position |
|---|---|---|---|
| 2026 | SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing ScenariosabstractJunkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang, Haoye Tian, Yikun Li, Zhenhao Li, Xin Zhou, Xing Hu, David Lo. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. Junkai Chen, Huihui Huang, Yunbo Lyu, Junwen An, Jieke Shi, Chengran Yang, Ting Zhang 0011, Haoye Tian, Zhenhao Li 0002, Xin Zhou 0014, Xing Hu 0008, David Lo 0001 |
ACL (1) | 5 |
| 2026 | SeCuRepair: Semantics-Aligned, Curriculum-Driven, and Reasoning-Enhanced Vulnerability Repair FrameworkabstractChengran Yang, Ting Zhang, Jinfeng Jiang, Xin Zhou, Haoye Tian, Mingzhe Du, Jieke Shi, Junkai Chen, Yikun Li, Eng Lieh Ouh, Lwin Khin Shar, David Lo. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2026. Chengran Yang, Ting Zhang 0011, Jinfeng Jiang, Xin Zhou 0014, Haoye Tian, Mingzhe Du, Jieke Shi, Junkai Chen, Eng Lieh Ouh, Lwin Khin Shar, David Lo 0001 |
ACL (1) | 7 |
| 2026 | What You Trust is Insecure: Demystifying How Developers (Mis)Use Trusted Execution Environments in Practice
Yuqing Niu, Jieke Shi, Ruidong Han, Ye Liu 0012, Chengyan Ma 0001, Yunbo Lyu, David Lo 0001 |
SANER | 2 |
| 2026 | Ecosystem of Large Language Models for CodeabstractThe extensive availability of publicly accessible source code and the advances in language models, coupled with increasing computational resources, have led to a remarkable rise of large language models for code (LLM4Code). These models do not exist in isolation but rather depend on and interact with each other, forming a complex ecosystem that is worth studying. It motivates us to introduce a pioneering analysis of the LLM4Code ecosystem . Utilizing Hugging Face [ ]—the premier hub for transformer-based models—as our primary source, we manually curate a list of datasets and models focused on software engineering tasks. We first identify key datasets, models, and users in the ecosystem and quantify their contributions and importance. We then examine each model’s documentation to trace its base model and understand the process for deriving new models. We categorize LLM4Code model reuse into nine categories, with the top three being fine-tuning , architecture sharing , and quantization . Additionally, we examine documentation and licensing practices, revealing that LLM4Code documentation is less detailed than that of general AI repositories on GitHub. The license usage pattern is also different from other software repositories, and we further analyze potential license incompatibility issues. To analyze the rapidly growing LLM4Code, we explore the potential of using LLMs to assist in constructing and analyzing the ecosystem. Advanced LLMs from OpenAI identify LLM4Code with 98% accuracy, infer base models with 87% accuracy, and predict reuse types with 89% accuracy. We employ LLMs to expand the ecosystem and find that conclusions from the manually curated dataset align with those from the automatically created one. Based on our findings, we discuss the implications and suggestions to facilitate the healthy growth of LLM4Code. Zhou Yang 0003, Jieke Shi, Premkumar T. Devanbu, David Lo 0001 |
ACM Trans. Softw. Eng. Methodol. | 2 |
| 2025 | "My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding AssistantsabstractThis paper aims to explore fundamental questions in the era when AI coding assistants like GitHub Copilot are widely adopted: what do developers truly value and criticize in AI coding assistants, and what does this reveal about their needs and expectations in real-world software development? Unlike previous studies that conduct observational research in controlled and simulated environments, we analyze extensive, first-hand user reviews of AI coding assistants, which capture developers’ authentic perspectives and experiences drawn directly from their actual day-to-day work contexts. We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace. Although they only account for 1.64% of all extensions, we observe a surge in these assistants: over 90% of them are released within the past two years. We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants. We manually annotate each review’s attitude when mentioning certain aspects of coding assistants, yielding nuanced insights into user satisfaction and dissatisfaction regarding specific features, concerns, and overall tool performance. Built on top of the findings-including how users demand not just intelligent suggestions but also context-aware, customizable, and resource-efficient interactions—we propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs. Yunbo Lyu, Zhou Yang 0003, Jieke Shi, Jianming Chang, Yue Liu 0011, David Lo 0001 |
ASE | 3 |
| 2025 | Finding Safety Violations of AI-Enabled Control Systems through the Lens of Synthesized Proxy ProgramsabstractGiven the increasing adoption of modern AI-enabled control systems, ensuring their safety and reliability has become a critical task in software testing. One prevalent approach to testing control systems is falsification, which aims to find an input signal that causes the control system to violate a formal safety specification using optimization algorithms. However, applying falsification to AI-enabled control systems poses two significant challenges: (1) it requires the system to execute numerous candidate test inputs, which can be time-consuming, particularly for systems with AI models that have many parameters, and (2) multiple safety requirements are typically defined as a conjunctive specification, which is difficult for existing falsification approaches to comprehensively cover. This paper introduces Synthify , a falsification framework tailored for AI-enabled control systems, i.e., control systems equipped with AI controllers. Our approach performs falsification in a two-phase process. At the start, Synthify synthesizes a program that implements one or a few linear controllers to serve as a proxy for the AI controller. This proxy program mimics the AI controller's functionality but is computationally more efficient. Then, Synthify employs the \(\epsilon\) -greedy strategy to sample a promising sub-specification from the conjunctive safety specification. It then uses a Simulated Annealing-based falsification algorithm to find violations of the sampled sub-specification for the control system. To evaluate Synthify , we compare it to PSY-TaLiRo , a state-of-the-art and industrial-strength falsification tool, on 8 publicly available control systems. On average, Synthify achieves a 83.5% higher success rate in falsification compared to PSY-TaLiRo with the same budget of falsification trials. Additionally, our method is 12.8 \(\times\) faster in finding a single safety violation than the baseline. The safety violations found by Synthify are also more diverse than those found by PSY-TaLiRo , covering 137.7% more sub-specifications. Jieke Shi, Zhou Yang 0003, Junda He, Dongsun Kim 0001, DongGyun Han, David Lo 0001 |
ACM Trans. Softw. Eng. Methodol. | 1 |
| 2025 | Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road AheadabstractLarge Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This article aims to redirect the focus of the research community toward the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the article presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The article then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the article aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering. Jieke Shi, Zhou Yang 0003, David Lo 0001 |
ACM Trans. Softw. Eng. Methodol. | 1 |
| 2025 | Prioritizing Speech Test CasesabstractAs Automated Speech Recognition (ASR) systems gain widespread acceptance, there is a pressing need to rigorously test and enhance their performance. Nonetheless, the process of collecting and executing speech test cases is typically both costly and time-consuming. This presents a compelling case for the strategic prioritization of speech test cases, which consist of a piece of audio and the corresponding reference text . The central question we address is: In what sequence should speech test cases be collected and executed to identify the maximum number of errors at the earliest stage ? In this study, we introduce PRiOritizing sPeecH tEsT ( Prophet ) cases, a tool designed to predict the likelihood that speech test cases will identify errors. Consequently, Prophet can assess and prioritize these test cases without having to run the ASR system, facilitating large-scale analysis. Our evaluation encompasses \(6\) distinct prioritization techniques across \(3\) ASR systems and \(12\) datasets. When constrained by the same test budget, our approach identified \(15.44\%\) more misrecognized words than the leading state-of-the-art method. We select top-ranked speech test cases from the prioritized list to fine-tune ASR systems and analyze how our approach can improve the ASR system performance. Statistical evaluations show that our method delivers a considerably higher performance boost for ASR systems compared to established baseline techniques. Moreover, our correlation analysis confirms that fine-tuning an ASR system with a dataset where the model initially underperforms tends to yield greater performance improvements. Zhou Yang 0003, Jieke Shi, Muhammad Hilmi Asyrofi, Xin Zhou 0014, Dong-Gyun Han, David Lo 0001 |
ACM Trans. Softw. Eng. Methodol. | 2 |
| 2024 | Unveiling Memorization in Code ModelsabstractThe availability of large-scale datasets, advanced architectures, and powerful computational resources have led to effective code models that automate diverse software engineering activities. The datasets usually consist of billions of lines of code from both open-source and private repositories. A code model memorizes and produces source code verbatim, which potentially contains vulnerabilities, sensitive information, or code with strict licenses, leading to potential security and privacy issues. Zhou Yang 0003, Chenyu Wang 0005, Jieke Shi, Dongsun Kim 0001, DongGyun Han, David Lo 0001 |
ICSE | 4 |
| 2024 | Curiosity-Driven Testing for Sequential Decision-Making ProcessabstractSequential decision-making processes (SDPs) are fundamental for complex real-world challenges, such as autonomous driving, robotic control, and traffic management. While recent advances in Deep Learning (DL) have led to mature solutions for solving these complex problems, SDMs remain vulnerable to learning unsafe behaviors, posing significant risks in safety-critical applications. However, developing a testing framework for SDMs that can identify a diverse set of crash-triggering scenarios remains an open challenge. To address this, we propose CureFuzz, a novel curiosity-driven black-box fuzz testing approach for SDMs. CureFuzz proposes a curiosity mechanism that allows a fuzzer to effectively explore novel and diverse scenarios, leading to improved detection of crash-triggering scenarios. Additionally, we introduce a multi-objective seed selection technique to balance the exploration of novel scenarios and the generation of crash-triggering scenarios, thereby optimizing the fuzzing process. We evaluate CureFuzz on various SDMs and experimental results demonstrate that CureFuzz outperforms the state-of-the-art method by a substantial margin in the total number of faults and distinct types of crash-triggering scenarios. We also demonstrate that the crash-triggering scenarios found by CureFuzz can repair SDMs, highlighting CureFuzz as a valuable tool for testing SDMs and optimizing their performance. Junda He, Zhou Yang 0003, Jieke Shi, Chengran Yang, Kisub Kim, Xin Zhou 0014, David Lo 0001 |
ICSE | 3 |
| 2024 | Baffle: Hiding Backdoors in Offline Reinforcement Learning DatasetsabstractReinforcement learning (RL) makes an agent learn from trial-and-error experiences gathered during the interaction with the environment. Recently, offline RL has become a popular RL paradigm because it saves the interactions with environments. In offline RL, data providers share large pre-collected datasets, and others can train high-quality agents without interacting with the environments. This paradigm has demonstrated effectiveness in critical tasks like robot control, autonomous driving, etc. However, less attention is paid to investigating the security threats to the offline RL system. This paper focuses on backdoor attacks, where some perturbations are added to the data (observations) such that given normal observations, the agent takes high-rewards actions, and low-reward actions on observations injected with triggers. In this paper, we propose Baffle (Backdoor Attack for Offline Reinforcement Learning), an approach that automatically implants backdoors to RL agents by poisoning the offline RL dataset, and evaluate how different offline RL algorithms react to this attack. Our experiments conducted on four tasks and nine offline RL algorithms expose a disquieting fact: none of the existing offline RL algorithms has been immune to such a backdoor attack. More specifically, Baffle modifies 10% of the datasets for four tasks (3 robotic controls and 1 autonomous driving). Agents trained on the poisoned datasets perform well in normal settings. However, when triggers are presented, the agents’ performance decreases drastically by 63.2%, 53.9%, 64.7%, and 47.4% in the four tasks on average. The backdoor still persists after fine-tuning poisoned agents on clean datasets. We further show that the inserted backdoor is also hard to be detected by a popular defensive method. This paper calls attention to developing more effective protection for the open-source offline RL dataset. Chen Gong 0005, Zhou Yang 0003, Yunpeng Bai, Junda He, Jieke Shi, Kecen Li, Arunesh Sinha, Xinwen Hou, David Lo 0001, Tianhao Wang 0001 |
SP | 5 |
| 2024 | Stealthy Backdoor Attack for Code ModelsabstractCode models, such as CodeBERT and CodeT5, offer general-purpose representations of code and play a vital role in supporting downstream automated software engineering tasks. Most recently, code models were revealed to be vulnerable to backdoor attacks. A code model that is backdoor-attacked can behave normally on clean examples but will produce pre-defined malicious outputs on examples injected withtriggersthat activate the backdoors. Existing backdoor attacks on code models use unstealthy and easy-to-detect triggers. This paper aims to investigate the vulnerability of code models withstealthybackdoor attacks. To this end, we propose Afraidoor(AdversarialFeature asAdaptive Backdoor). Afraidoorachieves stealthiness by leveraging adversarial perturbations to inject adaptive triggers into different inputs. We apply Afraidoorto three widely adopted code models (CodeBERT, PLBART, and CodeT5) and two downstream tasks (code summarization and method name prediction). We evaluate three widely used defense methods and find that Afraidooris more unlikely to be detected by the defense methods than by baseline methods. More specifically, when using spectral signature as defense, around 85% of adaptive triggers in Afraidoorbypass the detection in the defense process. By contrast, only less than 12% of the triggers from previous work bypass the defense. When the defense method is not applied, both Afraidoorand baselines have almost perfect attack success rates. However, once a defense is applied, the attack success rates of baselines decrease dramatically, while the success rate of Afraidoorremains high. Our finding exposes security weaknesses in code models under stealthy backdoor attacks and shows that state-of-the-art defense methods cannot provide sufficient protection. We call for more research efforts in understanding security threats to code models and developing more effective countermeasures. Zhou Yang 0003, Jie Zhang 0050, Hong Jin Kang, Jieke Shi, Junda He, David Lo 0001 |
IEEE Trans. Software Eng. | 5 |
| 2024 | Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code ModelsabstractLeveraging large-scale datasets from open-source projects and advances in large language models, recent progress has led to sophisticated code models for key software engineering tasks, such as program repair and code completion. These models are trained on data from various sources, including public open-source projects like GitHub and private, confidential code from companies, raising significant privacy concerns. This paper investigates a crucial but unexplored question:What is the risk of membership information leakage in code models?Membership leakage refers to the vulnerability where an attacker can infer whether a specific data point was part of the training dataset. We presentGotcha, a novel membership inference attack method designed for code models, and evaluate its effectiveness on Java-based datasets.Gotchasimultaneously considers three key factors: model input, model output, and ground truth. Our ablation study confirms that each factor significantly enhances attack performance. Our ablation study confirms that each factor significantly enhances attack performance. Our investigation reveals a troubling finding:membership leakage risk is significantly elevated. While previous methods had accuracy close to random guessing,Gotchaachieves high precision, with a true positive rate of 0.95 and a low false positive rate of 0.10. We also demonstrate that the attacker's knowledge of the victim model (e.g., model architecture and pre-training data) affects attack success. Additionally, modifying decoding strategies can help reduce membership leakage risks. This research highlights the urgent need to better understand the privacy vulnerabilities of code models and develop strong countermeasures against these threats. Zhou Yang 0003, Chenyu Wang 0005, Jieke Shi, Dongsun Kim 0001, DongGyun Han, David Lo 0001 |
IEEE Trans. Software Eng. | 4 |
| 2023 | What Do Users Ask in Open-Source AI Repositories? An Empirical Study of GitHub IssuesabstractArtificial Intelligence (AI) systems, which benefit from the availability of large-scale datasets and increasing computational power, have become effective solutions to various critical tasks, such as natural language understanding, speech recognition, and image processing. The advancement of these AI systems is inseparable from open-source software (OSS). Specifically, many benchmarks, implementations, and frameworks for constructing AI systems are made open source and accessible to the public, allowing researchers and practitioners to reproduce the reported results and broaden the application of AI systems. The development of AI systems follows a data-driven paradigm and is sensitive to hyperparameter settings and data separation. Developers may encounter unique problems when employing open-source AI repositories.This paper presents an empirical study that investigates the issues in the repositories of open-source AI repositories to assist developers in understanding problems during the process of employing AI systems. We collect 576 repositories from the PapersWithCode platform. Among these repositories, we find 24,953 issues by utilizing GitHub REST APIs. Our empirical study includes three phases. First, we manually analyze these issues to categorize the problems that developers are likely to encounter in open-source AI repositories. Specifically, we provide a taxonomy of 13 categories related to AI systems. The two most common issues are runtime errors (23.18%) and unclear instructions (19.53%). Second, we see that 67.5% of issues are closed. We also find that half of these issues resolve within four days. Moreover, issue management features, e.g., label and assign, are not widely adopted in open-source AI repositories. In particular, only 7.81% and 5.9% of repositories label issues and assign these issues to assignees, respectively. Finally, we empirically show that employing GitHub issue management features and writing issues with detailed descriptions facilitate the resolution of issues. Based on our findings, we make recommendations for developers to help better manage the issues of open-source AI repositories and improve their quality. Zhou Yang 0003, Chenyu Wang 0005, Jieke Shi, Thong Hoang, Pavneet Singh Kochhar, Qinghua Lu 0001, Zhenchang Xing, David Lo 0001 |
MSR | 3 |
| 2022 | Curiosity-Driven and Victim-Aware Adversarial PoliciesabstractRecent years have witnessed great potential in applying Deep Reinforcement Learning (DRL) in various challenging applications, such as autonomous driving, nuclear fusion control, complex game playing, etc. However, recently researchers have revealed that deep reinforcement learning models are vulnerable to adversarial attacks: malicious attackers can train adversarial policies to tamper with the observations of a well-trained victim agent, the latter of which fails dramatically when faced with such an attack. Understanding and improving the adversarial robustness of deep reinforcement learning is of great importance in enhancing the quality and reliability of a wide range of DRL-enabled systems. Chen Gong 0005, Zhou Yang 0003, Yunpeng Bai, Jieke Shi, Arunesh Sinha, David Lo 0001, Xinwen Hou |
ACSAC | 4 |
| 2022 | Natural Attack for Pre-trained Models of CodeabstractPre-trained models of code have achieved success in many important software engineering tasks. However, these powerful models are vulnerable to adversarial attacks that slightly perturb model inputs to make a victim model produce wrong outputs. Current works mainly attack models of code with examples that preserve operational program semantics but ignore a fundamental requirement for adversarial example generation: perturbations should be natural to human judges, which we refer to as naturalness requirement. Zhou Yang 0003, Jieke Shi, Junda He, David Lo 0001 |
ICSE | 2 |
| 2022 | Compressing Pre-trained Models of Code into 3 MBabstractAlthough large pre-trained models of code have delivered significant advancements in various code processing tasks, there is an impediment to the wide and fluent adoption of these powerful models in software developers’ daily workflow: these large models consume hundreds of megabytes of memory and run slowly on personal devices, which causes problems in model deployment and greatly degrades the user experience. Jieke Shi, Zhou Yang 0003, Hong Jin Kang, David Lo 0001 |
ASE | 1 |
| 2022 | Answer Summarization for Technical Queries: Benchmark and New ApproachabstractPrior studies have demonstrated that approaches to generate an answer summary for a given technical query in Software Question and Answer (SQA) sites are desired. We find that existing approaches are assessed solely through user studies. Hence, a new user study needs to be performed every time a new approach is introduced; this is time-consuming, slows down the development of the new approach, and results from different user studies may not be comparable to each other. There is a need for a benchmark with ground truth summaries as a complement assessment through user studies. Unfortunately, such a benchmark is non-existent for answer summarization for technical queries from SQA sites. Chengran Yang, Ferdian Thung, Yucen Shi, Ting Zhang 0011, Zhou Yang 0003, Xin Zhou 0014, Jieke Shi, Junda He, DongGyun Han, David Lo 0001 |
ASE | 8 |
| 2022 | Can Identifier Splitting Improve Open-Vocabulary Language Model of Code?abstractStatistical language models on source code have successfully assisted software engineering tasks. However, developers can create or pick arbitrary identifiers when writing source code. Freely chosen identifiers lead to the notorious out-of-vocabulary (OOV) problem that negatively affects model performance. Recently, Karampatsis et al. showed that using the Byte Pair Encoding (BPE) algorithm to address the OOV problem can improve the language models' predictive performance on source code. However, a drawback of BPE is that it cannot split the identifiers in a way that preserves the meaningful semantics. Prior researchers also show that splitting compound identifiers into sub-words that reflect the semantics can benefit software development tools. These two facts motivate us to explore whether identifier splitting techniques can be utilized to augment the BPE algorithm and boost the performance of open-vocabulary language models considered in Karampatsis et al.'s work. This paper proposes to split identifiers in both constructing vocabulary and processing model inputs procedures, thus exploiting three different settings of applying identifier splitting to language models for the code completion task. We contrast models' performance under these settings and find that simply inserting identifier splitting into the pipeline hurts the model performance, while a hybrid strategy combining identifier splitting and the BPE algorithm can outperform the original open-vocabulary models on predicting identifiers by 3.68% of recall and 6.32% of Mean Reciprocal Rank. The results also show that the hybrid strategy can improve the entropy of language models by 2.02%. Jieke Shi, Zhou Yang 0003, Junda He, David Lo 0001 |
SANER | 1 |
| 2022 | On the Influence of Biases in Bug Localization: Evaluation and BenchmarkabstractBug localization is the task of identifying parts of the source code that needs to be changed to resolve a bug report. As this task is difficult, automatic bug localization tools have been proposed. The development and evaluation of these tools rely on the availability of high-quality bug report datasets. In 2014, Kochhar et al. identified three biases in datasets used to evaluate bug localization techniques: (1) misclassified bug report, (2) already localized bug report, and (3) incorrect ground truth file in a bug report. They reported that already localized bug reports statistically significantly and substantially impact bug localization results, and thus should be removed. However, their evaluation is still limited, as they only investigated 3 projects written in Java. In this study, we replicate the study of Kochhar et al. on the effect of biases in bug report dataset for bug localization. Further investigation on this topic is necessary as new and larger bug report datasets have been proposed without being checked for these biases. We conduct our analysis on a collection of 2,913 bug reports taken from the recently released Bugzbook dataset that fix Python files. To investigate the prevalence of the biases, we check the bias distributions. For each bias, we select and label a set of bug reports that may contain the bias and compute the proportion of bug reports in the set that exhibit the bias. We find that 5%, 23%, and 30% of the bug reports that we investigated are affected by biases 1, 2, and 3 respectively. Then, we investigate the effect of the three biases on bug localization by measuring the performance of IncBL, a recent bug localization tool, and the classical Vector Space Model (VSM) based bug localization tool, which was used in the Kochhar et al. study. Our experiment results highlight that bias 2 significantly impact the bug localization results, while bias 1 and 3 do not have a significant impact. We also find that the effect sizes of bias 2 to IncBL and VSM are different, where IncBL has a higher effect size than VSM. Our findings corroborate the result reported by Kochhar et al. and demonstrate that bias 2 not only affects the 3 Java projects investigated in their study, but also others in another programming language (i.e., Python). This highlights the need to eliminate bias 2 from the evaluation of future bug localization tools. As a by-product of our replication study, we have released a benchmark dataset, which we refer to as CAPTURED, that has been cleaned from the three biases. CAPTURED contains Python programs and therefore augments the cleaned dataset released by Kochhar et al., which only contains Java programs. Ratnadira Widyasari, Stefanus A. Haryono, Ferdian Thung, Jieke Shi, Constance Tan, Fiona Wee, Jack Phan, David Lo 0001 |
SANER | 4 |
| 2022 | Revisiting Neuron Coverage Metrics and Quality of Deep Neural NetworksabstractDeep neural networks (DNN) have been widely applied in modern life, including critical domains like autonomous driving, making it essential to ensure the reliability and robustness of DNN-powered systems. As an analogy to code coverage metrics for testing conventional software, researchers have proposed neuron coverage metrics and coverage-driven methods to generate DNN test cases. However, Yan et al. doubt the usefulness of existing coverage criteria in DNN testing. They show that a coverage-driven method is less effective than a gradient-based method in terms of both uncovering defects and improving model robustness. In this paper, we conduct a replication study of the work by Yan et al. and extend the experiments for deeper analysis. A larger model and a dataset of higher resolution images are included to examine the generalizability of the results. We also extend the experiments with more test case generation techniques and adjust the process of improving model robustness to be closer to the practical life cycle of DNN development. Our experiment results confirm the conclusion from Yan et al. that coverage-driven methods are less effective than gradient-based methods. Yan et al. find that using gradient-based methods to retrain cannot repair defects uncovered by coverage-driven methods. They attribute this to the fact that the two types of methods use different perturbation strategies: gradient-based methods perform differentiable transformations while coverage-driven methods can perform additional non-differentiable transformations. We test several hypotheses and further show that even coverage-driven methods are constrained only to perform differentiable transformations, the uncovered defects still cannot be repaired by adversarial training with gradient-based methods. Thus, defensive strategies for coverage-driven methods should be further studied. Zhou Yang 0003, Jieke Shi, Muhammad Hilmi Asyrofi, David Lo 0001 |
SANER | 2 |
| 2021 | Can Differential Testing Improve Automatic Speech Recognition Systems?abstractDue to the widespread adoption of Automatic Speech Recognition (ASR) systems in many critical domains, ensuring the quality of recognized transcriptions is of great importance. A recent work, CrossASR++, can automatically uncover many failures in ASR systems by taking advantage of the differential testing technique. It employs a Text-To-Speech (TTS) system to synthesize audios from texts and then reveals failed test cases by feeding them to multiple ASR systems for cross-referencing. However, no prior work tries to utilize the generated test cases to enhance the quality of ASR systems. In this paper, we explore the subsequent improvements brought by leveraging these test cases from two aspects, which we collectively refer to as a novel idea, evolutionary differential testing. On the one hand, we fine-tune a target ASR system on the corresponding test cases generated for it. On the other hand, we fine-tune a cross-referenced ASR system inside CrossASR++, with the hope to boost CrossASR++'s performance in uncovering more failed test cases. Our experiment results empirically show that the above methods to leverage the test cases can substantially improve both the target ASR system and CrossASR++ itself. After fine-tuning, the number of failed test cases uncovered decreases by 25.81% and the word error rate of the improved target ASR system drops by 45.81%. Moreover, by evolving just one cross-referenced ASR system, CrossASR++ can find 5.70%, 7.25%, 3.93%, and 1.52% more failed test cases for 4 target ASR systems, respectively. Muhammad Hilmi Asyrofi, Zhou Yang 0003, Jieke Shi, Chu Wei Quan, David Lo 0001 |
ICSME | 3 |
| 2021 | BiasHeal: On-the-Fly Black-Box Healing of Bias in Sentiment Analysis SystemsabstractAlthough Sentiment Analysis (SA) is widely applied in many domains, existing research has revealed that the unfairness in SA systems can be harmful to the welfare of less privileged people. Several works propose pre-processing and in-processing methods to eliminate bias in SA systems, but little attention is paid to utilizing post-processing methods to heal bias. Postprocessing methods are particularly important for systems that use third-party SA services. Systems that use such services have no access to the SA engine or its training data and thus cannot apply pre-processing nor in-processing methods. Therefore, this paper proposes a black-box post-processing method to make an SA system heal bias and construct fair results when bias is detected. We propose and investigate six self-healing strategies. Our evaluation results on two datasets show that the best strategy can construct fair results and improve accuracy on the two datasets by 2.76% and 2.85%, respectively. To the best of our knowledge, our work is the first self-healing method that can be deployed to ensure SA fairness without requiring access to the SA engine or its training data. Zhou Yang 0003, Harshit Jain, Jieke Shi, Muhammad Hilmi Asyrofi, David Lo 0001 |
ICSME | 3 |
| 2021 | IncBL: Incremental Bug LocalizationabstractNumerous efforts have been invested in improving the effectiveness of bug localization techniques, whereas little attention is paid to making these tools run more efficiently in continuously evolving software repositories. This paper first analyzes the information retrieval model behind a classic bug localization tool, BugLocator, and builds a mathematical foundation illustrating that the model can be updated incrementally when codebase or bug reports evolve. Then, we present IncBL, a tool for Incremental Bug Localization in evolving software repositories. IncBL is evaluated on the Bugzbook dataset, and the results show that IncBL can significantly reduce the running time by 77.79% on average compared with the re-computing the model, while maintaining the same level of accuracy. We also implement IncBL as a Github App that can be easily integrated into open-source projects on GitHub. Users can deploy and use IncBL locally as well. The demo video for IncBL can be viewed at https://youtu.be/G4gMuvlJSb0, and the source code can be found at https://github.com/soarsmu/IncBL. Zhou Yang 0003, Jieke Shi, Shaowei Wang 0002, David Lo 0001 |
ASE | 2 |
| 2021 | Ontology negotiation: Knowledge interchange between distributed ontologies through agent negotiationabstractSummary With the proliferation of knowledge source on the internet as well as the widely professional agents, the knowledge interchange is drawing much attention. Ontology is recognized as the crucial technology due to their nature of sharing, formalization, and conceptualization to integrate and share the knowledge. In this paper, by interpreting and negotiating the communication content, a unified understanding of knowledge is formed; then, we can realize the interoperability between ontologies. We have developed the ontology automatic negotiation by agent elect protocol (AEP) to elect optimal participants and encourage agents to obey the protocol, concept mapping protocol (CMP) to find the corresponding concept mappings with the highest relevancy, and in addition, agent negotiation protocol (ANP). In ANP, we define the simultaneous negotiation protocol and agents' strategies to combine distributed ontologies interchange with agent negotiation. Finally, the implementation and preliminary results are given to verify the validity of the proposed ontology negotiation. Junwu Zhu, Ling Teng, Huimin Lu 0001, Jieke Shi, Bin Li 0006 |
Concurr. Comput. Pract. Exp. | 4 |
| 2020 | Network-aware Virtual Machine Migration Based on Gene Aggregation Genetic Algorithm
Yi Jiang 0004, Jieke Shi, Junwu Zhu, Ling Teng |
Mob. Networks Appl. | 3 |
| 2020 | An auction-based rescue task allocation approach for heterogeneous multi-robot system
Jieke Shi, Zhou Yang 0003, Junwu Zhu |
Multim. Tools Appl. | 1 |
| 2019 | An Efficient Double Auction Mechanism for Job AllocationabstractJob allocation is a common application of human resource competition in the market. The third-party platform plays a vital role in coordinating the supply relationship between job suppliers and job seekers in order to help the talent market to allocate resources reasonably. For maximal social welfare, this paper proposes and implements a greedy double auction mechanism (GDA) based on the descending order of bid differentials. In the process of auction, virtual positions are regarded as heterogeneous commodities, and greedy algorithm is used to solve the optimal solution which maximizes social welfare and satisfies constraints. GDA has the economic attributes of individual rationality, budget balance as well as authenticity by the theoretical proof. At the same time, the feasibility and effectiveness of the mechanism are verified by simulation experiments. The experimental results show that the GDA mechanism has higher social welfare and average utility of job seekers, which can make job providers and excellent job seekers have a stable foothold in the increasingly fierce human resources competition market. Jieke Shi, Junwu Zhu, Yunbo Lv |
CSCWD | 1 |