Large-scale Android apps that provide complex functions are gradually becoming the mainstream in Android app markets. They tend to display many GUI widgets on a single GUI page, which, unfortunately, can cause more redundant test actions—actions with similar functions—to automatic testing approaches. The effectiveness of existing testing approaches is still limited, suggesting the necessity of reducing the test effort on redundant actions. In this article, we first identify three types of GUI structures that can cause redundant actions and then propose a novel approach, called action equivalence evaluation, to find the actions with similar functions by exploiting both GUI structure and functionality. By integrating this approach with existing testing tools, the test efficacy can be improved. We conducted experiments on 17 large-scale Android apps, including three industrial apps Google News , Messenger , and WeChat . The results show that more instructions can be covered, and more crashes can be detected, compared to the state-of-the-art Android testing tools. Twenty-nine real bugs were found in our experiment, and moreover, 760 bugs over 40 versions of WeChat had been detected in the real test environment during a 3-month testing period.

Code completion, a crucial task in software engineering that enhances developer productivity, has seen substantial improvements with the rapid advancement of large language models (LLMs). In recent years, retrieval-augmented generation (RAG) has emerged as a promising method to enhance the code completion capabilities of LLMs, which leverages relevant context from codebases without requiring model retraining. While existing studies have demonstrated the effectiveness of RAG on public repositories and benchmarks, the potential distribution shift between open-source and closed-source codebases presents unique challenges that remain unexplored. To mitigate the gap, we conduct an empirical study to investigate the performance of widely-used RAG methods for code completion in the industrialscale codebase of WeChat, one of the largest proprietary software systems. Specifically, we extensively explore two main types of RAG methods, namely identifier-based RAG and similaritybased RAG, across 26 open-source LLMs ranging from 0.5B to 671B parameters. For a more comprehensive analysis, we employ different retrieval techniques for similarity-based RAG, including lexical and semantic retrieval. Based on 1,669 internal repositories, we achieve several key findings: (1) both RAG methods demonstrate effectiveness in closed-source repositories, with similarity-based RAG showing superior performance, (2) the effectiveness of similarity-based RAG improves with more advanced retrieval techniques, where BM25 (lexical retrieval) and GTE-Qwen (semantic retrieval) achieve superior performance, and (3) the combination of lexical and semantic retrieval techniques yields optimal results, demonstrating complementary strengths. Furthermore, we conduct a developer survey to validate the practical utility of RAG methods in real-world development environments.

Attracting over one billion registered users globally, WeChat’s mini-game platform has become one of the largest gaming platforms with hundreds of thousands of published mini-games. To ensure the quality of experiences across a massive number of mini-games, automated UI testing has become essential for WeChat. However, sliding-gesture-induced exploration tarpits, states where a testing tool becomes trapped in repetitive, unsuccessful gesture attempts, cause the testing tool to waste up to 98% of its testing budget due to its inability to execute proper sliding gestures. While mini-games typically contain visual hints (e.g., sliding indicators) guiding the desired sliding gestures, exploiting these hints to escape exploration tarpits faces two major challenges in industrial settings: (1) robustness challenge when exploiting hints from only several discontinuous screenshots, and (2) efficiency challenge to support thousands of concurrent testing services with minimal overhead and costs.To address the preceding challenges, we report our experiences in developing and deploying SlideScout, a three-stage approach for detecting and escaping sliding-gesture-induced exploration tarpits via efficient exploitation of visual hints. First, SlideScout concurrently monitors the testing progress and detects sliding indicators alongside screenshot collection, improving efficiency by reusing preprocessed results in subsequent stages. Second, SlideScout reconstructs potential sliding trajectories using multiple heuristics, addressing robustness challenges when precise trajectories are unavailable due to discontinuous screenshots. Third, SlideScout applies the inferred sliding gestures until it successfully escapes the tarpit, enabling easy integration with existing testing tools. Deployed at WeChat internally for six months, SlideScout has helped reveal 25,000 crashes and 120,000 JavaScript errors, detecting 50% more crashes compared to the pre-deployment baseline within the same time period. We summarize three major lessons learned from developing and deploying SlideScout.

Large Code Models (LCMs) have demonstrated potential in advancing various code intelligence tasks. However, their effectiveness can be greatly influenced by the quality of the prompts. Current prompt design strategies in code intelligence studies are mostly manually generated, which could be time-consuming and extremely rely on the base LCMs and tasks. Although automated prompt generation (APG) has been investigated in the natural language processing field, it has not attracted sufficient attention and been well explored in the code intelligence tasks. Considering the various tasks and black-box nature of LCMs faced by developers in practice, it is essential to automate the prompt generation process.To mitigate the gap, we empirically investigate the two important parts in APG, including Instruction Generation (IG) and Muti-Step Reasoning (MSR). The instruction generation part aims at providing a task-related description for instructing LCMs to effectively accomplish specific tasks; while the multistep reasoning part aims at guiding LCMs to produce a series of logical steps before arriving at the final answer. For each part, we evaluate the widely-used APG methods on four open-source LCMs and three code intelligence tasks, i.e., code translation (PL-PL), code summarization (PL-NL) and API recommendation (NL-PL). Experimental results indicate that the two parts in APG can dramatically enhance the performance of the code intelligence tasks compared with the basic prompts. Based on the results, we further propose a novel APG approach by combining the best methods of the two studied parts of APG. Experiments show that the proposed APG approach achieves an average improvement of 28.38% with respect to CodeBLEU for the code translation, 58.11% in terms of ROUGE-L for the code summarization and 84.53% in SuccessRate@1 for the API recommendation over the basic prompts, respectively. To validate the effectiveness in industrial scenario, we further evaluate our approach on WeChat-Bench, a proprietary dataset from the WeChat Group in Tencent for API recommendation, achieving an average improvement of 148.89% in MRR.

The explosive growth of mini-game platforms has led to widespread code plagiarism, where malicious users access popular games’ source code and republish them with modifications. While existing static analysis tools can detect simple obfuscation techniques like variable renaming and dead code injection, they fail against sophisticated deep obfuscation methods such as encrypted code with local or cloud-based decryption keys that completely destroy code structure and render traditional Abstract Syntax Tree analysis ineffective. To address these challenges, we present JSidentify-V2, a novel dynamic analysis framework that detects mini-game plagiarism by capturing memory invariants during program execution. Our key insight is that while obfuscation can severely distort static code characteristics, runtime memory behavior patterns remain relatively stable. JSidentify-V2 employs a four-stage pipeline: (1) static pre-analysis and instrumentation to identify potential memory invariants, (2) adaptive hot object slicing to maximize execution coverage of critical code segments, (3) Memory Dependency Graph construction to represent behavioral fingerprints resilient to obfuscation, and (4) graph-based similarity analysis for plagiarism detection.We evaluate JSidentify-V2 against eight obfuscation methods on a comprehensive dataset of 1,200 mini-games. Our framework achieves over 90% similarity detection across all tested obfuscation techniques, maintaining high accuracy even against advanced decryption-based methods where existing tools achieve near 0% detection rates. In real-world deployment, JSidentify-V2 achieves 100% precision and 99.8% recall while delivering an 8× speedup compared to previous methods. Our production deployment demonstrates that plagiarism complaints have decreased by over 80%, proving JSidentify-V2’s effectiveness in protecting intellectual property rights in mini-game ecosystems.

User Interface (UI) testing is crucial for quality assurance of industrial mobile applications, and yet it remains labor-intensive and challenging to automate effectively. Recent advances in Vision-Language Models (VLMs) present a promising solution for automating GUI testing by mapping natural language instructions to pixel-level actions, significantly reducing the manual effort required for writing test scripts and even designing test cases. While numerous VLMs have been proposed and evaluated for GUI testing, they often fail to meet two critical industrial requirements: (1) effectiveness when handling complex, multi-step workflows in industrial applications, and (2) efficiency for large-scale, high-frequency testing environments typical in industrial settings. Toward addressing the preceding industrial requirements, in this paper, we report our experiences in developing and deploying RePeek, a novel approach employing a unified three-stage pipeline for both training and inference, enables a VLM to explicitly detect and reason over discrete GUI elements, thereby overcoming the limitations of pixel-based reasoning for both efficiency and effectiveness improvements. In the first stage, RePeek integrates a lightweight UI-element detector named OmniParser to decompose UI screenshots into a structured element list. In the second stage, RePeek adopts the vision encoder of the VLM to generate the embedding for each element. In the third stage, RePeek fuses these element embeddings with the textual instruction to reason and perform classification directly on the UI elements, empowering efficient small models to achieve superior performance against expensive large models. Comprehensive evaluations on public benchmarks and deployment at WeChat show that RePeek consistently achieves superior accuracy and efficiency compared to state-of-the-art VLMs. Specifically, RePeek enables a fine-tuned Qwen2.5-VL-3B model to outperform a 72B model with 75% less training data, validating the effectiveness of incorporating domain knowledge into VLM-based GUI testing. We conclude by summarizing three key lessons from developing and deploying RePeek, offering insights for both researchers and practitioners working on industrial-strength UI testing.

UI automation tests play a crucial role in ensuring the quality of mobile applications. Despite the growing popularity of machine learning techniques to generate these tests, they still face several challenges, such as the mismatch of UI elements. The recent advances in Large Language Models (LLMs) have addressed these issues by leveraging their semantic understanding capabilities. However, a significant gap remains in applying these models to industrial-level app testing, particularly in terms of cost optimization and knowledge limitation. To address this, we introduce CAT to create cost-effective UI automation tests for industry apps by combining machine learning and LLMs with best practices. Given the task description, CAT employs Retrieval Augmented Generation (RAG) to source examples of industrial app usage as the few-shot learning context, assisting LLMs in generating the specific sequence of actions. CAT then employs machine learning techniques, with LLMs serving as a complementary optimizer, to map the target element on the UI screen. Our evaluations on the WeChat testing dataset demonstrate the CAT's performance and cost-effectiveness, achieving 90% UI automation with $0.34 cost, outperforming the state-of-the-art. We have also integrated our approach into the real-world WeChat testing platform, demonstrating its usefulness in detecting 141 bugs and enhancing the developers' testing process.

Modern systems generate a massive amount of logs to detect and diagnose system faults, which incurs expensive storage costs and runtime overhead. After investigating real-world production logs, we observe that most of the logging overhead is due to a small number of log templates, referred to as log hotspots. Therefore, we conduct a systematical study about log hotspots in an industrial system WeChat, which motivates us to identify log hotspots and reduce them on the fly. In this paper, we propose LogReducer, a non-intrusive and language-independent log reduction framework based on eBPF (Extended Berkeley Packet Filter), consisting of both online and offline processes. After two months of serving the offline process of LogReducer in WeChat, the log storage overhead has dropped from 19.7 PB per day to 12.0 PB (i.e., about a 39.08% decrease). Practical implementation and experimental evaluations in the test environment demonstrate that the online process of LogReducer can control the logging overhead of hotspots while preserving logging effectiveness. Moreover, the log hotspot handling time can be reduced from an average of 9 days in production to 10 minutes in the test with the help of LogReducer,

Modern large-scale service-based systems such as microservice systems have become increasingly complex, making it hard to localize anomalous services when various issues emerge. Traces record the workflows of requests through service instances and have been widely used in anomaly detection and root cause analysis. Existing trace-based approaches widely use statistical methods or learning-based techniques to detect trace anomalies and localize anomalous services. However, these approaches often suffer from the concept drift problem, i.e., the statistical properties of traces change over time in unforeseen ways. In this paper, we propose TraceStream, an anomalous service localization approach based on trace data stream clustering. TraceStream uses data stream clustering to discover potential anomalous trace clusters in evolving trace data and uses spectrum analysis to localize anomalous services based on the clusters. Moreover, TraceStream can effectively incorporate the online feedback of operation engineers based on the trace clusters to improve the accuracy for localizing anomalous services. Our evaluation confirms that TraceStream can effectively detect anomalies and localize anomalous services in an evolving microservice system. It can effectively incorporate human feedback to further improve the performance of anomalous service localization. Moreover, TraceStream is efficient and its efficiency can be further improved by sampling a small portion of traces by cluster.

Development teams in large companies often maintain a huge codebase whose build time can be painfully long in a single machine. To reduce the build time, tools such as Bazel and distcc are used to build the code base in a distributed way. However, in the process of distributed build, certain remote slave machines can crash due to two types of errors: Out Of Memory (OOM) and Deadline Exceeded (DE) errors. These crashes lead to time-consuming rebuilds, as suffered by the WeiXin Group (WXG) of Tencent Inc. (the vendor of WeChat, a highly popular mobile app in China). Aiming to prevent these two types of errors, in this paper, we propose a new approach named PCRLINEAR, which predicts the memory and time requirements of the given C++ file, allowing the underlying distributed build system to schedule compilation resources adaptively according to the prediction results. Our experiments show that PCRLINEAR reduces the number of OOM and DE errors from 5% to 0.2% and, at the same time, achieves substantial build-performance improvement of 30% on average.

WeChat, a widely-used messenger app boasting over 1 billion monthly active users, requires effective app quality assurance for its complex features. Record-and-replay tools are crucial in achieving this goal. Despite the extensive development of these tools, the impact of waiting time between replay events has been largely overlooked. On one hand, a long waiting time for executing replay events on fully-rendered GUIs slows down the process. On the other hand, a short waiting time can lead to events executing on partially-rendered GUIs, negatively affecting replay effectiveness. An optimal waiting time should strike a balance between effectiveness and efficiency. We introduce WeReplay, a lightweight image-based approach that dynamically adjusts inter-event time based on the GUI rendering state. Given the real-time streaming on the GUI, WeReplay employs a deep learning model to infer the rendering state and synchronize with the replaying tool, scheduling the next event when the GUI is fully rendered. Our evaluation shows that our model achieves 92.1% precision and 93.3% recall in discerning GUI rendering states in the WeChat app. Through assessing the performance in replaying 23 common WeChat usage scenarios, WeReplay successfully replays all scenarios on the same and different devices more efficiently than the state-of-the-practice baselines.

Code completion has become a common practice for programmers during their daily programming activities. It automatically predicts the next tokens or statements that the programmers may use. Code completion aims to substantially save keystrokes and improve the programming efficiency for programmers. Although there exists substantial research on code completion, it is still unclear what practitioner expectations are on code completion and whether these expectations are met by the existing research. To address these questions, we perform a study by first interviewing 15 professionals and then surveying 599 practitioners from 18 IT companies about their expectations on code completion. We then compare the practitioner expectations with the existing research by conducting a literature review of papers on code completion published in major publication venues from 2012 to 2022. Based on the comparison, we highlight the directions desirable for researchers to invest efforts toward developing code completion techniques for meeting practitioner expectations.

Mobile games play an increasingly important role in our daily life. The quality of mobile games can substantially affect the user experience and game revenue. Different from traditional mobile games, the mini-games provided by our partner, Tencent, are embedded in the mobile app WeChat, so users do not need to install specific game apps and can directly play the games in the app. Due to the convenient installation, WeChat has attracted large numbers of developers to design and publish on the mini-game platform in the app. Until now, the platform has more than one hundred thousand published mini-games. Manually testing all the mini-games requires enormous effort and is impractical. There exist automated game testing methods; however, they are difficult to be applied for testing mini-games for the following reasons: 1) Effective game testing heavily relies on prior knowledge about game operations and extraction of GUI widget trees. However, this knowledge is specific and not always applicable when testing a large number of mini-games with complex game engines (e.g., Unity). 2) The highly diverse GUI widget design of mini-games deviates significantly from that of mobile apps. Such issue prevents the existing image-based GUI widget detection techniques from effectively detecting widgets in mini-games.

Built on the WeChat social platform, WeChat Mini-Programs are widely used by more than 400 million users every day. Consequently, the reliability of Mini-Programs is particularly crucial. However, WeChat Mini-Programs suffer from various bugs related to execution environment, lifecycle management, asynchronous mechanism, etc. These bugs have seriously affected users' experience and caused serious impacts.

GUI testing is an essential part of regression testing for Android apps. For regression GUI testing to remain effective, it is important that obsolete GUI test scripts get repaired after the app has evolved. In this paper, we propose a novel approach named GUIDER to automated repair of GUI test scripts for Android apps. The key novelty of the approach lies in the utilization of both structural and visual information of widgets on app GUIs to better understand what widgets of the base version app become in the updated version. A supporting tool has been implemented for the approach. Experiments conducted on the popular messaging and social media app WeChat show that GUIDER is both effective and efficient. Repairs produced by GUIDER enabled 88.8% and 54.9% more test actions to run correctly than those produced by existing approaches to GUI test repair that rely solely on visual or structural information of app GUIs.

Node.js has become a widely-used event-driven architecture for server-side and desktop applications. Node.js provides an effective asynchronous event-driven programming model, and supports asynchronous tasks and multi-priority event queues. Unexpected races among events and asynchronous tasks can cause severe consequences. Existing race detection approaches in Node.js applications mainly adopt random fuzzing technique, and can miss races due to large schedule space.In this paper, we propose a dynamic race detection approach NRace for Node.js applications. In NRace, we build precise happens-before relations among events and asynchronous tasks in Node.js applications, which also take multi-priority event queues into consideration. We further develop a predictive race detection technique based on these relations. We evaluate NRace on 10 realworld Node.js applications. The experimental result shows that NRace can precisely detect 6 races, and 5 of them have been confirmed by developers.

JavaScript is one of the most popular programming languages. WeChat Mini-Program is a large ecosystem of JavaScript applications that runs on the WeChat platform. Millions of Mini-Programs are accessed by WeChat users every week. Consequently, the performance and robustness of Mini-Programs are particularly important. Unfortunately, many Mini-Programs suffer from various defects and performance problems. Dynamic analysis is a useful technique to pinpoint application defects. However, due to the dynamic features of the JavaScript language and the complexity of the runtime environment, dynamic analysis techniques were rarely used to improve the quality of JavaScript applications running on industrial platforms such as WeChat Mini-Program previously. In this work, we report our experience of extending Jalangi, a dynamic analysis framework for JavaScript applications developed by academia, and applying the extended version, named WeJalangi, to diagnose defects in WeChat Mini-Programs. WeJalangi is compatible with existing dynamic analysis tools such as DLint, Smemory, and JITProf. We implemented a null pointer checker on WeJalangi and tested the tool's usability on 152 open-source Mini-Programs. We also conducted a case study in Tencent by applying WeJalangi on six popular commercial Mini-Programs. In the case study, WeJalangi accurately located six null pointer issues and three of them haven't been discovered previously. All of the reported defects have been confirmed by developers and testers.

For large industrial applications, system test cases are still often described in natural language (NL), and their number can reach thousands. Test automation is to automatically execute the test cases. Achieving test automation typically requires substantial manual effort for creating executable test scripts from these NL test cases. In particular, given that each NL test case consists of a sequence of NL test steps, testers first implement a test API method for each test step and then write a test script for invoking these test API methods sequentially for test automation. Across different test cases, multiple test steps can share semantic similarities, supposedly mapped to the same API method. However, due to numerous test steps in various NL forms under manual inspection, testers may not realize those semantically similar test steps and thus waste effort to implement duplicate test API methods for them. To address this issue, in this paper, we propose a new approach based on natural language processing to cluster similar NL test steps together such that the test steps in each cluster can be mapped to the same test API method. Our approach includes domain-specific word embedding training along with measurement based on Relaxed Word Mover’sDistance to analyze the similarity of test steps. Our approach also includes a technique to combine hierarchical agglomerative clustering and K-means clustering post-refinement to derive high-quality and manually-adjustable clustering results. The evaluation results of our approach on a large industrial mobile app, WeChat, show that our approach can cluster the test steps with high accuracy, substantially reducing the number of clusters and thus reducing the downstream manual effort. In particular, compared with the baseline approach, our approach achieves 79.8% improvement on cluster quality, reducing 65.9% number of clusters, i.e., the number of test API methods to be implemented.

Large-scale online systems are complex, fast-evolving, and hardly bug-free despite the testing efforts. Backend system monitoring cannot detect many types of issues, such as UI related bugs, bugs with small impact on backend system indicators, or errors from third-party co-operating systems, etc. However, users are good informers of such issues: They will provide their feedback for any types of issues. This experience paper discusses our design of iFeedback, a tool to perform real-time issue detection based on user feedback texts. Unlike traditional approaches that analyze user feedback with computation-intensive natural language processing algorithms, iFeedback is focusing on fast issue detection, which can serve as a system life-condition monitor. In particular, iFeedback extracts word combination-based indicators from feedback texts. This allows iFeedback to perform fast system anomaly detection with sophisticated machine learning algorithms. iFeedback then further summarizes the texts with an aim to effectively present the anomaly to the developers for root cause analysis. We present our representative experiences in successfully applying iFeedback in tens of large-scale production online service systems in ten months.

Lexical-based metrics such as BLEU, NIST, and WER have been widely used in machine translation (MT) evaluation. However, these metrics badly represent semantic relationships and impose strict identity matching, leading to moderate correlation with human judgments. In this paper, we propose a novel MT automatic evaluation metric Semantic Travel Distance (STD) based on word embeddings. STD incorporates both semantic and lexical features (word embeddings and n-gram and word order) into one metric. It measures the semantic distance between the hypothesis and reference by calculating the minimum cumulative cost that the embedded n-grams of the hypothesis need to “travel” to reach the embedded n-grams of the reference. Experiment results show that STD has a better and more robust performance than a range of state-of-the-art metrics for both the segment-level and system-level evaluation.

User Interface (UI) testing is a popular approach to ensure the quality of mobile apps. Numerous test generation tools have been developed to support UI testing on mobile apps, especially for Android apps. Previous work evaluates and compares different test generation tools using only relatively simple open-source apps, while real-world industrial apps tend to have more complex functionalities and implementations. There is no direct comparison among test generation tools with regard to effectiveness and ease-of-use on these industrial apps. To address such limitation, we study existing state-of-the-art or state-of-the-practice test generation tools on 68 widely-used industrial apps. We directly compare the tools with regard to code coverage and fault-detection ability. According to our results, Monkey, a state-of-the-practice tool from Google, achieves the highest method coverage on 22 of 41 apps whose method coverage data can be obtained. Of all 68 apps under study, Monkey also achieves the highest activity coverage on 35 apps, while Stoat, a state-of-the-art tool, is able to trigger the highest number of unique crashes on 23 apps. By analyzing the experimental results, we provide suggestions for combining different test generation tools to achieve better performance. We also report our experience in applying these tools to industrial apps under study. Our study results give insights on how Android UI test generation tools could be improved to better handle complex industrial apps.

Mobile applications, or apps for short, are gaining popularity. The input sources (e.g., touchscreen, sensors, transmitters) of the smart devices that host these apps enable the apps to offer a rich experience to the users, but these input sources pose testing complications to the developers (e.g., writing tests to accurately utilize multiple input sources together and be able to replay such tests at a later time). To alleviate these complications, researchers and practitioners in recent years have developed a variety of record-and-replay tools to support the testing expressiveness of smart devices. These tools allow developers to easily record and automate the replay of complicated usage scenarios of their app. Due to Android's large share of the smart-device market, numerous record-and-replay tools have been developed using a variety of techniques to test Android apps. To better understand the strengths and weaknesses of these tools, we present a comparison of popular record-and-replay tools from researchers and practitioners, by applying these tools to test three popular industrial apps downloaded from the Google Play store. Our comparison is based on three main metrics: (1) ability to reproduce common usage scenarios, (2) space overhead of traces created by the tools, and (3) robustness of traces created by the tools (when being replayed on devices with different resolutions). The results from our comparison show which record-and-replay tools may be the best for developers and identify future directions for improving these tools to better address testing complications of smart devices.

Given the ever increasing number of research tools to automatically generate inputs to test Android applications (or simply apps), researchers recently asked the question "Are we there yet?" (in terms of the practicality of the tools). By conducting an empirical study of the various tools, the researchers found that Monkey (the most widely used tool of this category in industrial practices) outperformed all of the research tools that they studied. In this paper, we present two significant extensions of that study. First, we conduct the first industrial case study of applying Monkey against WeChat, a popular messenger app with over 762 million monthly active users, and report the empirical findings on Monkey's limitations in an industrial setting. Second, we develop a new approach to address major limitations of Monkey and accomplish substantial code-coverage improvements over Monkey, along with empirical insights for future enhancements to both Monkey and our approach.

AGENDA is a tool set for testing relational database applications. An earlier prototype was targeted to applications consisting of a single query and included components for populating a database with data suitable for testing the application, generating inputs to the query, and checking relatively simple aspects of the results of executing the query. This paper describes substantial extensions to AGENDA, allowing it to test transactions with multiple queries and with complex intended behavior. The paper introduces a technique for checking complex properties of the database state transition performed by the transaction under test, as well as an improved input generation heuristic. Results of using AGENDA to test three applications with seeded faults are presented.

Abstract Database systems play an important role in nearly every modern organization, yet relatively little research effort has focused on how to test them. This paper discusses issues arising in testing database systems, presents an approach to testing database applications, and describes AGENDA, a set of tools to facilitate the use of this approach. In testing such applications, the state of the database before and after the user's operation plays an important role, along with the user's input and the system output. A framework for testing database applications is introduced. A complete tool set, based on this framework, has been prototyped. The components of this system are a parsing tool that gathers relevant information from the database schema and application, a tool that populates the database with meaningful data that satisfy database constraints, a tool that generates test cases for the application, a tool that checks the resulting database state after operations are performed by a database application, and a tool that assists the tester in checking the database application's output. The design and implementation of each component of the system are discussed. The prototype described here is limited to applications consisting of a single SQL query. Copyright © 2004 John Wiley & Sons, Ltd.

Database systems play an important role in nearly every modern organization, yet relatively little research effort has focused on how to test them. AGENDA, A (test) GENerator for Database Applications, is a research prototype tool set for testing DB application programs. In testing such applications, the states of the database before and after execution play an important role, along with the user's input and system output. AGENDA components populate the database, generate inputs, and check aspects of the correctness of output and new DB state.

Database application programs are often designed to be executed concurrently by many users. By grouping related database queries into transactions, DBMS (database management system) can guarantee that each transaction satisfies the well-known ACID properties: atomicity, consistency, isolation, and durability. However, if a database application is decomposed into transactions in an incorrect manner, the application may fail when executed concurrently due to potential offline concurrency problems. This paper presents a dataflow analysis technique for identifying schedules of transaction execution aimed at revealing concurrency faults of this nature, along with techniques for controlling the DBMS or the application so that execution of transaction sequences follows generated schedules. The techniques have been integrated into AGENDA, a tool set for testing relational database application programs. Preliminary empirical evaluation is presented.

Many analytical and empirical studies of software testing effectiveness have used the probability that a test set exposes at least one fault as the measure of effectiveness. That measure is useful for evaluating testing techniques when the goal of testing is to gain confidence that the program is free from faults. However, if the goal of testing is to improve the reliability of the program (by discovering and removing those faults that are most likely to cause failures when the software is in the field) then the measure of test effectiveness must distinguish between those faults that are likely to cause failures and those that are unlikely to do so. Delivered reliability was previously introduced as a means of comparing testing techniques in that setting. This paper empirically compares reliability delivered by three testing techniques, branch testing, the all-uses data flow testing criterion, and operational testing. The subject program is a moderate-sized C-program (about 10,000 LOC) produced by professional programmers and containing naturally occurring faults.