Understanding how software bugs affect user experience and how well large language models (LLMs) can detect them is critical for building usable, high-quality mobile applications.In this study, we analyze 370 user reviews to assess the impact of bugs on usability and evaluate the effectiveness of state-of-the-art LLMs-including GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro, and LLaMA 3-in extracting usability-related information.Our results show that the most common impacts are unexpected behavior and blocked access to core features.Among the evaluated models, GPT-4 consistently achieves the highest accuracy across bug type classification, UI component extraction, and heuristic violation detection.Our findings highlight the promise of instruction-tuned LLMs in automating usability analysis and improving mobile app quality through user feedback.A replication package is available at

Applications of machine learning to automated testing are gaining momentum.This work applies machine learning to test suite reduction in gray box test suites to reduce the time and cost of web application testing.A family of empirical studies examines the effectiveness of several techniques across seven JavaScript-based web applications, with test suites generated using the Playwright framework.Supervised learning models-Support Vector Machines (SVM), K-Nearest Neighbors (KNN), and Decision Trees (DT)-are compared with unsupervised K-Means clustering to detect redundancy based on execution count and code coverage.Our results show that over 56% of test cases across applications are redundant, with DT achieving the highest accuracy in matching manually labeled redundancy.KNN also performed competitively, while unsupervised clustering was less effective in fine-grained redundancy detection.Notably, one application achieved a 75% reduction in test execution time with only moderate losses in statement coverage.On average, statement coverage dropped by 30-50%, depending on the technique and application.These findings highlight the tradeoff between time savings and coverage preservation.This study demonstrates that ML-based test suite reduction is a viable and scalable approach for modern web testing pipelines, with supervised learning offering the most practical benefits.

Visual GUI Testing (VGT) is redefining software interface testing by shifting the emphasis from internal code structures to the user's visual experience.As modern applications become increasingly dynamic and visually complex, traditional GUI testing methods reliant on code or object properties struggle with adaptability and require extensive maintenance.VGT addresses these challenges through techniques such as image-based recognition, screen-level interaction, and visual verification, providing a more resilient and user-centered testing paradigm adaptable across diverse platforms and development environments.In this survey, we present a comprehensive analysis of the VGT landscape, with a particular emphasis on automatic test case generation.We explore both conventional methodologies and recent AI-driven approaches, including supervised learning, layout clustering, reinforcement learning, and search-based optimization.Drawing insights from over 40 research studies, we critically evaluate the capabilities of state-of-the-art tools, identify prevalent evaluation metrics, and highlight persistent challenges within visual GUI testing.The primary objective of this work is to synthesize current research trends, clarify future development directions for VGT, and facilitate the advancement of intelligent, adaptable, and automated GUI testing solutions.This survey serves as a valuable resource for researchers and practitioners aiming to enhance visual GUI software testing through innovative AI methodologies and empirical evaluations.

This article asserts that use of IoT (Internet of Things) devices have significantly increased over the last decade, specifically smartphones as compared to desktops, and laptops have become an integral part of our everyday lives. Smartphone applications operate in dynamic environments and generate huge and vast amount of context events such as screen orientation, location, battery life, and network connectivity throughout the day. This work proposes a modified method of Compact Prediction Tree (CPT) called Context Data Compact Prediction Tree (CD-CPT) to predict real-world context data for multiple users. The experiments conducted used Transition Directed Acyclic Graph (TDAG) and All-k Order Markov (AKOM) algorithms to generate short-term predictions based on current context events and compare with baseline models such as Prediction by Pattern Mining (PPM), Dependency Graph (DG), CPT, and CPT+.

Context driven environments are growing in popularity.Mobile applications, Internet of Things devices, autonomous vehicles, and future technologies respond to context events in their environments.This work uses a set of context events from real users to guide the generation of context driven test cases.Context event sequences are obtained by applying Conditional Random Fields (CRF).Test suites are then constructed by interleaving the context event sequences with GUI events.The choice of context event is made based on transitions obtained from the CRF.Results of the empirical studies show that techniques that incorporate context events provide better code coverage than NoContext for the subject applications.A heuristic technique introduced in this work, ISFreqOne, yields 4x better coverage than NoContext, 0.06x better coverage than Random Start Context, 0.05x better coverage than Iterative Start Context, which are control context generation techniques, and 0.04x better coverage than ISFreqTwo, another heuristic introduced in this work.

Mobile devices such as smart phones and smart watches utilize apps that run in context aware environments and must respond to context changes such as changes in network connectivity, battery level, screen orientation, and more.The large number of GUI events and context events often complicate the testing process.This work expands the AutoDroid tool to automatically generate tests that are guided by PairwiseInterleaved coverage of GUI event and context event sequences.We systematically weave context and GUI events into testing using the pairwise interleaved algorithm.The results show that the pairwise interleaved algorithm achieves up to five times higher code coverage compared to a technique that generates test suites in a single predefined context (without interleaving context and GUI events), a technique that changes the context at the beginning of each test case (without interleaving context and GUI events), and Monkey-Context-GUI (which randomly chooses context and GUI events).Future work will expand this strategy to include more context variables and test emerging technologies such as IoT and autonomous vehicles.

The Caret-HM framework allows Android developers to record and replay user sessions and convert them into heatmaps. One advantage of our framework over existing solutions is that it allows developers to control the environment while simplifying the recording process by giving users access to their applications via a web browser. The heatmap generation using Android user sessions and clustering UI states is a unique feature of our framework. Heat maps allow developers to identify the usage of application features for testing and guiding business decisions. We provide a qualitative comparison to the existing solutions. The video with demonstration is available at https://www.youtube.com/watch?v=eMSNAKM1Bj4

Network security is a growing area of interest for cyber systems, especially given the increasing number of attacks on companies each year. Though there are a vast amount of tools already available, System Identification (SI) complements intrusion detection systems to help manage network traffic stability. SI is the science of building mathematical models of dynamic systems. This paper introduces the use of SI for modeling network traffic and utilizes a linear time invariant model to analyze performance of real connections and attack instances. We generated several ARX models where each represented a different threat state in the network. We utilized the KDD CUP 1999's DARPA dataset to analyze the performance when dealing with different attacks. Results show that the average model fit was 84.14% when determining if the system was experiencing normal traffic. This value is promising because it shows how well the system is able to determine a network state in a given time when fed input.

Bug Catcher is a web-based system for running software testing competitions. While programming competitions are a way to engage students, they require students to have coding experience. On the other hand, software testing competitions may reach high school students that do not have access to a programming course. In this paper, we present the Bug Catcher system and the results from four sessions of a competition that include a total of 94 high school students. Bug Catcher provides students with requirements, buggy code, and input fields to enter test cases. We observed that most students began entering test cases based on requirements, but then many took an interest in the code as time went on. Our results show that 90% of students would recommend this activity in the future and 72% of students report that the activity increased their interest in Computer Science. Students also provided feedback on the system from the perspective of students without background in Computer Science, allowing us to create and modify features for future use.

Regression testing tasks of test case prioritization, test suite reduction/minimization, and regression test selection are typically centered around criteria that are based on code coverage, test execution costs, and code modifications. Researchers have developed and evaluated new individual criteria; others have combined existing criteria in different ways to form what we--and some others--call hybrid criteria. In this paper, we formalize the notion of combining multiple criteria into a hybrid. Our goal is to create a uniform representation of such combinations so that they can be described unambiguously and shared among researchers. We envision that such sharing will allow researchers to implement, study, extend, and evaluate the hybrids using a common set of techniques and tools. We precisely formulate three hybrid combinations, Rank, Merge, and Choice, and demonstrate their usefulness in two ways. First, we recast, in terms of our formulations, others' previously reported work on hybrid criteria. Second, we use our previous results on test case prioritization to create and evaluate new hybrid criteria. Our findings suggest that hybrid criteria of others can be described using our Merge and Rank formulations, and that the hybrid criteria we developed most often outperformed their constituent individual criteria.

Introductory Computer Science students often encounter programming bugs. Our previous work gathers and classifies data for 450 programming bugs brought to our tutor lab over a one year period. We use the data to identify the most common bugs as the basis for activities that improve our curriculum. This paper discusses several activities that rely on this data: (1) the “Mystery Bug Theater” website contains games and movies about common bugs, (2) professors quickly respond to the common bugs in lectures, and (3) class exercises use buggy code from the repository. Future work will distribute software testing material across the curriculum to help students with the most common bugs in specific courses and will analyze whether bug patterns change based on curriculum changes.

Test suite prioritization and reduction are two approaches to managing large test suites. They play an important role in regression testing, where a large number of tests accumulate over time from previous versions of the system. Accumulation of tests is exacerbated in user-session-based testing of web applications, where field usage data is continually logged and converted into test cases. This paper presents a tool that allows testers to easily collect, prioritize, and reduce user-session-based test cases. Our tool provides four contributions: (1) guidance to users on how to configure their web server to log important usage information, (2) automated parsing of web logs into XML formatted test cases that can be used by test replay tools, (3) automated prioritization of test cases by length-based and combinatorial-based criteria, and (4) automated reduction of test cases by combinatorial coverage.

Event-Driven Software (EDS) can change state based on incoming events; common examples are GUI and Web applications. These EDSs pose a challenge to testing because there are a large number of possible event sequences that users can invoke through a user interface. While valuable contributions have been made for testing these two subclasses of EDS, such efforts have been disjoint. This work provides the first single model that is generic enough to study GUI and Web applications together. In this paper, we use the model to define generic prioritization criteria that are applicable to both GUI and Web applications. Our ultimate goal is to evolve the model and use it to develop a unified theory of how all EDS should be tested. An empirical study reveals that the GUI and Web-based applications, when recast using the new model, show similar behavior. For example, a criterion that gives priority to all pairs of event interactions did well for GUI and Web applications; another criterion that gives priority to the smallest number of parameter value settings did poorly for both. These results reinforce our belief that these two subclasses of applications should be modeled and studied together.

Abstract Algorithmic construction of software interaction test suites has focussed on pairwise coverage; less is known about the efficient construction of test suites for t‐way interactions with t≥3. This study extends an efficient density‐based algorithm for pairwise coverage to generate t‐way interaction test suites and shows that it guarantees a logarithmic upper bound on the size of the test suites as a function of the number of factors. To complement this theoretical guarantee, an implementation is outlined and some practical improvements are made. Computational comparisons with other published methods are reported. Many of the results improve upon those in the literature. However, limitations on the ability of one‐test‐at‐a‐time algorithms are also identified. Copyright © 2008 John Wiley & Sons, Ltd.

Web applications have rapidly become a critical part of business for many organizations. However, increased usage of Web applications has not been reciprocated with corresponding increases in reliability. Unique characteristics, such as quick turnaround time, coupled with growing popularity motivate the need for efficient and effective Web application testing strategies. In this paper, we propose several new test suite prioritization strategies for Web applications and examine whether these strategies can improve the rate of fault detection for three Web applications and their preexisting test suites. We prioritize test suites by test lengths, frequency of appearance of request sequences, and systematic coverage of parameter-values and their interactions. Experimental results show that the proposed prioritization criteria often improve the rate of fault detection of the test suites when compared to random ordering of test cases. In general, the best prioritization metrics either (1) consider frequency of appearance of sequences of requests or (2) systematically cover combinations of parameter-values as early as possible.

Algorithms for the construction of software interaction test suites have focussed on the special case of pairwise coverage; less is known about efficiently constructing test suites for higher strength coverage. The combinatorial growth of t-tuples associated with higher strength hinders the efficacy of interaction testing. Test suites are inherently large, so testers may not run entire test suites. To address these problems, we combine a simple greedy algorithmallwith heuristic search to construct and dispense one test at a time. Our algorithm attempts to maximize the number of t-tuples covered by the earliest tests so that if a tester only runs a partial test suite, they test as many t-tuples as possible.allHeuristic search is shown to provide effective methods for achieving such coverage.

Abstract There are many published algorithms for generating interaction test suites for software testing, exemplified by AETG, IPO, TCG, TConfig, simulated annealing and other heuristic search, and combinatorial design techniques. Among these, greedy one‐test‐at‐a‐time methods (such as AETG and TCG) have proven to be a reasonable compromise between the needs for small test suites, fast test‐suite generation, and flexibility to accommodate a variety of testing scenarios. However, such methods suffer from the lack of a worst‐case logarithmic guarantee on test suite size, while methods that provide such a guarantee at present are less efficient or flexible, or do not produce test suites that are competitive in size for practical testing scenarios. In this paper, a new algorithm establishes that efficient, greedy, one‐test‐at‐a‐time methods can indeed produce a logarithmic worst‐case guarantee on the test suite size. In addition, this can be done while still producing test suites that are of competitive size, and in a time that is comparable to the published methods. It is deterministic, guaranteeing reproducibility. It generates only one candidate test at a time, permits users to ‘seed’ the test suite with specified tests, and allows users to specify constraints of combinations that should be avoided. Further, statistical analysis examines the impact of five variables used to tune this density algorithm for execution time and test suite size: weighting of density for factors, scaling of density, tie‐breaking, use of multiple candidates, and multiple repetitions using randomization. Copyright © 2007 John Wiley & Sons, Ltd.

Model-based software development is gaining interest in domains such as avionics, space, and automotives. The model serves as the central artifact for the development efforts (such as, code generation), therefore, it is crucial that the model be extensively validated. Automatic generation of interaction test suites is a candidate for partial automation of this model validation task. Interaction testing is a combinatorial approach that systematically tests all t-way combinations of inputs for a system. In this paper, we report how well interaction test suites (2-way through 5-way interaction test suites) structurally cover a model of the modelogic of a flight guidance system. We conducted experiments to (1) compare the coverage achieved with interaction test suites to that of randomly generated tests and (2) determine if interaction test suites improve the coverage of black-box test suites derived from system requirements. The experiments show that the interaction test suites provide little benefit over the randomly generated tests and do not improve coverage of the requirements-based tests. These findings raise questions on the application of interaction testing in this domain.

Greedy algorithms for the construction of software interaction test suites are studied. A framework is developed to evaluate a large class of greedy methods that build suites one test at a time. Within this framework are many instantiations of greedy methods generalizing those in the literature. Greedy algorithms are popular when the time for test suite construction is of paramount concern. We focus on the size of the test suite produced by each instantiation. Experiments are analyzed using statistical techniques to determine the importance of the implementation decisions within the framework. This framework provides a platform for optimizing the accuracy and speed of "one-test-at-a-time" greedy methods.

Combinatorial approaches to testing are used in several fields, and have recently gained momentum in the field of software testing through software interaction testing. One-test-at-a-time greedy algorithms are used to automatically construct such test suites. This paper discusses basic criteria of why greedy algorithms have been appropriate for this test gen-eration problem in the past and then expands upon how greedy algorithms can be utilized to address test suite pri-oritization.