Abstract Formal verification of transformers has become increasingly important due to their widespread deployment in safety-critical applications. Compared to classic neural networks, the inferences of transformers involve highly complex computations, such as dot products in self-attention layers, rendering their verification extremely difficult. Existing approaches explored over-approximation methods by constructing convex constraints to bound the output ranges of transformers, which can achieve high efficiency. However, they may sacrifice verification precision, and consequently introduce significant approximation error that leads to frequent occurrences of false alarms. In this paper, we propose a transformer verification approach that can achieve improved precision. At the core of our approach is a novel usage of ReLU, by which we represent a precise but non-linear bound for dot products such that we can further exploit the rich body of literature for convex relaxation of ReLU to derive precise bounds. We extend two classic approaches to the context of transformers, a rule-based one and an optimization-based one, resulting in two new frameworks for efficient and precise verification. We evaluate our approaches on different model architectures and robustness properties derived from two datasets about sentiment analysis, and compare with the state-of-the-art baseline approach. Compared to the baseline, our approach can achieve significant precision improvement for most of the verification tasks with acceptable compromise of efficiency, which demonstrates the effectiveness of our approach.

Yang Yuguang, Yu Pan, Jixun Yao, Xiang Zhang, Jianhao Ye, Hongbin Zhou, Lei Xie, Lei Ma, Jianjun Zhao. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

This paper introduces QuanUML, an extension of the Unified Modeling Language (UML) tailored for quantum software systems. QuanUML integrates quantum-specific constructs, such as qubits and quantum gates, into the UML framework, enabling the modeling of both quantum and hybrid quantum-classical systems. We apply QuanUML to Efficient Long-Range Entanglement using Dynamic Circuits and Shor’s Algorithm, demonstrating its utility in designing and visualizing quantum algorithms. Our approach supports model-driven development of quantum software and offers a structured framework for quantum software design. We also highlight its advantages over existing methods and discuss future improvements.

Formal verification is a rigorous approach that can provably ensure the quality of neural networks, and to date, Branch and Bound (BaB) is the state-of-the-art that performs verification by splitting the problem as needed and applying off-the-shelf verifiers to sub-problems for improved performance. However, existing BaB may not be efficient, due to its naive way of exploring the space of sub-problems that ignores the importance of different sub-problems. To bridge this gap, we first introduce a notion of “importance” that reflects how likely a counterexample can be found with a sub-problem, and then we devise a novel verification approach, called ABONN, that explores the sub-problem space of BaB adaptively, in a Monte-Carlo tree search (MCTS) style. The exploration is guided by the “importance” of different sub-problems, so it favors the sub-problems that are more likely to find counterexamples. As soon as it finds a counterexample, it can immediately terminate; even though it cannot find, after visiting all the sub-problems, it can still manage to verify the problem. We evaluate ABONN with 552 verification problems from commonlyused datasets and neural network models, and compare it with the state-of-the-art verifiers as baseline approaches. Experimental evaluation shows that ABONN demonstrates speedups of up to 15.2× on MNIST and 24.7× on CIFAR-10. We further study the influences of hyperparameters to the performance of ABONN, and the effectiveness of our adaptive tree exploration.

The vulnerability of neural networks to adversarial perturbations has necessitated formal verification techniques that can rigorously certify the quality of neural networks. As the state-of-the-art, branch-and-bound (BaB) is a "divide-and-conquer" strategy that applies off-the-shelf verifiers to sub-problems for which they perform better. While BaB can identify the sub-problems that are necessary to be split, it explores the space of these sub-problems in a naive "first-come-first-served" manner, thereby suffering from an issue of inefficiency to reach a verification conclusion. To bridge this gap, we introduce an order over different sub-problems produced by BaB, concerning with their different likelihoods of containing counterexamples. Based on this order, we propose a novel verification framework Oliva that explores the sub-problem space by prioritizing those sub-problems that are more likely to find counterexamples, in order to efficiently reach the conclusion of the verification. Even if no counterexample can be found in any sub-problem, it only changes the order of visiting different sub-problems and so will not lead to a performance degradation. Specifically, Oliva has two variants, including Oliva^GR, a greedy strategy that always prioritizes the sub-problems that are more likely to find counterexamples, and Oliva^SA, a balanced strategy inspired by simulated annealing that gradually shifts from exploration to exploitation to locate the globally optimal sub-problems. We experimentally evaluate the performance of Oliva on 690 verification problems spanning over 5 models with datasets MNIST and CIFAR-10. Compared to the state-of-the-art approaches, we demonstrate the speedup of Oliva for up to 25× in MNIST, and up to 80× in CIFAR-10.

Deep object tracking has made significant strides, demonstrating impressive accuracy across diverse visual scenarios. However, recent studies revealed that visual object trackers remain vulnerable to adversarial attacks specifically designed to disrupt tracking tasks. While image resampling—reconstructing images using resampled coordinates and bilinear interpolation—has shown promise in enhancing tracker robustness by disrupting adversarial patterns, this naive approach overlooks crucial semantic information and scale variations between frames relative to the initial object template. These variations typically arise from changes in the target object or background. To address this limitation, we propose a template-guided semantic resampling (TGSR) method to counter adversarial tracking attacks. Our approach comprises two key components: template-aware joint semantic and appearance implicit representation (T-SAIR) and template-aware predictive resampling (T-PRES). T-SAIR estimates semantic embeddings and corresponding pixel colors at arbitrary coordinates based on incoming frames and the object template, while T-PRES predicts pixel-wise coordinate shifts in response to scale changes relative to the template. The integration of these modules enables our method to effectively reconstruct frames, neutralizing adversarial perturbations while preserving semantic information relative to the object template. Extensive experimental evaluation against three attacks across typical tracking methods demonstrates the effectiveness of our approach.

With the growing interest in quantum computing, the emergence of quantum supremacy has marked a pivotal milestone in the field. As a result, numerous quantum programming languages (QPLs) have been introduced to support the development of quantum algorithms. However, the application of Model-Driven Development (MDD) in quantum system engineering remains largely underexplored. This paper presents an MDD-based approach to support the structured design and implementation of quantum systems. Our framework enables the automatic generation of quantum code for multiple QPLs, thereby enhancing development efficiency and consistency across heterogeneous quantum platforms. The effectiveness and practicality of our approach have been demonstrated through multiple case studies.

Large language models (LLMs) have demonstrated good performance in general code generation; however, their capabilities in quantum code generation remain insufficiently studied. This paper presents QuanBench, a benchmark for evaluating LLMs on quantum code generation. QuanBench includes 44 programming tasks that cover quantum algorithms, state preparation, gate decomposition, and quantum machine learning. Each task has an executable canonical solution and is evaluated by functional correctness (Pass@K) and quantum semantic equivalence (Process Fidelity). We evaluate several recent LLMs, including general-purpose and code-specialized models. The results show that current LLMs have limited capability in generating the correct quantum code, with overall accuracy below 40% and frequent semantic errors. We also analyze common failure cases, such as outdated API usage, circuit construction errors, and incorrect algorithm logic. QuanBench provides a basis for future work on improving quantum code generation with LLMs.

Quantum programs are designed to run on quantum computers, leveraging quantum circuits to solve problems that are intractable for classical machines. As quantum computing advances, ensuring the reliability of quantum programs has become increasingly important. This paper introduces NovaQ, a diversity-guided testing framework for quantum programs. NovaQ combines a distribution-based test case generator with a novelty-driven evaluation module. The generator produces diverse quantum state inputs by mutating circuit parameters, while the evaluator quantifies behavioral novelty based on internal circuit state metrics, including magnitude, phase, and entanglement. By selecting inputs that map to infrequently covered regions in the metric space, NovaQ effectively explores under-tested program behaviors. We evaluate NovaQ on quantum programs of varying sizes and complexities. Experimental results show that NovaQ consistently achieves higher test input diversity and detects more bugs than existing baseline approaches.

As quantum computing continues to emerge, ensuring the quality of quantum programs has become increasingly critical. Quantum program testing has emerged as a prominent research area within the scope of quantum software engineering. While numerous approaches have been proposed to address quantum program quality assurance, our analysis reveals that most existing methods rely on measurement-based validation in practice. However, due to the inherently probabilistic nature of quantum programs, measurement-based validation methods face significant limitations.To investigate these limitations, we conducted an empirical study of recent research on quantum program testing, analyzing measurement-based validation methods in the literature. Our analysis categorizes existing measurement-based validation methods into two groups: distribution-level validation and output-value-level validation. We then compare measurement-based validation with statevector-based validation methods to evaluate their pros and cons. Our findings demonstrate that measurement-based validation is suitable for straightforward assessments, such as verifying the existence of specific output values, while statevector-based validation proves more effective for complicated tasks such as assessing the program behaviors.

Incremental verification is an emerging neural network verification approach that aims to accelerate the verification of a neural network N* by reusing the existing verification result (called a template ) of a similar neural network N . To date, the state‐of‐the‐art incremental verification approach leverages the problem splitting history produced by branch and bound ( BaB ) in verification of N , to select only a part of the sub‐problems for verification of N* , thus more efficient than verifying N* from scratch. While this approach identifies whether each sub‐problem should be re‐assessed, it neglects the information of how necessary each sub‐problem should be re‐assessed, in the sense that the sub‐problems that are more likely to contain counterexamples should be prioritized, in order to terminate the verification process as soon as a counterexample is detected. To bridge this gap, we first define a counterexample potentiality order over different sub‐problems based on the template, and then we propose Olive, an incremental verification approach that explores the sub‐problems of verifying N* orderly guided by counterexample potentiality. Specifically, Olive has two variants, including Olive g , a greedy strategy that always prefers to exploit the sub‐problems that are more likely to contain counterexamples, and Olive b , a balanced strategy that also explores the sub‐problems that are less likely, in case the template is not sufficiently precise. We experimentally evaluate the efficiency of Olive on 1445 verification problem instances derived from 15 neural networks spanning over two datasets MNIST and CIFAR‐10 . Our evaluation demonstrates significant performance advantages of Olive over state‐of‐the‐art classic verification and incremental approaches. In particular, Olive shows evident superiority on the problem instances that contain counterexamples, and performs as well as Ivan on the certified problem instances.

Falsification, whose aim is to detect unsafe behaviors of cyber-physical systems (CPS) that violate signal temporal logic (STL) specifications, has been actively investigated in the past decade. Although numerous falsification approaches have been proposed, the falsification community suffers from a shortage of benchmarks that hinders a thorough assessment of those falsification approaches. In this article, we bridge this gap by proposing an automated approach for generating falsification benchmarks. Our approach is data-driven: first, we generate different time-variant traces (acting as system output traces) that satisfy a given STL specification, and we associate these with corresponding system input traces; then, we use these input and output traces to train an LSTM model that generalizes them. These models can serve as benchmarks for assessing falsification approaches against the given specification. In the experimental evaluation, we validate the generated models by measuring their ability to differentiate the performance of different falsification approaches. Our generated models expose strengths and weaknesses of all the considered falsification approaches, which was not achieved by benchmarks currently used in the falsification community. These results demonstrate the usefulness of our approach and can potentially push forward subsequent research in falsification.

Cyber-physical systems (CPSs) are increasingly adopting deep neural networks (DNNs) as controllers, giving birth to AI-enabled CPSs . Despite their advantages, many concerns arise about the safety of DNN controllers. Numerous efforts have been made to detect system executions that violate safety specifications; however, once a violation is detected, to fix the issue, it is necessary to localise the parameters of the DNN controller responsible for the wrong decisions leading to the violation. This is particularly challenging, as it requires to consider a sequence of control decisions, rather than a single one, preceding the violation. To tackle this problem, we propose SpectAcle , that can localise the faulty parameters in DNN controllers. SpectAcle considers the DNN inferences preceding the specification violation and uses forward impact to determine the DNN parameters that are more relevant to the DNN outputs. Then, it identifies which of these parameters are responsible for the specification violation, by adapting classic suspiciousness metrics. Moreover, we propose two versions of SpectAcle , that consider differently the timestamps that precede the specification violation. We experimentally evaluate the effectiveness of SpectAcle on 6,067 faulty benchmarks, spanning over different application domains. The results show that SpectAcle can detect most of the faults.

As quantum computers advance, the complexity of the software they can execute increases as well. To ensure this software is efficient, maintainable, reusable, and cost-effective—key qualities of any industry-grade software—mature software engineering practices must be applied throughout its design, development, and operation. However, the significant differences between classical and quantum software make it challenging to directly apply classical software engineering methods to quantum systems. This challenge has led to the emergence of Quantum Software Engineering (QSE) as a distinct field within the broader software engineering landscape. In this work, a group of active researchers analyze in depth the current state of QSE research. From this analysis, the key areas of QSE are identified and explored in order to determine the most relevant open challenges that should be addressed in the next years. These challenges help identify necessary breakthroughs and future research directions for advancing QSE.

In AI-enabled CPSs, DNNs are used as controllers for the physical system. Despite their advantages, DNN controllers can produce wrong control decisions, which can lead to safety risks for the system. Once wrong behaviors are detected, the DNN controller should be fixed. DNN repair is a technique that allows to perform this fine-grained improvement. However, state-of-the-art DNN repair techniques require ground-truth labels to guide the repair. For AI-enabled CPSs, these are not available, as it is not possible to assess whether a specific control decision is correct. Nevertheless, it is possible to assess whether the DNN controller leads to wrong behaviors of the controlled system by considering system-level requirements. In this paper, following this observation, we propose a novel DNN repair approach that is guided by system-level specifications. The approach takes in input a system-level specification, some tests violating the specification, and some faulty DNN weights. The approach searches for alternative weight values with the goal of fixing the behavior on the failing tests without breaking the passing tests. We also propose a heuristic that allows us to accelerate the search by avoiding the execution of some tests. Experiments on real-world AI-enabled CPSs show that the approach effectively repairs their controllers.

Contrastive cross-modality pretraining has recently exhibited impressive success in diverse fields, whereas there is limited research on their merits in speech emotion recognition (SER). In this paper, we propose GEmo-CLAP, a kind of gender-attribute-enhanced contrastive language-audio pretraining (CLAP) method for SER. Specifically, we first construct an effective emotion CLAP (Emo-CLAP) for SER, using pretrained text and audio encoders. Second, given the significance of gender information in SER, two novel multi-task learning based GEmo-CLAP (ML-GEmo-CLAP) and soft label based GEmo-CLAP (SL-GEmo-CLAP) models are further proposed to incorporate gender information of speech signals, forming more reasonable objectives. Experiments on IEMOCAP indicate that our proposed two GEmo-CLAPs consistently outperform Emo-CLAP with different pre-trained models. Remarkably, the proposed WavLM-based SL-GEmo-CLAP obtains the best WAR of 83.16%, which performs better than state-of-the-art SER methods.

Visual object tracking plays a critical role in visual-based autonomous systems, as it aims to estimate the position and size of the object of interest within a live video. Despite significant progress made in this field, state-of-the-art (SOTA) trackers often fail when faced with adversarial perturbations in the incoming frames. This can lead to significant robustness and security issues when these trackers are deployed in the real world. To achieve high accuracy on both clean and adversarial data, we propose building a spatial-temporal continuous representation using the semantic text guidance of the object of interest. This novel continuous representation enables us to reconstruct incoming frames to maintain semantic and appearance consistency with the object of interest and its clean counterparts. As a result, our proposed method successfully defends against different SOTA adversarial tracking attacks while maintaining high accuracy on clean data. In particular, our method significantly increases tracking accuracy under adversarial attacks with around 90% relative improvement on UAV123, which is even higher than the accuracy on clean data.

Deep learning (DL) compilers, such as TVM and TensorFlow, encompass a variety of passes for optimizing computation graphs (i.e., DL models). Despite the efforts on developing optimization passes, it remains a challenge in arranging these passes - most compilers employ fixed pass sequences that do not fit with computation graphs of diverse structures; on the other hand, optimization passes have cascade effects, making the structures of graphs under compilation volatile and as well making it difficult to generate optimal sequences for graphs. Inspired by recent progresses on static computing memory footprints (i.e., memory usages) of computation graphs, we introduce in this paper OPASS, a novel approach to orchestrating TVM's optimization passes for lowering memory footprints of computation graphs, and finally allowing the graphs to run on memory-constrained devices. The key idea is, given a computation graph$G$, to optimize the graph heuristically and iteratively: OPASS learns the effects of passes on the graph; it then optimizes$G$iteratively - each iteration picks up a pass by the reduction of the memory footprint of$G$and as well the implicit effects of the pass for further optimizations, letting the pass be applied. We evaluate OPASS on Rebench (a suite of computation graphs) and two real-world models (Transformer and ResNet). The results clearly show the strength of OPASS: it outperforms TVM's default sequence by$1.77\times$in reducing graphs' memory footprints, with affordable costs; it also offers extra memory reductions of$5\sim 12\%$by catching the implicit effects of passes. Furthermore, OPASS helps analyze positive/negative effects of passes to graphs' memory footprints, providing TVM developers with best practices for designing optimization pass sequences.

The continuous evolution of pre-trained speech models has greatly advanced Speech Emotion Recognition (SER). However, current research typically relies on utterance-level emotion labels, inadequately capturing the complexity of emotions within a single utterance. In this paper, we introduce GMP-TL, a novel SER framework that employs gender-augmented multi-scale pseudo-label (GMP) based transfer learning to mitigate this gap. Specifically, GMPTL initially uses the pre-trained HuBERT, implementing multi-task learning and multi-scale k -means clustering to acquire frame-level GMPs. Subsequently, to fully leverage frame-level GMPs and utterance-level emotion labels, a two-stage model fine-tuning approach is presented to further optimize GMP-TL. Experiments on IEMOCAP show that our GMP-TL attains a WAR of 80.0% and an UAR of 82.0%, achieving superior performance compared to state-of-the-art unimodal SER methods while also yielding comparable results to multimodal SER approaches.

Smart contract transactions are increasingly interleaved by cross-contract calls. While many tools have been developed to identify a common set of vulnerabilities, the cross-contract vulnerability is overlooked by existing tools. Cross-contract vulnerabilities are exploitable bugs that manifest in the presence of more than two interacting contracts. Existing methods are however limited to analyze a maximum of two contracts at the same time. Detecting cross-contract vulnerabilities is highly non-trivial. With multiple interacting contracts, the search space is much larger than that of a single contract. To address this problem, we presentxFuzz, a machine learning guided smart contract fuzzing framework. The machine learning models are trained with novel features (e.g., word vectors and instructions) and are used to filter likely benign program paths. Comparing with existing static tools, machine learning model is proven to be more robust, avoiding directly adopting manually-defined rules in specific tools. We comparexFuzzwith three state-of-the-art tools on 7,391 contracts.xFuzzdetects 18 exploitable cross-contract vulnerabilities, of which 15 vulnerabilities are exposed for the first time. Furthermore, our approach is shown to be efficient in detecting non-cross-contract vulnerabilities as well—using less than 20% time as that of other fuzzing tools,xFuzzdetects twice as many vulnerabilities.

Quantum computing has emerged as a promising field with the potential to revolutionize various domains by harnessing the principles of quantum mechanics. As quantum hardware and algorithms continue to advance, developing high-quality quantum software has become crucial. However, testing quantum programs poses unique challenges due to the distinctive characteristics of quantum systems and the complexity of multi-subroutine programs. This article addresses the specific testing requirements of multi-subroutine quantum programs. We begin by investigating critical properties by surveying existing quantum libraries and providing insights into the challenges of testing these programs. Building upon this understanding, we focus on testing criteria and techniques based on the whole testing process perspective, spanning from unit testing to integration testing. We delve into various aspects, including IO analysis, quantum relation checking, structural testing, behavior testing, integration of subroutine pairs, and test case generation. We also introduce novel testing principles and criteria to guide the testing process. We conduct comprehensive testing on typical quantum subroutines, including diverse mutants and randomized inputs, to evaluate our proposed approach. The analysis of failures provides valuable insights into the effectiveness of our testing methodology. Additionally, we present case studies on representative multi-subroutine quantum programs, demonstrating the practical application and effectiveness of our proposed testing principles and criteria.

Many existing static analysis algorithms suffer from cubic bottlenecks because of the need to compute a dynamic transitive closure (DTC). For the first time, this article studies the quantum speedups on searching subtasks in DTC-based static analysis algorithms using quantum search (e.g., Grover’s algorithm). We first introduce our oracle implementation in Grover’s algorithm for DTC-based static analysis and illustrate our quantum search subroutine. Then, we take two typical DTC-based analysis algorithms: context-free-language reachability and set constraint-based analysis, and show that our quantum approach can reduce the time complexity of these two algorithms to truly subcubic ( \(O(N^2\sqrt {N}{\it polylog}(N))\) ), yielding better results than the upper bound ( O ( N 3 /log N )) of existing classical algorithms. Finally, we conducted a classical simulation of Grover’s search to validate our theoretical approach, due to the current quantum hardware limitation of lacking a practical, large-scale, noise-free quantum machine. We evaluated the correctness and efficiency of our approach using IBM Qiskit on nine open-source projects and randomly generated edge-labeled graphs/constraints. The results demonstrate the effectiveness of our approach and shed light on the promising direction of applying quantum algorithms to address the general challenges in static analysis.

In recent years, as various realistic face forgery techniques known as DeepFake improves by leaps and bounds, more and more DeepFake detection techniques have been proposed. These methods typically rely on detecting statistical differences between natural (i.e., real) and DeepFake-generated images in both spatial and frequency domains. In this work, we propose to explicitly minimize the statistical differences to evade state-of-the-art DeepFake detectors. To this end, we propose a statistical consistency attack (StatAttack) against DeepFake detectors, which contains two main parts. First, we select several statistical-sensitive natural degradations (i.e., exposure, blur, and noise) and add them to the fake images in an adversarial way. Second, we find that the statistical differences between natural and DeepFake images are positively associated with the distribution shifting between the two kinds of images, and we propose to use a distribution-aware loss to guide the optimization of different degradations. As a result, the feature distributions of generated adversarial examples is close to the natural images. Furthermore, we extend the StatAttack to a more powerful version, MStatAttack, where we extend the single-layer degradation to multi-layer degradations sequentially and use the loss to tune the combination weights jointly. Comprehensive experimental results on four spatial-based detectors and two frequency-based detectors with four datasets demonstrate the effectiveness of our proposed attack method in both white-box and black-box settings.

TVM is a popular deep learning (DL) compiler. It is designed for compiling DL models, which are naturally computation graphs, and as well promoting the efficiency of DL computation. State-of-the-art methods, such as Muffin and NNSmith, allow developers to generate computation graphs for testing DL compilers. However, these techniques are inefficient — their generated computation graphs are either type-invalid or inexpressive, and hence not able to test the core functionalities of a DL compiler.

The reliability of decision-making policies is urgently important today as they have established the fundamentals of many critical applications, such as autonomous driving and robotics. To ensure reliability, there have been a number of research efforts on testing decision-making policies that solve Markov decision processes (MDPs). However, due to the deep neural network (DNN)-based inherit and infinite state space, developing scalable and effective testing frameworks for decision-making policies still remains open and challenging. In this paper, we present an effective testing framework for decision-making policies. The framework adopts a generative diffusion model-based test case generator that can easily adapt to different search spaces, ensuring the practicality and validity of test cases. Then, we propose a termination state novelty-based guidance to diversify agent behaviors and improve the test effectiveness. Finally, we evaluate the framework on five widely used benchmarks, including autonomous driving, aircraft collision avoidance, and gaming scenarios. The results demonstrate that our approach identifies more diverse and influential failure-triggering test cases compared to current state-of-the-art techniques. Moreover, we employ the detected failure cases to repair the evaluated models, achieving better robustness enhancement compared to the baseline method.

Quantum Intermediate Representation (QIR) is an LLVM-based intermediate representation developed by Microsoft for quantum program compilers. QIR is designed to offer a universal solution for quantum program compilers, decoupled from both front-end languages and back-end hardware, thereby eliminating the need for redundant development of intermediate representations and compilers. However, the lack of a formal definition and reliance on natural language descriptions in the current state of QIR result in interpretational ambiguity and a dearth of rigor in implementing quantum functions. In this paper, we present formal definitions for QIR's data types and instruction sets to establish correctness and safety assurances for operations and intermediate code conversions within QIR. To demonstrate the effectiveness of our approach, we provide examples of unsafe QIR codes where errors can be identified with our method.

The recent fast development of quantum computers breaks several computation limitations that are difficult for conventional computers. Up to the present, although many approaches and tools have been proposed to test quantum programs, the fundamental features of quantum programs, i.e., magnitude, phase, and entanglement, have been largely overlooked, leading to limited fault detection capability and reduced testing effectiveness. To address this problem, we propose an automated testing framework named QURATEST, equipped with three test case generators (including two newly proposed techniques, UCNOT and IQFT in this paper, as well as one based on Random techniques) to test quantum programs. Overall, the proposed generators enable the generation of diverse test inputs by considering the quantum features of quantum programs. In the experiments, we perform an in-depth evaluation of QURATEST from three aspects: generated test case diversity, output coverage of the program under test, and fault detection capability. The results demonstrate the potential of our newly proposed techniques in that IQFT can generate the most diverse test cases regarding magnitude, phase, and entanglement, with 66% cell coverage. Comparatively, the Random approach only has 10% cell coverage. Regarding the evaluations of the output coverage, IQFT can achieve the highest output coverage in 70.2% (33 out of 47) of all quantum programs. In terms of fault detection, UCNOT outperforms the other two techniques. Specifically, the test cases generated by UCNOT have the best mutation score in 88.4% (23 out of 26) quantum programs.

Deep neural networks (DNNs) achieved a significant performance breakthrough over the past decade and have been widely adopted in various industrial domains. However, a fundamental problem regarding DNN robustness is still not adequately addressed, which can potentially lead to many quality issues after deployment, e.g., safety, security, and reliability. An adversarial attack is one of the most commonly investigated techniques to penetrate a DNN by misleading the DNN’s decision through the generation of minor perturbations in the original inputs. More importantly, the adversarial attack is a crucial way to assess, estimate, and understand the robustness boundary of a DNN. Intuitively, a stronger adversarial attack can help obtain a tighter robustness boundary, allowing us to understand the potential worst-case scenario when a DNN is deployed. To push this further, in this paper, we propose DeepRover, a fuzzing-based blackbox attack for deep neural networks used for image classification. We show that DeepRover is more effective and query-efficient in generating adversarial examples than state-of-the-art blackbox attacks. Moreover, DeepRover can find adversarial examples at a finer-grained level than other approaches.

Inspired by the great success of Deep Neural Networks (DNNs) in natural language processing (NLP), DNNs have been increasingly applied in source code analysis and attracted significant attention from the software engineering community. Due to its data-driven nature, a DNN model requires massive and high-quality labeled training data to achieve expert-level performance. Collecting such data is often not hard, but the labeling process is notoriously laborious. The task of DNN-based code analysis even worsens the situation because source code labeling also demands sophisticated expertise. Data augmentation has been a popular approach to supplement training data in domains such as computer vision and NLP. However, existing data augmentation approaches in code analysis adopt simple methods, such as data transformation and adversarial example generation, thus bringing limited performance superiority. In this paper, we propose a data augmentation approach MixCode that aims to effectively supplement valid training data, inspired by the recent advance named Mixup in computer vision. Specifically, we first utilize multiple code refactoring methods to generate transformed code that holds consistent labels with the original data. Then, we adapt the Mixup technique to mix the original code with the transformed code to augment the training data. We evaluate MixCode on two programming languages (Java and Python), two code tasks (problem classification and bug detection), four benchmark datasets (JAVA250, Python800, CodRep1, and Refactory), and seven model architectures (including two pretrained models CodeBERT and GraphCodeBERT). Experimental results demonstrate that MixCode outperforms the baseline data augmentation approach by up to 6.24% in accuracy and 26.06% in robustness.

Visual object tracking (VOT) is a fundamental computer vision task that aims to track a target in a sequence of video frames. It has been broadly adopted in safety- and security-critical applications, such as self-driving systems and traffic control systems . However, the VOT models (i.e., the trackers) that rely on third-party training resources face a severe threat of backdoor attacks , which refer to the type of the attacks that poison a portion of training data and mislead the tracker to track a wrong target. A surge of research interest has arisen in backdoor attacks in the domain of image classification , as a measure to expose the potential security risks of the classifiers and inspire new defense techniques. Despite the prosperity of the research in backdoor attacks in image classification , there still lacks investigation in backdoor attacks against VOT, due to their unique challenges: first, the architecture of a VOT model is much more complicated than that of an image classifier; second, VOT targets a sequence of video frames rather than individual images. To bridge the gap, we propose a novel and effective targeted backdoor attack approach TAT specifically against VOT tasks. In particular, TAT includes a basic version TAT-BA that can achieve effective and stealthy backdoor attacks against VOT trackers, and an advanced version TAT-DA that can evade two representative defense techniques. Our large-scale experimental evaluation demonstrates the effectiveness and the stealthiness of TAT . Moreover, we also demonstrate the performances of TAT-BA under real-world settings and the abilities of TAT-DA to counter defense techniques. The code will be available at https://github.com/MisakaZipi/TAT .

Over the past few years, deep neural networks (DNNs) have achieved tremendous success and have been continuously applied in many application domains. However, during the practical deployment in industrial tasks, DNNs are found to be erroneous-prone due to various reasons such as overfitting and lacking of robustness to real-world corruptions during practical usage. To address these challenges, many recent attempts have been made to repair DNNs for version updates under practical operational contexts by updating weights (i.e., network parameters) through retraining, fine-tuning, or direct weight fixing at a neural level. Nevertheless, existing solutions often neglect the effects of neural network architecture and weight relationships across neurons and layers. In this work, as the first attempt, we initiate to repair DNNs by jointly optimizing the architecture and weights at a higher (i.e., block level). We first perform empirical studies to investigate the limitation of whole network-level and layer-level repairing, which motivates us to explore a novel repairing direction for DNN repair at the block level. To this end, we need to further consider techniques to address two key technical challenges, i.e., block localization , where we should localize the targeted block that we need to fix; and how to perform joint architecture and weight repairing . Specifically, we first propose adversarial-aware spectrum analysis for vulnerable block localization that considers the neurons’ status and weights’ gradients in blocks during the forward and backward processes, which enables more accurate candidate block localization for repairing even under a few examples. Then, we further propose the architecture-oriented search-based repairing that relaxes the targeted block to a continuous repairing search space at higher deep feature levels. By jointly optimizing the architecture and weights in that space, we can identify a much better block architecture. We implement our proposed repairing techniques as a tool, named ArchRepair , and conduct extensive experiments to validate the proposed method. The results show that our method can not only repair but also enhance accuracy and robustness, outperforming the state-of-the-art DNN repair techniques.

Modern Cyber-Physical Systems (CPSs) that need to perform complex control tasks (e.g., autonomous driving) are increasingly using AI-enabled controllers, mainly based on deep neural networks (DNNs). The quality assurance of such types of systems is of vital importance. However, their verification can be extremely challenging, due to their complexity and uninterpretable decision logic. Falsification is an established approach for CPS quality assurance, which, instead of attempting to prove the system correctness, aims at finding a time-variant input signal violating a formal specification describing the desired behavior; it often employs a search-based testing approach that tries to minimize therobustnessof the specification, given by its quantitative semantics. However, guidance provided by robustness is mostly black-box and only related to the system output, but does not allow to understand whether the temporal internal behavior determined by multiple consecutive executions of the neural network controller has been explored sufficiently. To bridge this gap, in this paper, we make an early attempt at exploring the temporal behavior determined by the repeated executions of the neural network controllers in hybrid control systems and first propose eight time-aware coverage criteria specifically designed for neural network controllers in the context of CPS, which consider different features by design: the simple temporal activation of a neuron, the continuous activation of a neuron for a given duration, and the differential neuron activation behavior over time. Second, we introduce a falsification framework, named$\mathtt {FalsifAI}$, that exploits the coverage information for better falsification guidance. Namely, inputs of the controller that increase the coverage (so improving theexplorationof the DNN behaviors), are prioritized in theexploitationphase of robustness minimization. Our large-scale evaluation over a total of 3 typical CPS tasks, 6 system specifications, 18 DNN models and more than 12,000 experiment runs, demonstrates 1) the advantage of our proposed technique in outperforming two state-of-the-art falsification approaches, and 2) the usefulness of our proposed time-aware coverage criteria for effective falsification guidance.

While the ability to build quantum computers is improving dramatically, developing quantum algorithms is very limited and relies on human insight and ingenuity. Although several quantum programming languages have been developed, it is challenging for software developers unfamiliar with quantum computing to learn and use these languages. It is, therefore, necessary to develop tools to support developing new quantum algorithms and programs automatically. This paper proposes AutoQC, an approach to automatically synthesizing quantum circuits using the neural network from input and output pairs. We consider a quantum circuit a sequence of quantum gates and synthesize a quantum circuit probabilistically by prioritizing through a neural network at each step. The experimental results highlight the ability of AutoQC to synthesize some essential quantum circuits at a lower cost.

As quantum programming evolves, more and more quantum programming languages are being developed. As a result, debugging and testing quantum programs have become increasingly important. While bug fixing in classical programs has come a long way, there is a lack of research in quantum programs. To this end, this paper presents a comprehensive study on bug fixing in quantum programs. We collect and investigate 96 real-world bugs and their fixes from four popular quantum programming languages (Qiskit, Cirq, Q#, and ProjectQ). Our study shows that a high proportion of bugs in quantum programs are quantum-specific bugs (over 80%), which requires further research in the bug fixing domain. We also summarize and extend the bug patterns in quantum programs and subdivide the most critical part, math-related bugs, to make it more applicable to the study of quantum programs. Our findings summarize the characteristics of bugs in quantum programs and provide a basis for studying testing and debugging quantum programs.

Deep neural networks (DNNs) are continuously expanding their application to various domains due to their high performance. Nevertheless, a well-trained DNN after deployment could oftentimes raise errors during practical use in the operational environment due to the mismatching between distributions of the training dataset and the potential unknown noise factors in the operational environment, e.g., weather, blur, noise, etc. Hence, it poses a rather important problem for the DNNs’ real-world applications: how to repair the deployed DNNs for correcting the failure samples under the deployed operational environment while not harming their capability of handling normal or clean data with limited failure samples we can collect. In this article, we propose astyle-guided data augmentation for repairing DNN in the operational environment, which learns and introduces the unknown failure patterns within the failure samples into the training data via the style transfer. Moreover, we further propose theclustering-based failure data generationfor much more effective style-guided data augmentation. We conduct a large-scale evaluation with 15 degradation factors that may happen in the real world and compare with four state-of-the-art data augmentation methods and two DNN repairing methods. Our technique successfully repairs three convolutional neural networks and two recurrent neural networks with averaging 62.88% and 39.02% accuracy enhancements on the 15 failure patterns, respectively, achieving higher repairing performance than state-of-the-art repairing methods on the most failure patterns with even better accuracy on clean datasets.

Single-image deraining is rather challenging due to the unknown rain model. Existing methods often make specific assumptions of the rain model, which can hardly cover many diverse circumstances in the real world, compelling them to employ complex optimization or progressive refinement. This, however, significantly affects these methods' efficiency and effectiveness for many efficiency-critical applications. To fill this gap, in this paper, we regard the single-image deraining as a general image-enhancing problem and originally propose a model-free deraining method, i.e., EfficientDeRain, which is able to process a rainy image within 10 ms (i.e., around 6 ms on average), over 80 times faster than the state-of-the-art method (i.e., RCDNet), while achieving similar de-rain effects. We first propose novel pixel-wise dilation filtering. In particular, a rainy image is filtered with the pixel-wise kernels estimated from a kernel prediction network, by which suitable multi-scale kernels for each pixel can be efficiently predicted. Then, to eliminate the gap between synthetic and real data, we further propose an effective data augmentation method (i.e., RainMix) that helps to train the network for handling real rainy images. We perform a comprehensive evaluation on both synthetic and real-world rainy datasets to demonstrate the effectiveness and efficiency of our method. We release the model and code in https://github.com/tsingqguo/efficientderain.git.

Abstract Hybrid system falsification is an important quality assurance method for cyber-physical systems with the advantage of scalability and feasibility in practice than exhaustive verification. Falsification, given a desired temporal specification, tries to find an input of violation instead of a proof guarantee. The state-of-the-art falsification approaches often employ stochastic hill-climbing optimization that minimizes the degree of satisfaction of the temporal specification, given by its quantitativerobust semantics. However, it has been shown that the performance of falsification could be severely affected by the so-calledscale problem, related to the different scales of the signals used in the specification (e.g., rpm and speed): in the robustness computation, the contribution of a signal could bemaskedby another one. In this paper, we propose a novel approach to tackle this problem. We first introduce a new robustness definition, calledQB-Robustness, which combines classical Boolean satisfaction and quantitative robustness. We prove that QB-Robustness can be used to judge the satisfaction of the specification and avoid the scale problem in its computation. QB-Robustness is exploited by a falsification approach based on Monte Carlo Tree Search over the structure of the formal specification. First, tree traversal identifies the sub-formulas for which it is needed to compute the quantitative robustness. Then, on the leaves, numerical hill-climbing optimization is performed, aiming to falsify such sub-formulas. Our in-depth evaluation on multiple benchmarks demonstrates that our approach achieves better falsification results than the state-of-the-art falsification approaches guided by the classical quantitative robustness, and it is largely not affected by the scale problem.

Motion blur caused by the moving of the object or camera during the exposure can be a key challenge for visual object tracking, affecting tracking accuracy significantly. In this work, we explore the robustness of visual object trackers against motion blur from a new angle, i.e., adversarial blur attack (ABA). Our main objective is to online transfer input frames to their natural motion-blurred counterparts while misleading the state-of-the-art trackers during the tracking process. To this end, we first design the motion blur synthesizing method for visual tracking based on the generation principle of motion blur, considering the motion information and the light accumulation process. With this synthetic method, we propose optimization-based ABA (OP-ABA) by iteratively optimizing an adversarial objective function against the tracking w.r.t. the motion and light accumulation parameters. The OP-ABA is able to produce natural adversarial examples but the iteration can cause heavy time cost, making it unsuitable for attacking real-time trackers. To alleviate this issue, we further propose one-step ABA (OS-ABA) where we design and train a joint adversarial motion and accumulation predictive network (JAMANet) with the guidance of OP-ABA, which is able to efficiently estimate the adversarial motion and accumulation parameters in a one-step way. The experiments on four popular datasets (e.g., OTB100, VOT2018, UAV123, and LaSOT) demonstrate that our methods are able to cause significant accuracy drops on four state-of-the-art trackers with high transferability. Please find the source code at https://github.com/tsingqguo/ABA

Online updating of the object model via samples from historical frames is of great importance for accurate visual object tracking. Recent works mainly focus on constructing effective and efficient updating methods while neglecting the training samples for learning discriminative object models, which is also a key part of a learning problem. In this paper, we propose the DeepMix that takes historical samples’ embeddings as input and generates augmented embeddings online, enhancing the state-of-the-art online learning methods for visual object tracking. More specifically, we first propose the online data augmentation for tracking that online augments the historical samples through object-aware filtering. Then, we propose MixNet which is an offline trained network for performing online data augmentation within one-step, enhancing the tracking accuracy while preserving high speeds of the state-of-the-art online learning methods. The extensive experiments on three different tracking frameworks, i.e., DiMP, DSiam, and SiamRPN++, and three large-scale and challenging datasets, i.e., OTB-2015, LaSOT, and VOT, demonstrate the effectiveness and advantages of the proposed method.

The highly diversified deep learning (DL) frame-works and target hardware architectures bring big challenges for DL model deployment for industrial production. Up to the present, continuous efforts have been made to develop DL compilers with multiple state-of-the-arts available, e.g., TVM, Glow, nGraph, PlaidML, and Tensor Comprehensions (TC). Unlike traditional compilers, DL compilers take a DL model built by DL frameworks as input and generate optimized code as the output for a particular target device. Similar to other software, DL compilers are also error-prone. Buggy DL compilers can generate incorrect code and result in unexpected model behaviors. To better understand the current status and common bug characteristics of DL compilers, we performed a large-scale empirical study of five popular DL compilers covering TVM, Glow, nGraph, PlaidML, and TC, collecting a total of 2,717 actual bug reports submitted by users and developers. We made large manual efforts to investigate these bug reports and classified them based on their root causes, during which five root causes were identified, including environment, compatibility, memory, document, and semantic. After labeling the types of bugs, we further examined the important consequences of each type of bug and analyzed the correlation between bug types and impacts. Besides, we studied the time required to fix different types of bugs in DL compilers. Seven important findings are eventually obtained, with practical implications provided for both DL compiler developers and users.

Realistic benchmarks of reproducible bugs and fixes are vital to good experimental evaluation of debugging and testing approaches. However, there is no suitable benchmark suite that can systematically evaluate the debugging and testing methods of quantum programs until now. This paper proposes Bugs4Q, a benchmark of thirty-six real, manually validated Qiskit bugs from four popular Qiskit elements (Terra, Aer, Ignis, and Aqua), supplemented with the test cases for reproducing buggy behaviors. Bugs4Q also provides interfaces for accessing the buggy and fixed versions of the Qiskit programs and executing the corresponding test cases, facilitating the reproducible empirical studies and comparisons of Qiskit program debugging and testing tools. Bugs4Q is publicly available at https://github.com/Z-928/Bugs4Q

With the widespread adoption of smartphones in our daily life, mobile games experienced increasing demand over the past years. Meanwhile, the quality of mobile games has been continuously drawing more and more attention, which can greatly affect the player experience. For better quality assurance, general-purpose testing has been extensively studied for mobile apps. However, due to the unique characteristic of mobile games, existing mobile testing techniques may not be directly suitable and applicable. To better understand the challenges in mobile game testing, in this paper, we first initiate an early step to conduct an empirical study towards understanding the challenges and pain points of mobile game testing process at our industrial partner NetEase Games. Specifically, we first conduct a survey from the mobile test development team at NetEase Games via both scrum interviews and questionnaires. We found that accurate and effective GUI widget detection for mobile games could be the pillar to boost the automation of mobile game testing and other downstream analysis tasks in practice.

The state-of-the-art deep neural network (DNN) achieves impressive performance on the input that is similar to training data. However, it fails to make reasonable decisions on the input that is quite different from training data, i.e., out-of-distribution (OOD) examples. Although many techniques have been proposed to detect OOD examples in recent years, it is still a lack of a systematic study about the effectiveness and robustness of different techniques as well as the performance of OOD-aware DNN models. In this paper, we conduct a comprehensive study to unveil the mystery of current OOD detection techniques, and investigate the differences between OOD-unaware/-aware DNNs in model performance, robustness, and uncertainty. We first compare the effectiveness of existing detection techniques and identify the best one. Then, evasion attacks are performed to evaluate the robustness of techniques. Furthermore, we compare the accuracy and robustness between OOD-unaware/-aware DNNs. At last, we study the uncertainty of different models on various kinds of data. Empirical results show OOD-aware detection modules have better performance and are more robust against random noises and evasion attacks. OOD-awareness seldom degrades the accuracy of DNN models in training/test datasets. In contrast, it makes the DNN model more robust against adversarial attacks and noisy inputs. Our study calls for attention to the development of OOD-aware DNN models and the necessity to take data distribution into account when robust and reliable DNN models are desired.

Over the past decade, deep learning (DL) has been successfully applied to many industrial domain-specific tasks. However, the current state-of-the-art DL software still suffers from quality issues, which raises great concern especially in the context of safety- and security-critical scenarios. Adversarial examples (AEs) represent a typical and important type of defects needed to be urgently addressed, on which a DL software makes incorrect decisions. Such defects occur through either intentional attack or physical-world noise perceived by input sensors, potentially hindering further industry deployment. The intrinsic uncertainty nature of deep learning decisions can be a fundamental reason for its incorrect behavior. Although some testing, adversarial attack and defense techniques have been recently proposed, it still lacks a systematic study to uncover the relationship between AEs and DL uncertainty.

Although deep neural networks (DNNs) achieve rather high performance in many cutting-edge applications (e.g., autonomous driving, medical diagnose), their trustworthiness on real-world scenarios still posts concerns, where some specific failure examples are often encountered during the real-world operational environment. With the limited failure examples collected during the practical operation, how to effectively leverage such failure cases to repair and enhance DNN so as to generalize to more potentially suspicious samples is challenging, but of great importance. In this paper, we formulate the failure-data-driven DNN repairing as a data augmentation problem, and design a novel augmentation-based repairing method, which to the best extent leverages limited failure cases. To realize the DNN repairing effects that generalize to specific failure examples, we originally propose few-shot guided mix (FSGMix) that augments training data with the guidance of failure examples. As a result, our method is able to achieve high generalization to the collected failure examples and other similar suspicious data. The preliminary evaluation on CIFAR-10 dataset demonstrates the potential of our proposed technique, which automatically learns to resolve the potential failure patterns in the DNN operational environment.

Deep learning (DL) has been successfully applied to many cutting-edge applications, e.g., image processing, speech recognition, and natural language processing. As more and more DL software is made open-sourced, publicly available, and organized in model repositories and stores (Model Zoo, ModelDepot), there comes a need to understand the relationships of these DL models regarding their maintenance and evolution tasks. Although clone analysis has been extensively studied for traditional software, up to the present, clone analysis has not been investigated for DL software. Since DL software adopts the data-driven development paradigm, it is still not clear whether and to what extent the clone analysis techniques of traditional software could be adapted to DL software.

As Deep Learning (DL) is continuously adopted in many industrial applications, its quality and reliability start to raise concerns. Similar to the traditional software development process, testing the DL software to uncover its defects at an early stage is an effective way to reduce risks after deployment. According to the fundamental assumption of deep learning, the DL software does not provide statistical guarantee and has limited capability in handling data that falls outside of its learned distribution, i.e., out-of-distribution (OOD) data. Although recent progress has been made in designing novel testing techniques for DL software, which can detect thousands of errors, the current state-of-the-art DL testing techniques usually do not take the distribution of generated test data into consideration. It is therefore hard to judge whether the "identified errors" are indeed meaningful errors to the DL application (i.e., due to quality issues of the model) or outliers that cannot be handled by the current model (i.e., due to the lack of training data). Tofill this gap, we take the first step and conduct a large scale empirical study, with a total of 451 experiment configurations, 42 deep neural networks (DNNs) and 1.2 million test data instances, to investigate and characterize the impact of OOD-awareness on DL testing. We further analyze the consequences when DL systems go into production by evaluating the effectiveness of adversarial retraining with distribution-aware errors. The results confirm that introducing data distribution awareness in both testing and enhancement phases outperforms distribution unaware retraining by up to 21.5%.

State-of-the-art deep learning (DL) systems are vulnerable to adversarial examples, which hinders their potential adoption in safety-and security-critical scenarios. While some recent progress has been made in analyzing the robustness of feed-forward neural networks, the robustness analysis for stateful DL systems, such as recurrent neural networks (RNNs), still remains largely uncharted. In this paper, we propose Marble, a model-based approach for quantitative robustness analysis of real-world RNN-based DL systems. Marble builds a probabilistic model to compactly characterize the robustness of RNNs through abstraction. Furthermore, we propose an iterative refinement algorithm to derive a precise abstraction, which enables accurate quantification of the robustness measurement. We evaluate the effectiveness of Marble on both LSTM and GRU models trained separately with three popular natural language datasets. The results demonstrate that (1) our refinement algorithm is more efficient in deriving an accurate abstraction than the random strategy, and (2) Marble enables quantitative robustness analysis, in rendering better efficiency, accuracy, and scalability than the state-of-the-art techniques.

As the GAN-based face image and video generation techniques, widely known as DeepFakes, have become more and more matured and realistic, there comes a pressing and urgent demand for effective DeepFakes detectors. Motivated by the fact that remote visual photoplethysmography (PPG) is made possible by monitoring the minuscule periodic changes of skin color due to blood pumping through the face, we conjecture that normal heartbeat rhythms found in the real face videos will be disrupted or even entirely broken in a DeepFake video, making it a potentially powerful indicator for DeepFake detection. In this work, we propose DeepRhythm, a DeepFake detection technique that exposes DeepFakes by monitoring the heartbeat rhythms. DeepRhythm utilizes dual-spatial-temporal attention to adapt to dynamically changing face and fake types. Extensive experiments on FaceForensics++ and DFDC-preview datasets have confirmed our conjecture and demonstrated not only the effectiveness, but also the generalization capability of DeepRhythm over different datasets by various DeepFakes generation techniques and multifarious challenging degradations.

The past decade has seen the great potential of applying deep neural network (DNN) based software to safety-critical scenarios, such as autonomous driving. Similar to traditional software, DNNs could exhibit incorrect behaviors, caused by hidden defects, leading to severe accidents and losses. In this paper, we propose DeepHunter, a coverage-guided fuzz testing framework for detecting potential defects of general-purpose DNNs. To this end, we first propose a metamorphic mutation strategy to generate new semantically preserved tests, and leverage multiple extensible coverage criteria as feedback to guide the test generation. We further propose a seed selection strategy that combines both diversity-based and recency-based seed selection. We implement and incorporate 5 existing testing criteria and 4 seed selection strategies in DeepHunter. Large-scale experiments demonstrate that (1) our metamorphic mutation strategy is useful to generate new valid tests with the same semantics as the original seed, by up to a 98% validity ratio; (2) the diversity-based seed selection generally weighs more than recency-based seed selection in boosting the coverage and in detecting defects; (3) DeepHunter outperforms the state of the arts by coverage as well as the quantity and diversity of defects identified; (4) guided by corner-region based criteria, DeepHunter is useful to capture defects during the DNN quantization for platform migration.

As deep learning (DL) opens the way to many technological innovations in a wild range of fields, more and more researchers and developers from diverse domains start to take advantage of DLs. In many circumstances, a developer leverages a DL framework and programs the training software in the form of source code (e.g., Python, Java). However, not all of the developers across domains are skilled at programming. It is highly desirable to provide a way so that a developer could focus on how to design and optimize their DL systems instead of spending too much time on programming. To simplify the programming process towards saving time and effort especially for beginners, we propose and implement DeepVisual, a visual programming tool for the design and development of DL systems. DeepVisual represents each layer of a neural network as a component. A user can drag-and-drop components to design and build a DL model, after which the training code is automatically generated. Moreover, DeepVisual supports to extract the neural network architecture on the given source code as input. We implement DeepVisual as a PyCharm plugin and demonstrate its usefulness on two typical use cases.

Recurrent neural network (RNN) has achieved great success in processing sequential inputs for applications such as automatic speech recognition, natural language processing and machine translation. However, quality and reliability issues of RNNs make them vulnerable to adversarial attacks and hinder their deployment in real-world applications. In this paper, we propose a quantitative analysis framework - DeepStellar - to pave the way for effective quality and security analysis of software systems powered by RNNs. DeepStellar is generic to handle various RNN architectures, including LSTM and GRU, scalable to work on industrial-grade RNN models, and extensible to develop customized analyzers and tools. We demonstrated that, with DeepStellar, users are able to design efficient test generation tools, and develop effective adversarial sample detectors. We tested the developed applications on three real RNN models, including speech recognition and image classification. DeepStellar outperforms existing approaches three hundred times in generating defect-triggering tests and achieves 97% accuracy in detecting adversarial attacks. A video demonstration which shows the main features of DeepStellar is available at: https://sites.google.com/view/deepstellar/tool-demo.

Deep Learning (DL) has recently achieved tremendous success. A variety of DL frameworks and platforms play a key role to catalyze such progress. However, the differences in architecture designs and implementations of existing frameworks and platforms bring new challenges for DL software development and deployment. Till now, there is no study on how various mainstream frameworks and platforms influence both DL software development and deployment in practice. To fill this gap, we take the first step towards understanding how the most widely-used DL frameworks and platforms support the DL software development and deployment. We conduct a systematic study on these frameworks and platforms by using two types of DNN architectures and three popular datasets. (1) For development process, we investigate the prediction accuracy under the same runtime training configuration or same model weights/biases. We also study the adversarial robustness of trained models by leveraging the existing adversarial attack techniques. The experimental results show that the computing differences across frameworks could result in an obvious prediction accuracy decline, which should draw the attention of DL developers. (2) For deployment process, we investigate the prediction accuracy and performance (refers to time cost and memory consumption) when the trained models are migrated/quantized from PC to real mobile devices and web browsers. The DL platform study unveils that the migration and quantization still suffer from compatibility and reliability issues. Meanwhile, we find several DL software bugs by using the results as a benchmark. We further validate the results through bug confirmation from stakeholders and industrial positive feedback to highlight the implications of our study. Through our study, we summarize practical guidelines, identify challenges and pinpoint new research directions, such as understanding the characteristics of DL frameworks and platforms, avoiding compatibility and reliability issues, detecting DL software bugs, and reducing time cost and memory consumption towards developing and deploying high quality DL systems effectively.

Deep neural networks (DNNs) are increasingly expanding their real-world applications across domains, e.g., image processing, speech recognition and natural language processing. However, there is still limited tool support for DNN testing in terms of test data quality and model robustness. In this paper, we introduce a mutation testing-based tool for DNNs, DeepMutation++, which facilitates the DNN quality evaluation, supporting both feed-forward neural networks (FNNs) and stateful recurrent neural networks (RNNs). It not only enables to statically analyze the robustness of a DNN model against the input as a whole, but also allows to identify the vulnerable segments of a sequential input (e.g. audio input) by runtime analysis. It is worth noting that DeepMutation++ specially features the support of RNNs mutation testing. The tool demo video can be found on the project website https://sites.google.com/view/deepmutationpp.

Deep neural network (DNN) has been widely applied to safety-critical scenarios such as autonomous vehicle, security surveillance, and cyber-physical control systems. Yet, the incorrect behaviors of DNNs can lead to severe accidents and tremendous losses due to hidden defects. In this paper, we present DeepHunter, a general-purpose fuzzing framework for detecting defects of DNNs. DeepHunter is inspired by traditional grey-box fuzzing and aims to increase the overall test coverage by applying adaptive heuristics according to runtime feedback. Specifically, DeepHunter provides a series of seed selection strategies, metamorphic mutation strategies, and testing criteria customized to DNN testing; all these components support multiple built-in configurations which are easy to extend. We evaluated DeepHunter on two popular datasets and the results demonstrate the effectiveness of DeepHunter in achieving coverage increase and detecting real defects. A video demonstration which showcases the main features of DeepHunter can be found at https://youtu.be/s5DfLErcgrc.

Deep Learning (DL) has achieved tremendous success in many cutting-edge applications. However, the state-of-the-art DL systems still suffer from quality issues. While some recent progress has been made on the analysis of feed-forward DL systems, little study has been done on the Recurrent Neural Network (RNN)-based stateful DL systems, which are widely used in audio, natural languages and video processing, etc. In this paper, we initiate the very first step towards the quantitative analysis of RNN-based DL systems. We model RNN as an abstract state transition system to characterize its internal behaviors. Based on the abstract model, we design two trace similarity metrics and five coverage criteria which enable the quantitative analysis of RNNs. We further propose two algorithms powered by the quantitative measures for adversarial sample detection and coverage-guided test generation. We evaluate DeepStellar on four RNN-based systems covering image classification and automated speech recognition. The results demonstrate that the abstract model is useful in capturing the internal behaviors of RNNs, and confirm that (1) the similarity metrics could effectively capture the differences between samples even with very small perturbations (achieving 97% accuracy for detecting adversarial samples) and (2) the coverage criteria are useful in revealing erroneous behaviors (generating three times more adversarial samples than random testing and hundreds times more than the unrolling approach).

Deep learning (DL) has achieved remarkable progress over the past decade and has been widely applied to many industry domains. However, the robustness of DL systems recently becomes great concerns, where minor perturbation on the input might cause the DL malfunction. These robustness issues could potentially result in severe consequences when a DL system is deployed to safety-critical applications and hinder the real-world deployment of DL systems. Testing techniques enable the robustness evaluation and vulnerable issue detection of a DL system at an early stage. The main challenge of testing a DL system attributes to the high dimensionality of its inputs and large internal latent feature space, which makes testing each state almost impossible. For traditional software, combinatorial testing (CT) is an effective testing technique to balance the testing exploration effort and defect detection capabilities. In this paper, we perform an exploratory study of CT on DL systems. We propose a set of combinatorial testing criteria specialized for DL systems, as well as a CT coverage guided test generation technique. Our evaluation demonstrates that CT provides a promising avenue for testing DL systems.

As more and more domain specific big data become available, there comes a strong need on the fast development and deployment of deep learning (DL) systems with high quality for domain specific applications, including many safety-critical scenarios. In traditional software engineering, software visualization plays an important role to enhance developers' performance with many tools available. However, there are limited visualization supports existing for DL systems, especially in integrated development environments (IDEs) that allow a developer to visualize the source code of a deep neural network (DNN) and its graph architecture. In this paper, we propose DeepGraph, a visualization tool for visualizing and understanding a deep neural network. DeepGraph analyzes the training program to construct the graph representation of a DNN, and establishes and maintains the linkage (mapping) between the source code of the training program and its corresponding neural network architecture. We implemented DeepGraph as a PyCharm plugin and performed preliminary empirical study to demonstrate its usefulness for understanding deep neural networks.

Deep learning (DL) defines a new data-driven programming paradigm where the internal system logic is largely shaped by the training data. The standard way of evaluating DL models is to examine their performance on a test dataset. The quality of the test dataset is of great importance to gain confidence of the trained models. Using an inadequate test dataset, DL models that have achieved high test accuracy may still lack generality and robustness. In traditional software testing, mutation testing is a well-established technique for quality evaluation of test suites, which analyzes to what extent a test suite detects the injected faults. However, due to the fundamental difference between traditional software and deep learning-based software, traditional mutation testing techniques cannot be directly applied to DL systems. In this paper, we propose a mutation testing framework specialized for DL systems to measure the quality of test data. To do this, by sharing the same spirit of mutation testing in traditional software, we first define a set of source-level mutation operators to inject faults to the source of DL (i.e., training data and training programs). Then we design a set of model-level mutation operators that directly inject faults into DL models without a training process. Eventually, the quality of test data could be evaluated from the analysis on to what extent the injected faults could be detected. The usefulness of the proposed mutation testing techniques is demonstrated on two public datasets, namely MNIST and CIFAR-10, with three DL models.

Deep learning (DL) defines a new data-driven programming paradigm that constructs the internal system logic of a crafted neuron network through a set of training data. We have seen wide adoption of DL in many safety-critical scenarios. However, a plethora of studies have shown that the state-of-the-art DL systems suffer from various vulnerabilities which can lead to severe consequences when applied to real-world applications. Currently, the testing adequacy of a DL system is usually measured by the accuracy of test data. Considering the limitation of accessible high quality test data, good accuracy performance on test data can hardly provide confidence to the testing adequacy and generality of DL systems. Unlike traditional software systems that have clear and controllable logic and functionality, the lack of interpretability in a DL system makes system analysis and defect detection difficult, which could potentially hinder its real-world deployment. In this paper, we propose DeepGauge, a set of multi-granularity testing criteria for DL systems, which aims at rendering a multi-faceted portrayal of the testbed. The in-depth evaluation of our proposed testing criteria is demonstrated on two well-known datasets, five DL systems, and with four state-of-the-art adversarial attack techniques against DL. The potential usefulness of DeepGauge sheds light on the construction of more generic and robust DL systems.

AngularJS is a popular framework for single page web applications.In AngularJS applications, the programming logic is implemented in Javascript, while the layout is defined separately in HTML files.Due to this separation, data and control flow is usually hard to track.We propose a method to visualize the data and control flow in AngularJS-based single page web applications and separate interactions from each other.Our method helps to get a better understanding of the application's work flow, to realize the boundaries of the interactions, and to know what is updated in an interaction and what is not.

Code clones are prevalent in software systems due to many factors in software development. Detecting code clones and managing consistency between them along code evolution can be very useful for reducing clone-related bugs and maintenance costs. Despite some early attempts at detecting code clones and managing the consistency between them, the state-of-the-art tool can only handle simple code clones whose structures are identical or quite similar. However, existing empirical studies show that clones can have quite different structures with their evolution, which can easily go beyond the capability of the state-of-the-art tool. In this paper, we propose CCSync, a novel, rule-directed approach, which paves the structure differences between the code clones and synchronizes them even when code clones become quite different in their structures. The key steps of this approach are, given two code clones, to (1) extract a synchronization rule from the relationship between the clones, and (2) once one code fragment is updated, propagate the modifications to the other following the synchronization rule. We have implemented a tool for CCSync and evaluated its effectiveness on five Java projects. Our results shows that there are many code clones suitable for synchronization, and our tool achieves precisions of up to 92% and recalls of up to 84%. In particular, more than 76% of our generated revisions are identical with manual revisions.

Test cases are useful for program comprehension. Developers often understand dynamic behavior of systems by running their test cases. As manual testing is expensive, automatic testing has been extensively studied to reduce the cost. However, without sufficient knowledge of the software under test, it is difficult for automated testing techniques to create effective test cases, especially for software that requires complex inputs. In this paper, we propose to reuse existing test cases from the libraries of software under test, to generate better test cases. We have the observation that, when developers start to test the target software, the test cases of its dependent libraries are often available. Therefore, we propose to perform program analysis on these artifacts to extract relevant code fragments to create test sequences. We further seed these sequences to a random test generator GRT to generate test cases for target software. The preliminary experiments show that the technique significantly improves the effectiveness of GRT. Our in-depth analysis reveals that several dependency metrics are good indicators of the potential benefits of applying our technique on specific programs and their libraries.

To attract more users on different platforms, many projects release their versions in multiple programming languages (e.g., Java and C#). They typically have many code snippets that implement similar functionalities, i.e., cross-language clones. Programmers often need to track and modify cross-language clones consistently to maintain similar functionalities across different language implementations. In literature, researchers have proposed approaches to detect cross- language clones, mostly for languages that share a common intermediate language (such as the .NET language family) so that techniques for detecting single-language clones can be applied. As a result, those approaches cannot detect cross-language clones for many projects that are not implemented in a .NET language. To overcome the limitation, in this paper, we propose a novel approach, CLCMiner, that detects cross-language clones automatically without the need of an intermediate language. Our approach mines such clones from revision histories, which reflect how programmers maintain cross-language clones in practice. We have implemented a prototype tool for our approach and conducted an evaluation on five open source projects that have versions in Java and C#. The results show that CLCMiner achieves high accuracy and point to promising future work.

Detecting lock-related defects has long been a hot research topic in software engineering. Many efforts have been spent on detecting such deadlocks in concurrent software systems. However, latent locks may be hidden in application programming interface (API) methods whose source code may not be accessible to developers. Many APIs have latent locks. For example, our study has shown that J2SE alone can have 2,000+ latent locks. As latent locks are less known by developers, they can cause deadlocks that are hard to perceive or diagnose. Meanwhile, the state-of-the-art tools mostly handle API methods as black boxes, and cannot detect deadlocks that involve such latent locks. In this paper, we propose a novel black-box testing approach, called LockPeeker, that reveals latent locks in Java APIs. The essential idea of LockPeeker is that latent locks of a given API method can be revealed by testing the method and summarizing the locking effects during testing execution. We have evaluated LockPeeker on ten real-world Java projects. Our evaluation results show that (1) LockPeeker detects 74.9% of latent locks in API methods, and (2) it enables state-of-the-art tools to detect deadlocks that otherwise cannot be detected.

Java virtual machine (JVM) is a core technology, whose reliability is critical. Testing JVM implementations requires painstaking effort in designing test classfiles (*.class) along with their test oracles. An alternative is to employ binary fuzzing to differentially test JVMs by blindly mutating seeding classfiles and then executing the resulting mutants on different JVM binaries for revealing inconsistent behaviors. However, this blind approach is not cost effective in practice because most of the mutants are invalid and redundant. This paper tackles this challenge by introducing classfuzz, a coverage-directed fuzzing approach that focuses on representative classfiles for differential testing of JVMs’ startup processes. Our core insight is to (1) mutate seeding classfiles using a set of predefined mutation operators (mutators) and employ Markov Chain Monte Carlo (MCMC) sampling to guide mutator selection, and (2) execute the mutants on a reference JVM implementation and use coverage uniqueness as a discipline for accepting representative ones. The accepted classfiles are used as inputs to differentially test different JVM implementations and find defects. We have implemented classfuzz and conducted an extensive evaluation of it against existing fuzz testing algorithms. Our evaluation results show that classfuzz can enhance the ratio of discrepancy-triggering classfiles from 1.7% to 11.9%. We have also reported 62 JVM discrepancies, along with the test classfiles, to JVM developers. Many of our reported issues have already been confirmed as JVM defects, and some even match recent clarifications and changes to the Java SE 8 edition of the JVM specification.

Due to various considerations, programmers often need to backtrack their code. Furthermore, as the most recent edit may not be the wrong edit, programmers sometimes have to backtrack their code for arbitrary edits, which is referred as selective undo in this paper. To meet the needs, researchers have proposed various approaches to support selective undo. However, to the best of our knowledge, these approaches can support only simple edits, and cannot handle refactoring, although most code editors already provide various refactoring actions. Indeed, it is challenging to support selective undo for refactoring, since multiple code elements and complicated actions can be involved. In this paper, we present a novel approach that leverages Bidirectional Transformation (BX) to support selective undo for refactoring. We evaluate our approach on a recent refactoring tool that transfers enhanced for loops to lambda expressions. Our results show that our approach achieves an accuracy of up to 89%.

JavaScript is the de facto dominant programming language for developing web applications. Most popular websites are using JavaScript, especially to develop client-side features. Being syntactically flexible and highly dynamic, JavaScript is easy to use and productive, but its code is known to be less maintainable. The task of maintaining client-side JavaScript code is further complicated by the pervasive interactions between JavaScript code and HTML elements, through browsers. In this paper, we present JS-Slicer, a dynamic slicer for JavaScript, to ease the task of understanding and debugging practical client-side JavaScript code. JS-Slicer defines three types of dependences, including data dependences, control dependences, and DOM dependences, to capture all relationships between program elements. JS-Slicer extends a novel dynamic analysis framework and combines dynamic and static analysis to precisely capture the dependences at run-time. A lot of language specific issues are properly handled, which enables JS-Slicer to slice practical JavaScript code. Our evaluation on six real-world web applications and JavaScript libraries shows that JS-Slicer is both precise and efficient: on average it captures around 40K dependences in 2.5K lines of code, in less than 3.0 seconds.

Researchers have proposed various approaches to detect concurrency bugs and improve multi-threaded programs, but performing evaluations of the effectiveness of these approaches still remains a substantial challenge. We survey the existing evaluations and find out that they often use code or bugs not representative of real world. To improve representativeness, we have prepared JaConTeBe, a benchmark suite of 47 confirmed concurrency bugs from 8 popular open-source projects, supplemented with test cases for reproducing buggy behaviors. Running three approaches on JaConTeBe shows that our benchmark suite confirms some limitations of the three approaches. We submitted JaConTeBe to the SIR repository (a software-artifact repository for rigorous controlled experiments), and it was included as a part of SIR.

Over the years of software development, a vast amount of source code has been accumulated. Many code search tools were proposed to help programmers reuse previously-written code by performing free-text queries over a large-scale codebase. Our experience shows that the accuracy of these code search tools are often unsatisfactory. One major reason is that existing tools lack of query understanding ability. In this paper, we propose CodeHow, a code search technique that can recognize potential APIs a user query refers to. Having understood the potentially relevant APIs, CodeHow expands the query with the APIs and performs code retrieval by applying the Extended Boolean model, which considers the impact of both text similarity and potential APIs on code search. We deploy the backend of CodeHow as a Microsoft Azure service and implement the front-end as a Visual Studio extension. We evaluate CodeHow on a large-scale codebase consisting of 26K C# projects downloaded from GitHub. The experimental results show that when the top 1 results are inspected, CodeHow achieves a precision score of 0.794 (i.e., 79.4% of the first returned results are relevant code snippets). The results also show that CodeHow outperforms conventional code search tools. Furthermore, we perform a controlled experiment and a survey of Microsoft developers. The results confirm the usefulness and effectiveness of CodeHow in programming practices.

Aspect-oriented software development (AOSD) has been developed for supporting a long-standing idea of Separation of Concerns (SoC) and enhancing software non-functional attributes including modularity, reusability, and maintainability. Many architectural description languages (ADLs) also provide software developers with support in specifying aspects in software architectures. However, when applied to describe trustworthy software systems, these ADLs face difficulties in using aspects to specify trustworthy attributes and maintaining consistency between the architectures before and after weaving of aspects. In this paper, we extend Breeze, a graph grammar supported ADL, to AspectBreeze. AspectBreeze allows trustworthiness aspects to be easily defined and seamlessly woven into base architectures. Further, architectures are defined along with graph grammars that allow any change to a base architecture to be reflected in the corresponding architecture in which aspects are woven. This paper also presents a case study of an online auction system to show how AspectBreeze is used and how graph grammars can maintain the consistency between the architectures before and after weaving of trustworthiness aspects.

X10 is a HPC (High Performance Computing) programming language proposed by IBM for supporting a PGAS (Partitioned Global Address Space) programming model offering a shared address space. The address space can be further partitioned into several logical locations where objects and activities (or threads) will be dynamically created. An analysis of locations can help to check the safety of object accesses through exploring which objects and activities may reside in which locations, while in practice the objects and activities are usually designated at runtime and their locations may also vary under different environments. In this paper, we propose a constraint-based locality analysis method called Leopard for X10. Leopard calculates the points-to relations for analyzing the objects and activities in a program and uses a place constraint graph to analyze their locations.We have developed a tool to support Leopard, and conducted an experiment to evaluate its effectiveness and efficiency. The experimental results show that Leopard can calculate the locations of objects and activities precisely.

With the extensive use of client-side JavaScript in web applications, web contents are becoming more dynamic than ever before. This poses significant challenges for search engines, because more web URLs are now embedded or hidden inside JavaScript code and most web crawlers are script-agnostic, significantly reducing the coverage of search engines. We present a hybrid approach that combines static analysis with dynamic execution, overcoming the weakness of a purely static or dynamic approach that either lacks accuracy or suffers from huge execution cost. We also propose to integrate program analysis techniques such as statement coverage and program slicing to improve the performance of URL mining.

Programmers extensively use application programming interfaces (APIs) to leverage existing libraries and frameworks. However, correctly and efficiently choosing and using APIs from unfamiliar libraries and frameworks is still a non-trivial task. Programmers often need to ruminate on API documentations (that are often incomplete) or inspect code examples (that are often absent) to learn API usage patterns. Recently, various techniques have been proposed to alleviate this problem by creating API summarizations, mining code examples, or showing common API call sequences. However, few techniques focus on recommending API parameters. In this paper, we propose an automated technique, called Precise, to address this problem. Differing from common code completion systems, Precise mines existing code bases, uses an abstract usage instance representation for each API usage example, and then builds a parameter usage database. Upon a request, Precise queries the database for abstract usage instances in similar contexts and generates parameter candidates by concretizing the instances adaptively. The experimental results show that our technique is more general and applicable than existing code completion systems, specially, 64% of the parameter recommendations are useful and 53% of the recommendations are exactly the same as the actual parameters needed. We have also performed a user study to show our technique is useful in practice.

Static analysis tools have been widely used to detect potential defects without executing programs. It helps programmers raise the awareness about subtle correctness issues in the early stage. However, static defect detection tools face the high false positive rate problem. Therefore, programmers have to spend a considerable amount of time on screening out real bugs from a large number of reported warnings, which is time-consuming and inefficient. To alleviate the above problem during the report inspection process, we present EFindBugs to employ an effective two-stage error ranking strategy that suppresses the false positives and ranks the true error reports on top, so that real bugs existing in the programs could be more easily found and fixed by the programmers. In the first stage, EFindBugs initializes the ranking by assigning predefined defect likelihood for each bug pattern and sorting the error reports by the defect likelihood in descending order. In the second stage, EFindbugs optimizes the initial ranking self-adaptively through the feedback from users. This optimization process is executed automatically and based on the correlations among error reports with the same bug pattern. Our experiment on three widely-used Java projects (AspectJ, Tomcat, and Axis) shows that our ranking strategy outperforms the original ranking in Find Bugs in terms of precision, recall and F1-score.

During debugging processes, breakpoints are frequently used to inspect and understand runtime behaviors of programs. Although most development environments offer convenient breakpoint facilities, the use of these environments usually requires considerable human efforts in order to generate useful breakpoints. Before setting breakpoints or typing breakpoint conditions, developers usually have to make some judgements and hypotheses on the basis of their observations and experience. To reduce this kind of efforts we present a tool, named BPGen, to automatically generate breakpoints for debugging. BPGen uses three well-known dynamic fault localization techniques in tandem to identify suspicious program statements and states, through which both conditional and unconditional breakpoints are generated. BPGen is implemented as an Eclipse plugin for supplementing the existing Eclipse JDT debugger.

Program call graph representation can be used to support many tasks in compiler optimization, program comprehension, and software maintenance. During software evolution, the call graph needs to remain fairly precise and be updated quickly in response to software changes. In this paper, we present an approach to incremental update, instead of exhaustive analysis of the initially constructed call graph in AspectJ software. Our approach first decomposes the source code edits between the updated and initial software versions into a set of atomic change representations, which capture the semantic differences. Then, we explore the relationship between atomic changes and call graph to incrementally update the initially constructed graph, instead of rebuilding it from the ground up. We implement the reanalysis approach on top of the ajc AspectJ compiler and perform an empirical study on 24 versions of eight AspectJ benchmarks. The experiment result shows that our approach can reduce a large portion of unnecessary reanalysis cost as program changes occur, and significant savings are observed for the incremental reconstruction of AspectJ call graph in comparison with an exhaustive analysis, with no loss in precision.

Program differencing is a common means of software debugging. Although many differencing algorithms have been proposed for procedural and object-oriented languages like C and Java, there is no differencing algorithm for aspect-oriented languages so far. In this paper we propose an approach for difference analysis of aspect-oriented programs. The proposed algorithm contains a novel way of matching two versions of a module of which the signature has been modified. For this, we also work out a set of well defined signatures for the new elements in the AspectJ language. In accordance with these signatures, and with those existent for elements of the Java language, we investigate a set of signature patterns to be used with the module matching algorithm. Furthermore, we demonstrate successful application of a node-by-node comparison algorithm originally developed for object-oriented programs. Using a tool which implements our algorithms, we set up and evaluate a set of test cases. The results demonstrate the effectiveness of our approach for a large subset of the AspectJ language.

Adaptive Random Testing (ART) is a testing technique which is based on an observation that a test input usually has the same potential as its neighbors in detection of a specific program defect. ART helps to improve the efficiency of random testing in that test inputs are selected evenly across the input spaces. However, the application of ART to object-oriented programs (e.g., C++ and Java) still faces a strong challenge in that the input spaces of object-oriented programs are usually high dimensional, and therefore an even distribution of test inputs in a space as such is difficult to achieve. In this paper, we propose a divergence-oriented approach to adaptive random testing of Java programs to address this challenge. The essential idea of this approach is to prepare for the tested program a pool of test inputs each of which is of significant difference from the others, and then to use the ART technique to select test inputs from the pool for the tested program. We also develop a tool called ARTGen to support this testing approach, and conduct experiment to test several popular open-source Java packages to assess the effectiveness of the approach. The experimental result shows that our approach can generate test cases with high quality.

Test case prioritization techniques have been empirically proved to be effective in improving the rate of fault detection in regression testing. However, most of previous techniques assume that all the faults have equal severity, which dose not meet the practice. In addition, because most of the existing techniques rely on the information gained from previous execution of test cases or source code changes, few of them can be directly applied to non-regression testing. In this paper, aiming to improve the rate of severe faults detection for both regression testing and non-regression testing, we propose a novel test case prioritization approach based on the analysis of program structure. The key idea of our approach is the evaluation of testing-importance for each module (e.g., method) covered by test cases. As a proof of concept, we implement $Apros$, a test case prioritization tool, and perform an empirical study on two real, non-trivial Java programs. The experimental result represents that our approach could be a promising solution to improve the rate of severe faults detection.

Change impact analysis is a useful technique for software evolution. It determines the effects of a source editing session and provides valuable feedbacks to the programmers for making correct decisions. Recently, many techniques have been proposed to support change impact analysis of procedural or object-oriented software, but seldom effort has been made for aspect-oriented software. In this paper we propose a new change impact analysis technique for AspectJ programs. At the core of our approach is the atomic change representation which captures the semantic differences between two versions of an AspectJ program. We also present an impact analysis model, based on AspectJ call graph construction, to determine the affected program fragments, affected tests and their responsible changes. The proposed techniques have been implemented in Celadon, a change impact analysis framework for AspectJ programs. We performed an empirical evaluation on 24 versions of eight AspectJ benchmarks. The result shows that our proposed technique can effectively perform change impact analysis and provide valuable information in AspectJ software evolution.

Aspect-oriented programming (AOP) is gaining popularity with the wider adoption of languages such as AspectJ. During AspectJ software evolution, when regression tests fail, it may be tedious for programmers to find out the failure-inducing changes by manually inspecting all code editing. To eliminate the expensive effort spent on debugging, we developed AutoFlow, an automatic debugging tool for AspectJ software. AutoFlow integrates the potential of delta debugging algorithm with the benefit of change impact analysis to narrow down the search for faulty changes. It first uses change impact analysis to identify a subset of responsible changes for a failed test, then ranks these changes according to our proposed heuristic (indicating the likelihood that they may have contributed to the failure), and finally employs an improved delta debugging algorithm to determine a minimal set of faulty changes. The main feature of AutoFlow is that it can automatically reduce a large portion of irrelevant changes in an early phase, and then locate faulty changes effectively.

We discussed some open research issues that should be addressed in the comprehension and maintenance of AO software, and gave a brief overview of the work in progress in this area. Some research results are quite encouraging, but many work still needs to be done to support the maintenance of AO software.

Aspect-oriented software development (AOSD) is gaining popularity with the wider adoption of languages such as AspectJ. However, though the state-of-the-art aspect-oriented programming environment (such as AJDT in the Eclipse IDE) provides powerful capabilities to check the syntactic or grammar errors in AspectJ programs, it fails to detect potential semantic defects in aspect-oriented software systems. In this paper, we present XFindBugs, an eXtended FindBugs for AspectJ, to help programmers find potential bugs in AspectJ applications through static analysis. XFindBugs supports 17 bug patterns to cover common error-prone features in an aspect-oriented system, and integrates the corresponding bug detectors into the FindBugs framework. We evaluate XFindBugs on a number of large-scale open source AspectJ projects (306,800 LOC in total). In our evaluation, XFindBugs confirms 7 reported bugs and finds 257 previously unknown defects. Our experiment also indicates that the bug patterns supported in XFindBugs exist in real-world software systems, even for mature applications by experienced programmers.

When regression tests fail unexpectedly after a long session of editing, it may be tedious for programmers to find out the failure-inducing changes by manually inspecting all code edits. To eliminate the expensive effort spent on debugging, we present a hybrid approach, which combines both static and dynamic analysis techniques, to automatically identify the faulty changes. Our approach first uses static change impact analysis to isolate a subset of responsible changes for a failed test, then utilizes the dynamic test execution information to rank these changes according to our proposed heuristic (indicating the likelihood that they may have contributed to the failure), and finally employs an improved Three-Phase delta debugging algorithm, working from the coarse method level to the fine statement level, to find a minimal set of faulty statements.

Points-to analysis is a fundamental analysis technique whose results are useful in compiler optimization and software engineering tools. Although many points-to analysis algorithms have been proposed for procedural and object-oriented languages like C and Java, there is no points-to analysis for aspect-oriented languages so far. Based on Andersen-style points-to analysis for Java, we propose flow- and context-insensitive points-to analysis for AspectJ. The main idea is to perform the analysis crossing the boundary between aspects and classes. Therefore, our technique is able to handle the uniqueaspectual features. To investigate the effectiveness of our technique, we implement our analysis approach on top of the ajc AspectJ compiler and evaluate it on nine AspectJ benchmarks. The experimental result indicates that, compared to existing Java approaches, the proposed technique can achieve a significant higher precision and run in practical time and space.

In this paper, we propose a fine-grained coupling metrics suite for aspect-oriented (AO) systems, to measure software changes during system evolution. We also present a correlation model in terms of intermediate processes, for better evaluating the relation between coupling metrics and system maintainability. To investigate the practicability of our proposed model, we have implemented a coupling metrics analysis tool called AJMetrics and performed an empirical study on eight AspectJ benchmarks. The experiment result suggests that our correlation model provides useful information to evaluate the maintainability of AO systems.

Program verification is a promising approach to improving program quality. To formally verify aspect- oriented programs, we have to find a way to formally specify programs written in aspect-oriented languages. Pipa is a BISL tailored to AspectJ for specifying AspectJ programs. However, Pipa has not provided specification method for pointcuts in AspectJ programs. Based on the exist work of Pipa, and related issues, this paper proposes an approach to specifying pointcuts using purity conception in JML. This paper also provides several examples to illustrate our pointcut specification approach.

Bug patterns are erroneous code idioms or bad coding practices that have been proved fail time and time again. They mainly arise from the misunderstanding of language features, the use of erroneous design patterns or simple mistakes sharing the common behaviors. Aspect-oriented programming (AOP) is a new technique to separate the cross-cutting concerns for improving modularity in software design and implementation. However, there is no effective debugging technique for aspect-oriented programs until now and none of the prior researches focused on the identification of bug patterns in aspect-oriented programs. In this paper, we present six bug patterns in AspectJprogramming language and show the corresponding example for each bug pattern to help to illustrate the symptoms of these patterns. We take this as the first step to provide an underlying basis on testing and debugging of AspectJ programs.

Aspect-oriented software development is gaining popularity with the adoption of languages such as AspectJ. Testing is an important part in any software development, including aspect-oriented development. To automate generation of unit tests for AspectJ programs, we can apply the existing tools that automate generation of unit tests for Java programs. However, these tools can generate a large number of test inputs, and manually inspecting the behavior of the software on all these inputs is time consuming. We propose Raspect, a framework for detecting redundant unit tests for AspectJ programs. We introduce three levels of units in AspectJ programs: advised methods, advice, and intertype methods. We show how to detect at each level redundant test that does not exercise new behavior. Our approach selects only non-redundant tests from the automatically generated test suites, thus allowing the developer to spend less time in inspecting this reduced set of tests. We have implemented Raspect and applied it on 12 subjects taken from a variety of sources; our experience shows that Raspect can effectively reduce the size of generated test suites for inspecting AspectJ programs

Separation of concerns is one of the main tenets of software engineering - allowing developers to reason about software systems in sensible portions, regardless which phase of the lifecycle they are working in. Many researchers in software engineering are actually in the field of aspect-orientation without realizing it.

The current research so far in aspect-oriented software development is focused on problem analysis, software design, and implementation techniques. Even though the importance of software testing is known, it has received little attention in the aspect-oriented paradigm. In this paper, we propose a data-flow-based unit testing approach for aspect oriented programs. Our approach tests two types of units for an aspect-oriented program, i.e., aspects that are modular units of crosscutting implementation of the program, and those classes whose behavior may be affected by one or more aspects. For each aspect or class, our approach performs three levels of testing, i.e., intra-module, inter-module, and intra-aspect or intra-class testing. For an individual module such as apiece of advice, a piece of introduction, and a method, we perform intra-module testing. For a public module along with other modules it calls in an aspect or class, we perform inter-module testing. For modules that can be accessed outside the aspect or class and can be invoked in any order by users of the aspect or class, we perform intra-aspect or intra-class testing. Our approach can handle unit testing problems that are unique to aspect-oriented programs. We use control flow graphs to compute def-use pairs of an aspect or class being tested and use such information to guide the selection of test for the aspect or class.

Classes are the basic modules in object-oriented (OO) software, which consist of attributes and methods. Thus, in an OO environment, cohesion mainly concerns how tight the attributes and methods of classes are. This paper discusses the relationships between attributes and attributes, attributes and methods, and methods and methods of a class based on dependence analysis. Then we discuss the properties of these relationships. According to these properties, this paper proposes a novel approach to measuring class cohesion. Our approach overcomes the limitations of previous class cohesion measures, which consider only one or two of the three relationships in a class. We also prove that this measure satisfies the properties that a good measurement should have.

Class cohesion could be used to evaluate the design quality of classes, to develop test measures for object-oriented software and to restructure poorly designed classes. Among a number of class cohesion measures proposed in the last decade, Chae's measure is based on the structure of the reference graph of a class, which overcomes the limitations of most class cohesion measures. However, it only considers the patterns of interactions among the members of a class partly and hence does not satisfy monotonicity, which may cause the measurement results to be inconsistent with intuition in some cases. This paper first analyzes the limitations of typical cohesion measures for classes in detail, and then proposes an improved cohesion measure ICBMC. Finally, this paper exemplifies the advantages and applications of ICBMC.

Abstract Change impact analysis is a useful technique in software maintenance and evolution. Many techniques have been proposed to support change impact analysis at the code level of software systems, but little effort has been made for change impact analysis at the architectural level. In this paper, we present an approach to supporting change impact analysis at the architectural level of software systems based on an architectural slicing and chopping technique. The main feature of our approach is to assess the effect of changes in a software architecture by analyzing its formal architectural specification, and, therefore, the process of change impact analysis can be automated completely. Copyright © 2002 John Wiley & Sons, Ltd.

In this paper we describe a program analysis system called CLPKIDS, that is based on program dependence analysis, to support development of software engineering tools for understanding, testing, debugging, maintenance and complexity measurement of concurrent logic programs.

The authors propose a program dependence model for concurrent logic programs. They present three types of primary program dependences named the sharing dependence, communication dependence, and unification dependence between arguments in a concurrent logic program. We formally define these primary program dependences based on mode information. We further present a dependence-based representation named the argument dependence net (ADN), which explicitly represents all primary program dependences between arguments in a concurrent logic program. We also discuss some applications of the ADN for developing software engineering tools of concurrent logic programs. Finally, we briefly introduce a program analysis system called CLPKIDS for concurrent logic programs, which is a prototype implementation of the techniques introduced.

Understanding program dependencies in a computer program is essential for many software engineering tasks such as program understanding, testing, debugging, reverse engineering, and maintenance. The article presents an approach to dependence analysis of Java bytecode, and discusses some applications of the technique, which include Java bytecode slicing, understanding, and testing.

Software architecture is receiving increasing attention as a critical level for software systems. As software architecture design resources (in the form of architectural specifications) are going to be accumulated, the development of techniques and tools to support architectural understanding, testing, reengineering, maintenance, and reuse will become an important issue. This paper introduces a new form of slicing, named architectural slicing, to aid architectural understanding and reuse. In contrast to traditional slicing, architectural slicing is designed to operate on the architectural specification of a software system, rather than the source code of a program. Architectural slicing provides knowledge about the high-level structure of a software system, rather than the low-level implementation details of a program. In order to compute an architectural slice, we present the architecture information flow graph which can be used to represent information flows in a software architecture. Based on the graph, we give a two-phase algorithm to compute an architectural slice.

Program slicing has many applications such as program debugging, testing, maintenance, and complexity measurement. This paper concerns the problem of slicing concurrent object-oriented programs that has not been addressed in the literatures until now. To solve this problem, we propose a new program dependence representation named the system dependence net (SDN), which extends previous program dependence representations to represent concurrent object-oriented programs. An SDN of a concurrent object-oriented program consists of a collection of procedure dependence nets each representing a main procedure, a free standing procedure, or a method in a class of the program, and some additional arcs to represent direct dependences between a call and the called procedure/method and transitive interprocedural data dependences. We construct the SDN to represent not only object-oriented features but also concurrency issues in a concurrent object-oriented program. Once a concurrent object-oriented program is represented by its SDN, the slices of the program can be computed based on the SDN as a simple vertex reachability problem in the net.

In this paper a formal model for program dependence analysis of concurrent logic programs is proposed with the following contributions. First, two language-independent program representations are presented for explicitly representing control flows and/or data flows in a concurrent logic program. Then based on these representations, program dependences between literals in concurrent logic programs are defined formally, and a dependence-based program representation named the Literal Dependence Net (LDN) is presented for explicitly representing primary program dependences in a concurrent logic program. Finally, as applications of the LDNs, some important software engineering activities including program slicing, debugging, testing, complexity measurement, and maintenance are discussed in a programming environment for concurrent logic programs.