The e X tensible M arkup L anguage (XML) is a file format widely used for data transmission in modern software development. In recent years, embedding SQL statements in XML files (i.e., XML-SQL) has become a popular way for developing applications with database access capability. Typically, XML-SQL code snippets demonstrate similar functionalities and structures, leading to repetitive programming work. Therefore, leveraging pre-trained code models for automated code generation presents a promising way to alleviate duplicated efforts and enhance the efficiency of developing XML-SQL code. However, XML-SQL code has strong domain-specific characteristics that general pre-trained code models typically struggle to fully harness, thereby leading to limited overall performance of general pre-trained code models. In this article, we aim to address the challenge of handling this domain-specific knowledge. First, we propose a code updating task and construct the corresponding TwinXSQL dataset to better evaluate the model’s code generation performance in the XML-SQL domain. Then, we leverage the common characteristics of XML-SQL and other programming languages (i.e., all programming languages impose grammar constraints on behavior) to design a bipartite-grammar–aware training framework (named BGA) for unsupervised pre-training, thereby improving the transfer of general-purpose code models to the XML-SQL domain. Specifically, we divide the XML-SQL code into two types of grammatical components: structure components and value components. During pre-training, we undertake three tasks, each designed to learn the internal information of these grammatical components and the relationships between them, enabling the pre-training process to better incorporate previously unlearned domain-specific knowledge of XML-SQL code. Our experimental results show that our trained model XSQLT5-base (220M) improves accuracy by 13.8% compared to the similarly sized CodeT5-base (220M). Additionally, our experiments reveal that ChatGPT, due to its inability to fully learn the XML-SQL domain knowledge, achieves a much lower generation accuracy even with few-shot samples compared to our XSQLT5-base (220M) model.

Functionality-specific vulnerabilities, which mainly occur in Application Programming Interfaces (APIs) with specific functionalities, are crucial for software developers to detect and avoid. When detecting individual functionality-specific vulnerabilities, the existing two categories of approaches are ineffective because they consider only the API bodies and are unable to handle diverse implementations of functionality-equivalent APIs. To effectively detect functionality-specific vulnerabilities, we propose APISS, the first approach to utilize API doc strings and signatures instead of API bodies. APISS first retrieves functionality-equivalent APIs for APIs with existing vulnerabilities and then migrates Proof-of-Concepts (PoCs) of the existing vulnerabilities for newly detected vulnerable APIs. To retrieve functionality-equivalent APIs, we leverage a Large Language Model for API embedding to improve the accuracy and address the effectiveness and scalability issues suffered by the existing approaches. To migrate PoCs of the existing vulnerabilities for newly detected vulnerable APIs, we design a semi-automatic schema to substantially reduce manual costs. We conduct a comprehensive evaluation to empirically compare APISS with four state-of-the-art approaches of detecting vulnerabilities and two state-of-the-art approaches of retrieving functionality-equivalent APIs. The evaluation subjects include 180 widely used Java repositories using 10 existing vulnerabilities, along with their PoCs. The results show that APISS effectively retrieves functionality-equivalent APIs, achieving a Top-1 Accuracy of 0.81 while the best of the baselines under comparison achieves only 0.55. APISS is highly efficient: the manual costs are within 10 minutes per vulnerability and the end-to-end runtime overhead of testing one candidate API is less than 2 hours. APISS detects 179 new vulnerabilities and receives 60 new CVE IDs, bringing high value to security practice.

To address security vulnerabilities arising from third-party libraries, security researchers maintain databases monitoring and curating vulnerability reports. Application developers can identify libraries affected by vulnerability reports (in short, affected libraries) by directly querying the databases with their used libraries. However, the querying results of affected libraries are not reliable due to the incompleteness of vulnerability reports. Thus, current approaches model the task of identifying affected libraries as a named-entity-recognition (NER) task or an extreme multi-label learning (XML) task. These approaches suffer from highly inaccurate results in identifying affected libraries with complex and similar names, e.g., Java libraries. To address these limitations, in this article, we propose VulLibMiner, the first to identify affected libraries from textual descriptions of both vulnerabilities and libraries, together with VulLib, a Java vulnerability dataset with their affected libraries. VulLibMiner consists of a TF-IDF matcher to efficiently screen out a small set of candidate libraries and a BERT-FNN model to effectively identify affected libraries from these candidates. We evaluate VulLibMiner using four state-of-the-art/practice approaches of identifying affected libraries on both their dataset named VeraJava and our VulLib dataset. Our evaluation results show that VulLibMiner can effectively identify affected libraries with an average F1 score of 0.669 while the state-of-the-art/practice approaches achieve only 0.547.

Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. After being pre-trained on a large-scale corpus of code, a model is further fine-tuned with datasets specifically for the target downstream task, e.g., generating code from natural language description. The target code being generated can be classified into two types: a standalone function, i.e., a function that invokes or accesses only built-in functions and standard libraries, and a non-standalone function, i.e., a function that invokes or accesses user-defined functions or third-party libraries. To effectively generate code especially non-standalone functions (largely ignored by existing work), in this article, we present Wenwang, an approach to improving the capability of a pre-trained model on generating code beyond standalone functions. Wenwang consists of two components: a fine-tuning dataset named WenwangData and a fine-tuned model named WenwangCoder. Compared with existing fine-tuning datasets, WenwangData additionally covers non-standalone functions. Besides the docstring and code snippet for a function, WenwangData also includes its contextual information collected via program analysis. Based on PanGu-Coder, we produce WenwangCoder by fine-tuning PanGu-Coder on WenwangData with our context-aware fine-tuning technique so that the contextual information can be fully leveraged during code generation. On CoderEval and HumanEval, WenwangCoder outperforms three state-of-the-art models with similar parameter sizes (at the scale of around 300M), namely CodeGen, PanGu-Coder, and PanGu-FT. Although WenwangCoder does not outperform ChatGPT on HumanEval, WenwangCoder with smaller model parameter sizes can achieve similar effects to ChatGPT on CoderEval. Our experimental results also shed light on a number of promising optimization directions based on existing pre-trained models.

Large Language Models (LLMs) such as GPT-4, trained on massive amounts of natural language and source code data, have exhibited remarkable proficiency in automating many aspects of software development and maintenance. As a result, these models have been extensively applied to various Software Service Engineering (SSE) tasks, including software requirement analysis, software coding, software testing, and Artificial Intelligence for IT Operations (AIOps). Despite their widespread adoption, numerous challenges persist in fully utilizing LLMs for SSE, such as the need for integrating domain-specific knowledge to generate project-level code or patches effectively. Furthermore, there remains a lack of clarity on how traditional SSE practices can adapt to support the full lifecycle of LLMs, from the initial training and fine-tuning with domain-specific data to the ongoing inference, application, and maintenance (i.e., LLMOps). Effective LLMOps require new methodologies and tools to manage the unique demands of LLMs, including data handling, model updates, performance monitoring, and scalability. These challenges underscore the need for innovative approaches to manage the integration of LLM capabilities within established SSE frameworks.

Tianyu Chen, Lin Li, ZhuLiuchuan ZhuLiuchuan, Zongyang Li, Xueqing Liu, Guangtai Liang, Qianxiang Wang, Tao Xie. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Numerous third-party libraries introduced into client projects are not actually required, resulting in modern software being gradually bloated. Software developers may spend much unnecessary effort to manage the bloated dependencies: keeping the library versions up-to-date, making sure that heterogeneous licenses are compatible, and resolving dependency conflict or vulnerability issues.

Code generation models based on the pre-training and fine-tuning paradigm have been increasingly attempted by both academia and industry, resulting in well-known industrial models such as Codex, CodeGen, and PanGu-Coder. To evaluate the effectiveness of these models, multiple existing benchmarks (e.g., HumanEval and AiXBench) are proposed, including only cases of generating a standalone function, i.e., a function that may invoke or access only built-in functions and standard libraries. However, non-standalone functions, which typically are not included in the existing benchmarks, constitute more than 70% of the functions in popular open-source projects, and evaluating models' effectiveness on standalone functions cannot reflect these models' effectiveness on pragmatic code generation scenarios (i.e., code generation for real settings of open source or proprietary code).

Framework-based multilingual software development (MLSD) is becoming prevalent in software engineering practice. Despite the advantages, framework-based MLSD also leads to reduced understandability and changeability of multilingual software, due to the introduced cross-language links (XLLs). To help alleviate this problem, there are existing practice and research crafting rules to specify and detect XLLs, but only focusing on specific frameworks. With the intention of coping with the diversity of XLL conventions across different multi-lingual frameworks, this paper proposes Focused, an extensible approach to framework-oriented cross-language link specification and detection. The basic idea is to decouple the two activities of XLL specification and detection as much as possible by mediating between them with a set of DSL-enabled XLL rules, making Focused configurable to different multilingual frameworks. We evaluated Focused on 3 widely-used multilingual frameworks and 15 high-starred open-source projects using these frameworks, showing the expressiveness, effectiveness, and efficiency of Focused.

In collaborative software development, developers generally make code changes and commit the changes to the repositories. Among others, "making small, single-purpose commits" is considered the best practice for making commits, allowing the team to quickly understand the code changes. Rather than following best practices, developers often make tangled commits, which wrap code changes that implement different purposes. Such commits make it difficult for other developers to understand the code changes when conducting subsequent development. Early works on untangling code changes rely on human-specified heuristic rules or features, do not consider context, and are labor intensive. Recent works model the local context of code changes as a graph at the statement level, with statements as nodes and code dependencies as edges, and then cluster the changed statements. However, recent works ignore the hidden dependencies in the global context, e.g. a pair of tangled code changes may have no code dependency, and a pair of untangled code changes may have obvious code dependency. To solve this problem, we focus on detecting hidden dependencies among code changes. We model the global context of code changes as graphs at finer-grained, hierarchical levels, i.e., at both entity and statement levels. Then we propose a Heterogeneous Directed Graph Neural Network (HD-GNN) to detect hidden dependencies among code changes by aggregating the global context in both connected or disconnected entity-level subgraphs that intersected with the code changes. Evaluation of common C # and Java datasets with 1,612 and 14k tangled commits and manually validated datasets (MVD) with 600 commits shows that HD-GNN achieves an average enhancement of effectiveness of 25% and 19.2% compared to existing approaches and far superior to existing approaches in MVD, without sacrificing time efficiency.

Unit testing is an essential activity in software development for verifying the correctness of software components. However, manually writing unit tests is challenging and time-consuming. The emergence of Large Language Models (LLMs) offers a new direction for automating unit test generation. Existing research primarily focuses on closed-source LLMs (e.g., ChatGPT and CodeX) with fixed prompting strategies, leaving the capabilities of advanced open-source LLMs with various prompting settings unexplored. Particularly, open-source LLMs offer advantages in data privacy protection and have demonstrated superior performance in some tasks. Moreover, effective prompting is crucial for maximizing LLMs' capabilities. In this paper, we conduct the first empirical study to fill this gap, based on 17 Java projects, five widely-used open-source LLMs with different structures and parameter sizes, and comprehensive evaluation metrics. Our findings highlight the significant influence of various prompt factors, show the performance of open-source LLMs compared to the commercial GPT-4 and the traditional Evosuite, and identify limitations in LLM-based unit test generation. We then derive a series of implications from our study to guide future research and practical use of LLM-based unit test generation.

Although existing techniques have proposed automated approaches to alleviate the path explosion problem of symbolic execution, users still need to optimize symbolic execution by applying various searching strategies carefully. As existing approaches mainly support only coarse-grained global searching strategies, they cannot efficiently traverse through complex code structures. In this paper, we propose Eunomia, a symbolic execution technique that supports fine-grained search with local domain knowledge. Eunomia uses Aes, a DSL that lets users specify local searching strategies for different parts of the program. Eunomia also isolates the context of variables for different local searching strategies, avoiding conflicts. We implement Eunomia for WebAssembly, which can analyze applications written in various languages. Eunomia is the first symbolic execution engine that supports the full features of WebAssembly. We evaluate Eunomia with a microbenchmark suite and six real-world applications. Our evaluation shows that Eunomia improves bug detection by up to three orders of magnitude. We also conduct a user study that shows the benefits of using Aes. Moreover, Eunomia verifies six known bugs and detects two new zero-day bugs in Collections-C.

Framework-based multi-lingual software is increasingly prevalent, but it also brings negative effects and extra burden on software maintenance and evolution, because of the introduced cross-language code coupling, which are usually mixed with framework-specific conventions. Researchers have proposed various approaches to code coupling detection, but there is still a lack of necessary support for cross-language coupling detection in framework-based software development. In this paper, we present a preliminary study about cross-language coupling detection in software development based on the Android application framework. We investigate the characteristics of multi-lingual changes in the top-100 starred open-source Android repositories on GitHub, and find that multi-lingual commits are non-trivial: their code changes are more scattered, and more inclined to introduce bugs than other commits. To mitigate the side-effect of multi-lingual development, we propose Grace, a Graph-based cross-language co-change suggestion approach for Android application development. Grace (a) designs a language-agnostic graph to represent code elements from different languages, and (b) employs an entity-based collaborative filtering algorithm to detect and rank candidates of cross-language code couplings, from the graph representation of the latest version as well as the historical multi-lingual commits of a repository. To evaluate the effectiveness of Grace, we apply it to the two tasks of cross-language co-change suggestion and inconsistency checking. Results show that Grace (a) can effectively suggest cross-language co-changed files and types, and (b) can also find existing and potential bugs or code smells caused by inconsistent co-changes.

In collaborative software development, it is considered to be a best practice to submit code changes as a sequence of cohesive commits, each of which records the work result of a specific development activity, such as adding a new feature, bug fixing, and refactoring. However, rather than following this best practice, developers often submit a set of loosely-related changes serving for different development activities as a composite commit, due to the tedious manual work and lack of effective tool support to decompose such a tangled changeset. Composite commits often obfuscate the change history of software artifacts and bring challenges to efficient collaboration among developers. To encourage activity-oriented commits, we propose SmartCommit, a graph-partitioning-based interactive approach to tangled changeset decomposition that leverages not only the efficiency of algorithms but also the knowledge of developers. To evaluate the effectiveness of our approach, we (1) deployed SmartCommit in an international IT company, and analyzed usage data collected from a field study with 83 engineers over 9 months; and (2) conducted a controlled experiment on 3,000 synthetic composite commits from 10 diverse open-source projects. Results show that SmartCommit achieves a median accuracy between 71–84% when decomposing composite commits without developer involvement, and significantly helps developers follow the best practice of submitting activity-oriented commits with acceptable interaction effort and time cost in real collaborative software development.

The wide adoption of third-party libraries in software projects is beneficial but also risky. An already-adopted third-party library may be abandoned by its maintainers, may have license incompatibilities, or may no longer align with current project requirements. Under such circumstances, developers need to migrate the library to another library with similar functionalities, but the migration decisions are often opinion-based and sub-optimal with limited information at hand. Therefore, several filtering-based approaches have been proposed to mine library migrations from existing software data to leverage "the wisdom of crowd," but they suffer from either low precision or low recall with different thresholds, which limits their usefulness in supporting migration decisions. In this paper, we present a novel approach that utilizes multiple metrics to rank and therefore recommend library migrations. Given a library to migrate, our approach first generates candidate target libraries from a large corpus of software repositories, and then ranks them by combining the following four metrics to capture different dimensions of evidence from development histories: Rule Support, Message Support, Distance Support, and API Support. We evaluate the performance of our approach with 773 migration rules (190 source libraries) that we borrow from previous work and recover from 21,358 Java GitHub projects. The experiments show that our metrics are effective to help identify real migration targets, and our approach significantly outperforms existing works, with MRR of 0.8566, top-1 precision of 0.7947, top-10 NDCG of 0.7702, and top-20 recall of 0.8939. To demonstrate the generality of our approach, we manually verify the recommendation results of 480 popular libraries not included in prior work, and we confirm 661 new migration rules from 231 of the 480 libraries with comparable performance. The source code, data, and supplementary materials are provided at: https://github.com/hehao98/MigrationHelper.

In modern software development, developers rely on version control systems like Git to collaborate in the branch-based development workflow. One downside of this workflow is the conflicts occurred when merging contributions from different developers: these conflicts are tedious and error-prone to be correctly resolved, reducing the efficiency of collaboration and introducing potential bugs. The situation becomes even worse, with the popularity of refactorings in software development and evolution, because current merging tools (usually based on the text or tree structures of source code) are unaware of refactorings. In this paper, we present IntelliMerge, a graph-based refactoring-aware merging algorithm for Java programs. We explicitly enhance this algorithm's ability in detecting and resolving refactoring-related conflicts. Through the evaluation on 1,070 merge scenarios from 10 popular open-source Java projects, we show that IntelliMerge reduces the number of merge conflicts by 58.90% comparing with GitMerge (the prevalent unstructured merging tool) and 11.84% comparing with jFSTMerge (the state-of-the-art semi-structured merging tool) without sacrificing the auto-merging precision (88.48%) and recall (90.22%). Besides, the evaluation of performance shows that IntelliMerge takes 539 milliseconds to process one merge scenario on the median, which indicates its feasibility in real-world applications.

API-library switching is to change an underlying API library to another one for a given program. API-library switching is frequent in software-development process, with typical examples as migrating programs across platforms or across incompatible versions of API libraries. Manual API-library switching is time-consuming and error-prone. To address this problem, previous research has proposed a series of rule based code-transformation approaches. However, these existing approaches are ineffective in conducting code transformation for those original code snippets containing a sequence of method invocations correlated with each other and in need of being considered together when transformed. We call the pair of an original code snippet and its transformed target code snippet as a compositional-mapping instance. Through our empirical investigation, we confirm that compositional-mapping instances widely exist during API-library switching. To handle compositional-mapping instances, in this paper, we propose a novel code-transformation approach named as CMIT. Our evaluations compare CMIT with an existing rule-based code transformation approach. The evaluation results show that, compared to the existing approach, CMIT is more flexible, and CMIT increases the recall on average by 33.8% without decreasing the precision.

With the rapid growth of cloud computing, developing business applications on Platform-as-a-Service (PaaS) systems is increasingly popular among industry companies. Various services are developed to support different business requirements on PaaS systems. However, to the best of our knowledge, currently there is no service that provides mobile social messaging services to enable users of their apps to share messages in their social networks (e.g., We Chat, Whats App, Kaka Talk). Lack of such mobile social messaging services prevents industry companies from succeeding in drastic market competitions (e.g., Capture high customer satisfactory). In this paper, we propose a service-based framework to enable the mobile social messaging in PaaS systems (e.g., IBM Blue mix). Using this framework, developers can focus on the service encapsulation of existing applications, and define their business process flows via the conversation management in our platform (no coding work is needed). As such, our framework can effectively reduce the development workload for mobile social messaging in PaaS systems.

Lightweight static bug-detection tools such as FindBugs, PMD, Jlint, and Lint4j detect bugs with the knowledge of generic bug patterns (e.g., objects of java.io.InputStream are not closed in time after used). Besides generic bug patterns, different projects under analysis may have some project-specific bug patterns. For example, in a revision of the Xerces project, the class field "fDTDHandler" is dereferenced without proper null-checks, while it could actually be null at runtime. We name such bug patterns directly related to objects instantiated in specific projects as Project-Specific Bug Patterns (PSBPs). Due to lack of such PSBP knowledge, existing tools usually fail in effectively detecting most of this kind of bugs. We name bugs belonging to the same project and sharing the same PSBP as sibling bugs. If some sibling bugs are fixed in a fix revision but some others remain, we treat such fix as an incomplete fix. To address such incomplete fixes, we propose a PSBP-based approach for detecting sibling bugs and implement a tool called Sibling-Bug Detector (SBD). Given a fix revision, SBD first infers the PSBPs implied by the fix revision. Then, based on the inferred PSBPs, SBD detects their related sibling bugs in the same project. To evaluate SBD, we apply it to seven popular open-source projects. Among the 108 warnings reported by SBD, 63 of them have been confirmed as real bugs by the project developers, while two existing popular static detectors (FindBugs and PMD) cannot report most of them.

Recently many popular websites such as Twitter and Flickr expose their data through web service APIs, enabling third-party organizations to develop client applications that provide function-alities beyond what the original websites offer. These client appli-cations should follow certain constraints in order to correctly in-teract with the web services. One common type of such constraints is Dependency Constraints on Parameters. Given a web service operation O and its parameters Pi, Pj, these constraints describe the requirement on one parameter Pi that is dependent on the conditions of some other parameter(s) Pj. For example, when requesting the Twitter operation "GET statuses/user_timeline", a user_id parameter must be provided if a screen_name parameter is not provided. Violations of such constraints can cause fatal errors or incorrect results in the client applications. However, these con-straints are often not formally specified and thus not available for automatic verification of client applications. To address this issue, we propose a novel approach, called INDICATOR, to automatically infer dependency constraints on parameters for web services, via a hybrid analysis of heterogeneous web service artifacts, including the service documentation, the service SDKs, and the web services themselves. To evaluate our approach, we applied INDICATOR to infer dependency constraints for four popular web services. The results showed that INDICATOR effectively infers constraints with an average precision of 94.4% and recall of 95.5%.

Failing to release unneeded system resources such as I/O streams can result in resource leaks, which can lead to performance degradation and system crashes. Existing resource-leak detectors are usually based on predefined defect patterns to detect resource leaks in software. However, they typically report too many false positives and negatives, and also lack effective warning prioritization. Our empirical investigation shows that, their predefined defect patterns are not precise enough, and moreover, their used defect detection processes are not suitable enough for the defect patterns. In our approach, we introduce a novel Expressive Defect Pattern Specification Notation (EDPSN). With EDPSN, a resource-leak defect pattern can be defined more precisely by specifying conditional method calls and more expressively by including guiding information for the defect detection and warning prioritization process, such as the characteristics of its preferred defect detection process and the effective prioritization impact factors for its related warnings. Based on the EDPSN-based defect pattern, our approach tries to flexibly tune out a suitable defect detection and warning prioritization process. Through evaluations on three real-world projects (Eclipse-3.0.1, JBoss-3.0.6, and Weka-3.6.4), we show that our approach achieves high average precision (96%) and recall (74%), 26% and 49% higher than existing approaches, respectively.

Software systems commonly use resources such as network connections or external file handles. Once finish using the resources, the software systems must release these resources by explicitly calling specific resource-releasing API methods. Failing to release resources properly could result in resource leaks or even outright system failures. Existing verification techniques could analyze software systems to detect defects related to failing to release resources. However, these techniques require resource-releasing specifications for specifying which API method acquires/releases certain resources, and such specifications are not well documented in practice, due to the large amount of manual effort required to document them. To address this issue, we propose an iterative mining approach, called RRFinder, to automatically mining resource-releasing specifications for API libraries in the form of (resource-acquiring, resource-releasing) API method pairs. RRFinder first identifies resource-releasing API methods, for which RRFinder then identifies the corresponding resource-acquiring API methods. To identify resource-releasing API methods, RRFinder performs an iterative process including three steps: model-based prediction, call-graph-based propagation, and class-hierarchy-based propagation. From heterogeneous information (e.g., source code, natural language), the model-based prediction employs a classification model to predict the likelihood that an API method is a resource-releasing method. The call-graph-based and class-hierarchy-based propagation propagates the likelihood information across methods. We evaluated RRFinder on eight open source libraries, and the results show that RRFinder achieved an average recall of 94.0% with precision of 86.6% in mining resource-releasing specifications, and the mined specifications are useful in detecting resource leak defects.

Due to the benefits of cloud computing, many desktop applications have been migrated into the cloud. In order to program in the cloud, lots of IDEs are also moved into the cloud at recent years. Although online IDE can bring developers a lot of convenience for their developing process, some tough problems are still less touched. This paper summarized three main kinds of actual problems from three aspects (Function implementation, Security guarantee, Advanced utilization) encountered when using the online IDE, and proposed three solutions (Services composition, Program behavior analysis, Program behavior mining) to handle these according problems. Finally, this paper introduce a real online IDE (CEclipse) developed by our research group and apply the three solutions in this online IDE.

In order to improve ineffective warning prioritization of static analysis tools, various approaches have been proposed to compute a ranking score for each warning. In these approaches, an effective training set is vital in exploring which factors impact the ranking score and how. While manual approaches to build a training set can achieve high effectiveness but suffer from low efficiency (i.e., high cost), existing automatic approaches suffer from low effectiveness. In this paper, we propose an automatic approach for constructing an effective training set. In our approach, we select three categories of impact factors as input attributes of the training set, and propose a new heuristic for identifying actionable warnings to automatically label the training set. Our empirical evaluations show that the precision of the top 22 warnings for Lucene, 20 for ANT, and 6 for Spring can achieve 100% with the help of our constructed training set.