With the increasing sophistication of Advanced Persistent Threats (APTs), the demand for effective detection and mitigation strategies and methods has escalated. Program execution leaves traces in the system audit log, which can be analyzed to detect malicious activities. However, collecting and analyzing large volumes of audit logs over extended periods is challenging, further compounded by insufficient labeling that hinders their usability. Addressing these challenges, this paper introduces SAGA (Synthetic Audit log Generation for APT campaigns), a novel approach for generating find-grained labeled synthetic audit logs that mimic real-world system logs while embedding stealthy APT attacks. SAGA generates configurable audit logs for arbitrary duration, blending benign logs from normal operations with malicious logs based on the definitions the MITRE ATT&CK framework. Malicious audit logs follow an APT lifecycle, incorporating various attack techniques at each stage. These synthetic logs can serve as benchmark datasets for training machine learning models and assessing diverse APT detection methods. To demonstrate the usefulness of synthetic audit logs, we ran established baselines of event-based technique hunting and APT campaign detection using various synthetic audit logs. In addition, we show that a deep learning model trained on synthetic audit logs can detect previously unseen techniques within audit logs.

In this article, we focus on the robustness of behavior-based malware analysis models, justified by the need to address the high mutation rates of malware executables that debilitate conventional signature-based approaches and even behavior-based AI solutions. In response to these challenges, we propose MAMBA + , an obfuscation-resistant dynamic analysis approach tailored for uncovering malware behavior. We have assembled a comprehensive collection of behavioral obfuscation attacks designed to undermine behavior-based models. The central concept behind MAMBA + involves treating obfuscated calls as perturbed data and introducing a novel loss function to effectively balance ground-truth predictions and the handling of these perturbations. To facilitate this approach, MAMBA + designs adapted embedding mechanisms to transform traces of API calls into high-dimensional vectors for attention calculations. Through a comprehensive empirical study with seven obfuscations and three unseen attacks, we reveal important qualitative properties of MAMBA + , and quantitatively demonstrate its superiority in performance and robustness to all compared methods.

Po-Kai Chen, Bo-Wei Tsai, Shao Kuan Wei, Chien-Yao Wang, Jia-Ching Wang, Yi-Ting Huang. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

The task of retrieving known Advanced Persistent Threat (APT) campaigns from system activity graphs, where nodes represent MITRE ATT&CK techniques and edges encode temporal or resource-level relationships, requires reasoning over structures. In operational settings, these target graphs are often noisy due to incomplete detection, technique misclassification, and benign-induced structural artifacts. To address this issue, we formulate the task as approximate subgraph matching between a known APT query graph and a noisy, partially observed technique graph. In this poster, we introduce a preliminary embedding-based retrieval method, aiming to promote it as a robust and practical framework for retrieving known APTs in real-world environments.

As Advanced Persistent Threats (APTs) grow increasingly sophisticated, the demand for effective detection methods has intensified. This study addresses the challenge of identifying APT campaign attacks through system event logs. A cascading approach, name SFM, combines Technique hunting and APT campaign attribution. The approach assumes that real-world system event logs contain a vast majority of normal events interspersed with few suspiciously malicious ones and that the logs are annotated with Techniques of MITRE ATT&CK framework for attack pattern recognition. After identifying Techniques from the log, we attribute APT campaign attacks by aligning detected Techniques with known attack sequences to determine the most likely APT campaign. Evaluations on five synthetic real-world APT campaigns indicate that the proposed approach demonstrates reliable performance.

This paper presents APILI, an innovative approach to behavior-based malware analysis that utilizes deep learning to locate the API calls corresponding to discovered malware techniques in dynamic execution traces. APILI defines multiple attentions between API calls, resources, and techniques, incorporating MITRE ATT&CK framework, adversary tactics, techniques and procedures, through a neural network. We employ fine-tuned BERT for arguments/resources embedding, SVD for technique representation, and several design enhancements, including layer structure and noise addition, to improve the locating performance. To the best of our knowledge, this is the first attempt to locate low-level API calls that correspond to high-level malicious behaviors (that is, techniques). Our evaluation demonstrates that APILI outperforms other traditional and machine learning techniques in both technique discovery and API locating. These results indicate the promising performance of APILI, thus allowing it to reduce the analysis workload.

Cyber Threat Intelligence (CTI) plays a crucial role in understanding and preemptively defending against emerging threats. Typically disseminated through unstructured reports, CTI encompasses detailed insights into threat actors, their actions, and attack patterns. The MITRE ATT&CK framework offers a comprehensive catalog of adversary tactics, techniques, and procedures (TTPs), serving as a valuable resource for deciphering attacker behavior and enhancing defensive measures. Addressing the challenge of time-consuming manual analysis of MITRE TTPs in unstructured CTI reports, this paper presents MITREtrieval, a novel system that leverages deep learning and ontology to efficiently extract MITRE techniques. This approach mitigates issues related to the implicit nature of TTPs, textual semantic dependencies, and the scarcity of adequately labeled datasets, enabling more effective analysis even with limited sample sizes. Our approach combines a sophisticated sentence-level BERT deep learning model with ontology knowledge to address sparse data challenges, using a voting algorithm to merge outcomes. This results in a more accurate classification of MITRE techniques, capturing contextual nuances effectively. Our evaluation confirms MITREtrieval’s effectiveness in identifying techniques, regardless of their representation in training samples. MITREtrieval has surpassed benchmarks, achieving F2 scores of 58%, 62%, and 69% in multi-label technique identification across 113, 46, and 23 CTI reports, respectively, thereby streamlining CTI analysis and improving threat intelligence.

Cyber threats have become more prevalent than ever. Cyber Threat Intelligence (CTI) reports and MITRE ATTCK® framework play an imperative role in helping experts and organizations assess current and potential attacks, such as Advanced Persistent Threats (APT). However, the task of extracting valuable information from unstructured texts remains an ongoing challenge. In this work, we present a framework for understanding and reasoning adversary tactics and techniques. We construct an ontology structure and propose an automatic information extraction method that is capable of integrating the parsed information from CTI reports into each instance. The ontology is represented in the Web Ontology Language (OWL) accessible with the SPARQL query language. Our evaluation shows that the proposed information extraction method outperforms other state-of-the-art neural network-based methods in terms of precision. Furthermore, our framework can effectively infer adversary information, which efficiently supports security analysts recognize tactics and techniques.

Cyber threats are one of the most pressing issues in the digital age. There has been a consensus on deploying a proactive defense to effectively detect and respond to adversary threats. The key to success is understanding the characteristics of malware, including their activities and manipulated resources on the target machines. The MITRE ATT&CK framework (ATT&CK), a popular source of open source intelligence (OSINT), provides rich information and knowledge about adversary lifecycles and attack behaviors. The main challenges of this study involve knowledge collection from ATT&CK, malicious behavior identification using deep learning, and the identification of associated API calls. A MITRE ATT&CK based Malicious Behavior Analysis system (MAMBA) for Windows malware is proposed, which incorporates ATT&CK knowledge and considers attentions on manipulated resources and malicious activities in the neural network model. To synchronize ATT&CK updates in a timely manner, knowledge collection can be an automatic and incremental process. Given these features, MAMBA achieves the best performance of malicious behavior discovery among all the compared learning-based methods and rule-based approaches on all datasets; it also yields a highly interpretable mapping from the discovered malicious behaviors to relevant ATT&CK techniques, as well as to the related API calls.

Abstract We describe the development, pilot-testing, refinement, and four evaluations of Diagnostic Question Generator (DQGen), which automatically generates multiple choice cloze (fill-in-the-blank) questions to test children's comprehension while reading a given text. Unlike previous methods, DQGen tests comprehension not only of an individual sentence but of the context preceding it. To test different aspects of comprehension, DQGen generates three types of distractors: ungrammatical distractors test syntax; nonsensical distractors test semantics; and locally plausible distractors test inter-sentential processing. (1) A pilot study of DQGen 2012 evaluated its overall questions and individual distractors, guiding its refinement into DQGen 2014. (2) Twenty-four elementary students generated 200 responses to multiple choice cloze questions that DQGen 2014 generated from forty-eight stories. In 130 of the responses, the child chose the correct answer. We define thedistractivenessof a distractor as the frequency with which students choose it over the correct answer. The incorrect responses were consistent with expected distractiveness: twenty-seven were plausible, twenty-two were nonsensical, fourteen were ungrammatical, and seven were null. (3) To compare DQGen 2014 against DQGen 2012, five human judges categorized candidate choices without knowing their intended type or whether they were the correct answer or a distractor generated by DQGen 2012 or DQGen 2014. The percentage of distractors categorized as their intended type was significantly higher for DQGen 2014. (4) We evaluated DQGen 2014 against human performance based on 1,486 similarly blind categorizations by twenty-seven judges of sixteen correct answers, forty-eight distractors generated by DQGen 2014, and 504 distractors authored by twenty-one humans. Surprisingly, DQGen 2014 did significantly better than humans at generating ungrammatical distractors and marginally better than humans at generating nonsensical distractors, albeit slightly worse at generating plausible distractors. Moreover, vetting DQGen 2014's output and writing distractors only when necessary would halve the time to write them all, and produce higher quality distractors.

Over the last few decades, there has been a growing interest in technology assisted multimedia learning. The purpose of this study was to develop an online multimedia learning system, TEDQuiz, for English language learners to practice their listening skills. The system is comprised of an easy-to-use browser extension and a personalized online learning management platform. This can help learners to manage listening materials by automatically generating multiple -- choice questions, synchronously connecting to friends in a social network website and personally managing their learning progress and word bank. The use of the system involves six steps: skimming, asking questions, listening, answering questions, linking and reviewing. Taking TED talks as an example, users can skim the website, have guiding questions, listen to talks, answer the questions, link to a social network website, and review their profile on the TEDQuiz website. The experimental results demonstrate that students with the TEDQuiz system spent more time on video watching. In the questionnaire, students had a positive view of the functions of the TEDQuiz system. They thought it is useful and helpful, and they were willing to use it in the future. We also found that the usefulness factor was statistically significant in predicting the future usage of TEDQuiz.

In the last few years, researchers in the field of e-learning and Natural Language Processing (NLP) have shown an increased interest in automatic question generation. However, little research has discussed the automatic evaluation of listening comprehension in multimedia learning. In this work, we present an automatic quiz generation for TED Talks video clips, called TED Quiz. TED Quiz generates multiple-choice questions in two question types, gist-content questions and detail questions. We use a graph-based algorithm, Lex Rank, to identify the most important part of a talk, as the main concept of a gist-content question. We also proposed an approach to distractor selection for detail question generation that generates grammatically correct but semantically wrong sentences as distractors. The experimental results demonstrated that the measured results from automatically generated questions are comparable with that from manually generated questions because their scores were significantly correlated. Moreover, most subjects agreed that the generated listening comprehension questions were of quality and usefulness.

In high-tech plants, the manufacturing stability and product quality are monitored through periodic sampling. As for those non-sampled workpieces, their quality is commonly monitored by a fault detection and classification (FDC) method. Nevertheless, it may fail to detect out-of-control (OOC) products if their corresponding manufacturing process parameters are all in-spec. In other words, unless those certain defected workpieces are selected for sampling measurements, they may not be detected through simply monitoring all the individual manufacturing process parameters. We have proposed a product quality fault detection scheme (FDS), which utilizes the classification and regression tree (CART) to build a single failure model (FML) for identifying the relationship between process parameters and OOC products. However, all the failure modes (FMs) are contained in the single FML, which makes it difficult to understand the causes of defected products. To remedy this problem, this paper develops an advanced fault detection scheme (AFDS). The AFDS builds the corresponding FM by CART for each individual failure cause and generates a FM manager via support vector machine (SVM) to manage all the FMs. Finally, the dual-phase concept is adopted to run the AFDS for achieving on-line real-time fault detection.

With growing interest in estimating true ability in contemporary learning, the demand for personalized learning and Web-based learning environments has become increasingly important. This paper develops a statistical and interpretable method of estimating ability. This method captures the succession of learning over time and provides an explainable interpretation of a statistical measurement, based on Item Response Theory and the quantiles of acquisition distributions. The results from the simulation and empirical study demonstrate that the estimated abilities can successfully recognize the actual abilities of students. The correlation values between the estimated abilities and the post-test score, which incorporate this testing history, are higher than values that only consider test responses at the time of testing. Furthermore, the pre-test and post-test administered to the experimental group show significant student improvement. These results suggest that this method serves as a successful alternative ability estimation and provides a better understanding of student competence.

Recent years have seen increased attention given to computer-aided question generation for language student testing and evaluation. However, this approach often directly provides examinees with exhaustive questions. This is inappropriate, because these questions are not designed for any specific testing purpose. In this work, we present a personalized automatic quiz generation model that generates multiple-choice questions at various difficulty levels and categories, including grammar, vocabulary, and reading comprehension. We combined this model with a quiz strategy for estimating examinee proficiency and question selection. The proficiency is estimated using Exponential Moving Average, combining the test responses with a student’s past history. The results show that the subjects in the experimental group corrected their mistakes more frequently as well as answered more difficult questions than the control group. The experimental group also demonstrated the most progress between the pre-test and post-test. In addition, most of subjects agree the quality of the generated questions in the questionnaire analysis.

Reading difficulty is a measurement for estimating the appropriate reading level of a document. Almost all prior studies are designed for first language learners, but not for second language learners. In this study, we propose a robust estimation scheme, including features such as word frequency, official word grade and grammar patterns, to train a linear model to estimate the difficulty of the document for second language learners. The experiment results show that the proposed estimation scheme outperforms other reading difficulty estimations.

MOTIVATION: The growing availability of full-text scientific articles raises the important issue of how to most efficiently digest full-text content. Although article titles and abstracts provide accurate and concise information on an article's contents, their brevity inevitably entails the loss of detail. Full-text articles provide those details, but require more time to read. The primary goal of this study is to combine the advantages of concise abstracts and detail-rich full-texts to ease the burden of reading. RESULTS: We retrieved abstract-related paragraphs from full-text articles through shared keywords between the abstract and paragraphs from the main text. Significant paragraphs were then recommended by applying a proposed paragraph ranking approach. Finally, the user was provided with a condensed text consisting of these significant paragraphs, allowing the user to save time from perusing the whole article. We compared the performance of the proposed approach with a keyword counting approach and a PageRank-like approach. Evaluation was conducted in two aspects: the importance of each retrieved paragraph and the information coverage of a set of retrieved paragraphs. In both evaluations, the proposed approach outperformed the other approaches. CONTACT: [email protected].

In current semiconductor and TFT-LCD factories, periodic sampling is commonly adopted to monitor the stability of manufacturing processes and the quality of products (or workpieces). As for those non-sampled workpieces, their quality is usually monitored by such as a fault-detection-and-classification (FDC) server. However, this method may fail to detect defected products. For example, a workpiece with all the individual manufacturing process parameters being in-spec may still result in out-of-spec product quality. Under this circumstance, unless this certain defected workpiece is selected for sampling by chance, it cannot be detected by simply monitoring the manufacturing process parameters collected from the production equipment. To solve the above mentioned problem, this research proposes a product quality fault detection scheme (FDS), which utilizes the classification and regression tree to implement a model for identifying the relationship between process parameters and out-of-spec products. Through this model, each set of normal manufacturing process parameters can be real-time and on-line examined to detect failure or defected products.

In this paper, we propose the English dialogue companion (EDC) system to help elementary school students learn second language, as English. Learners can practice conversation with the EDC system in English. In order to motivate the learners, we design the three learning activities to enhance the environment of English learning, which are the choice of learning companion phase, the conversation phase, and the teaching phase. We also conducted a pilot study to evaluate the EDC system. The results showed the subjects agreed that the system is positive and useful. For the most of learners, we found that the most favorite learning activity is to teach learning companion. In addition, the greater part of learners liked the same gender learning companion. And learners selected the learning companion depended on the learning companion is regards as a classmate or a teacher.

In this paper, the authors proposed a mobile video QA (Question Answering) system for online annotation and ubiquitous multimedia learning. Under the wireless and mobile environment, learners could interact with the system through natural language questions rather than traditional query-like text retrieval. The handheld device is not as well as traditional keyboard input, while the annotation is quite inefficient and inconvenient. To solve this, the authors designed a simple browsing and clicking model to enable learners annotating and auto-feedback as e-mail to learner's mail box. A pilot study was conducted to evaluate the preliminary experimental results for learning. The experimental results showed that the proposed system effectively engaged learners in multimedia video QA in handheld device where all subjects agreed that the system is positive and useful.

In this paper, the authors present a video content summarization for recommendation (called VCSR) system to auto-recommend suitable multimedia learning materials for learners. The VCSR system firstly extracts important content as summarization from input raw video data, while the generated summarization will be auto-routed to learners according to their profiles. Video captions are initially recognized using optical character recognition (OCR), then a set of key passages with corresponding frame images are extracted to form a video summary. The recommendation is achieved by calculating the relevance of the video summarization for each learner. Also, this paper indicates how the VCSR system effectively plays the intermediate role in a modern digital library.