This research aims at automatic clustering from a large-scale malware specimen set by constructing a phylogenetic tree. In our previous work, we proposed a scalable method for constructing a phylogenetic tree with a clustering algorithm. In this paper, we will introduce the current progress of our work. We are trying to improve our algorithm to achieve higher clustering accuracy and we are using a much larger IoT malware set containing 182,838 malware specimens to evaluate our method.

Most packets arriving in the darknet (or network telescope), which is unused IP address space on the Internet, are related to indiscriminate scanning and attack activities. In recent years, the number of indiscriminate scanning attacks observed on the darknet has increased in diversity and quantity. In our earlier study, we proposed a framework called Dark-TRACER that detects anomalies in spatiotemporal pattern synchronization by using darknet data, with the aim being early detection of malware-caused indiscriminate scanning attacks. Although Dark-TRACER has achieved an average of 126.4 days earlier threat detection, we have not been able to determine whether there is a relationship between the early detections and the actual threats. Hence, in this paper, we perform a cross-checking analysis to identify if any information links the detections and the actual threats. As a result, we confirmed the validity of our early threat detection framework by showing, e.g., that more than 60% of unique hosts overlapped in large-scale threats. In addition, we outline four future studies to address the issue that the present Dark-TRACER has many false-positive alerts. Lastly, the darknet data used in our research has been made publicly available.

Most IoT malware is variants generated by editing and reusing parts of the functions based on publicly available source codes. In our previous study, we proposed a method to estimate the functions of a specimen using the Function Call Sequence Graph (FCSG), which is a directed graph of execution sequence of function calls. In the FCSG-based method, the subgraph corresponding to a malware functionality is manually created and called a signature-FSCG. The specimens with the signature-FSCG are expected to have the corresponding functionality. However, this method cannot detect the specimens with a slightly different subgraph from the signature-FSCG. This paper found that these specimens were supposed to have the same functionality for a signature-FSCG. These specimens need more flexible signature matching, and we propose a graph embedding technique to realize it.

Research on network intrusion detection (NID) requires a large amount of traffic data with reliable labels indicating which packets are associated with particular network attacks. In this paper, we implement a prototype of an automated system to create labeled packet datasets for NID research. In this paper, we implement a prototype of an automated system to assign labels to packet datasets for NID research. By re-transmitting pre-captured packet data in a controlled network environment pre-installed with a network intrusion detection system, the system automatically assigns labels to attack packets within the packet data. In the feasibility study, we investigate factors that may influence the detection accuracy of the attacking packets and show an example using the prototype to label a packet file. Finally, we show an efficient way to locate the packets associated with issued NID alerts using this prototype.

Malware is increasingly diversified and sophisti-cated. It is essential to rapidly and accurately detect malware activities when malware infection spreads. However, accurately distinguishing potential malware activities from countless indis-criminate scanning attacks is a huge challenge. In this study, we introduce Dark-NMF, a darknet analysis engine using Non-negative Matrix Factorization (NMF). Dark-NMF focuses on synchronizing the spatiotemporal features seen when malware infection spreads and detects abnormally synchronous spatial features (source hosts and destination ports) automatically in near real-time. Dark-NMF measures the synchronization of spatial features by decomposing spatiotemporal patterns from darknet traffic using NMF. We tuned the hyperparameters of Dark- Nmfand evaluated the detection performance of malware activities against the performance of existing methods such as GLASSO and ChangeFinder using a human-labeled ground truth. We found that Dark-NMF detects all malware activities that should be detected in the ground truth without a miss. We also showed that Dark- Nmfhas many advantages over existing methods and provided a highly practical operation guideline. Consequently, Dark-NMF is expected to contribute as threat intelligence information for rapid response to malware activity.