Network-based anomaly detection plays a pivotal role in cybersecurity. Most detection models are based on unsupervised machine learning to learn such a normal flow pattern of network traffic as the numbers of incoming/outgoing packets, traffic volumes in bytes, duration time, etc., most of which are numerical features. On the contrary, non-numerical features have not been fully utilized yet although they often give a decisive hint to the detection of unseen attacks; for example, rarely observed combinations of IP addresses and port numbers can reveal an uncommon attack attempt. This heuristic has already been used by human experts for decades, but not fully utilized yet by deep learning models. In this paper, we present a new encoding scheme for non-numerical features such as IP addresses and port numbers that might have been mistakenly considered as numerical features. The new encoding scheme first ranks non-numerical features in their frequency order and then evenly places each rank between 0 and 1, which transforms raw data into a form that is easy for machines to understand. The anomaly detection performance is significantly improved when this new encoding scheme is applied to the same deep learning model. For example, a simple autoencoder model with the new encoding scheme achieved the Area Under Receiver Operating Characteristic, AUROC, of 0.99 for the well-known CICIDS2017 dataset while the previous record was 0.91. Experimental results from three different open datasets show that the proposed encoding scheme can significantly enhance the performance of anomaly detection models.

Network-based intrusion detection and data leakage prevention systems inspect packets to detect if critical files such as malware or confidential documents are transferred. However, this kind of detection requires heavy computing resources in reassembling packets and only well-known protocols can be interpreted. Besides, finding similar files from a storage requires pairwise comparisons. In this paper, we present a new network-based file identification scheme that inspects packets independently without reassembly and finds similar files through inverted indexing instead of pairwise comparison. We use a content-based chunking algorithm to consistently divide both files and packets into multiple byte sequences, called chunks. If a packet is a part of a file, they would have common chunks. The challenging problem is that packet chunking and inverted-index search should be fast and scalable enough for packet processing. The file identification should be accurate although many chunks are noises. In this paper, we use a small Bloom filter and a two-level threshold strategy to solve the problems. To the best of our knowledge, this is the first scheme that identifies a specific critical file from a packet over unknown protocols. Experimental results show that the proposed scheme can successfully identify a critical file from a packet without packet reassembly.

Network-based intrusion detection and data leakage prevention systems inspect packets to detect if critical files such as malware or confidential documents are transferred. However, this kind of detection requires heavy computing resources in reassembling packets and only well-known protocols can be interpreted. Besides, finding similar files from a storage requires pairwise comparisons. In this paper, we present a new network-based file identification scheme that inspects packets independently without reassembly and finds similar files through inverted indexing instead of pairwise comparison. We use a contents-based chunking algorithm to consistently divide both files and packets into multiple byte sequences, called chunks. If a packet is a part of a file, they would have common chunks. The challenging problem is that packet chunking and inverted-index search should be fast and scalable enough for packet processing. The file identification should be accurate although many chunks are noises. In this paper, we use a small Bloom filter and a delayed query strategy to solve the problems. To the best of our knowledge, this is the first scheme that identifies a specific critical file from a packet over unknown protocols. Experimental results show that the proposed scheme can successfully identify a critical file from a packet.