Malware refers to malicious software or a component of software intended for malicious purposes. The manual analysis and detection of malicious software is challenging due to its complexity. Thus, several automated solutions have become popular for real-time malware detection. A spread-out approach consists of generating images from the samples bytecode and giving them to convolutional neural networks (CNNs), which are used either as classifiers or feature extractors for further classification algorithms. These systems perform extremely well when trained and tested on partitions of the same dataset. However, cross-dataset tests and malware detection verification on emerging real-world samples are required in the real-world context. This is a crucial challenge when probing the robustness of the systems and models. This paper proposes CNN-AutoMIC,a robust automated approach to extract features from malware images. CNN-AutoMIC employs a specific CNN architecture to extract features, followed by an autoencoder-based compressor that reduces features to two fundamental components. The two-dimensional projection of these components is the basis of the predictions performed by the K-nearest neighbors (K-NN) algorithm. Moreover, the observable placement of new samples on the obtained scatter plot makes it possible to explain why the AI-based system produced a certain prediction. It was benchmarked against several CNN-based models and a Vision Transformer. They were trained on the Malevis dataset and cross-dataset evaluated on four different real-world datasets. CNN-AutoMIC outperformed the competitors for each classification performance metric, while requiring a reasonable training and prediction time. In addition, it achieves a promising Akaike information criterion (AIC) score, indicating its efficiency in terms of model complexity.

Cyber threats, primarily malware, have increased with rapid technological advancements in various fields. This growing complexity requires sophisticated and automated malware detection tools because traditional methods cannot keep up with the sheer volume of threats and their evolution. Detection mechanisms that are resilient against evolved malware behaviors, which are typically described by application programming interface (API) functions, are essential for real-time system protection. This paper presents APIARY, an innovative API-based Automatic Rule generator for the YARA tool, designed to enhance malware identification through customized signatures based on peculiar API-based patterns. It discovers distinctive APIs that distinguish malware from goodware, regardless of input data coming from dynamic and static analyses of Windows-like executable files. The algorithm assigns relevance scores to each variable and discards less significant features to identify critical malware indicators. In addition, the generation process optimizes the identified malware model categories to increase the detection rate while minimizing the number of rules produced. The experimental results obtained on nine datasets sourced from the literature demonstrate the potential of APIARY to automatically produce highly effective YARA rules in a short time. Moreover, the rules generated outperform those obtained using alternative state-of-the-art algorithms in terms of detection performance. Lastly, unlike competitors, the proposed procedure does not rely on additional malware analysis data, such as network connection attempts or API parameters, achieving a more streamlined and efficient detection process. • APIARY generates YARA rules based on API patterns to enhance malware detection • It tracks key indicators and ignores less relevant ones to produce effective rules. • APIARY ensures a robust detection rate with a minimum number of rules. • It outperforms competitors with fewer input variables and shorter execution time.

As the occurrence of Denial of Service and Distributed Denial of Service (DoS/DDoS) attacks increases, the demand for effective defense mechanisms increases. Recognition of such anomalies in the computer network is commonly performed through network-based intrusion detection and prevention systems (NIDPSs). Although NIDPSs allow the interception of all known attacks, they are not robust to the continuing variation over time of DoS/DDoS anomalies. The machine learning (ML) paradigm provides algorithms that can effectively reduce concept drift due to the evolution of cyber threat data patterns. These methodologies can be exploited for creating effective rules suitable for popular NIDPS engines such as Suricata. This paper proposes a new algorithm called Anomaly2Sign, which automatically produces rules for Suricata through an automatic Decision Tree (DT)-based generation process. The DT is trained on both anomalous and legitimate traffic, allowing the generation process to select anomaly features that can be mapped within the generated rule structure. Additionally, the DT hyperparameters are tuned at execution time to generate a minimal ruleset capable of detecting the largest number of anomalous packets. The proposed algorithm achieves classification metrics in the range of 99.7%–99.9% using the BOUN-DoS and BUET-DDoS datasets, outperforming the compared ML classifiers, i.e., Logistic Regression, Support Vector Machine, and Multi-Layer Perceptron. Furthermore, the leveraged DT model requires a shorter training and prediction time than the previously cited benchmark classifiers. To enforce the selection of the DT model, an analysis of model complexity is undertaken, including the evaluation of the Akaike Information Criterion (AIC) score. As a result of such an evaluation, the DT model achieved the lowest AIC score among the compared approaches denoting its low complexity. Finally, Anomaly2Sign has been compared with Syrius, i.e., an alternative state-of-the-art automatic NIDPS rules generator, obtaining better performance for detection rate and execution time.

Packet classification activity performed by a FireWall (FW) introduces high latency in network communications due to the computation time required to check whether any packet matches one of the FW rules. Such a classification process is done by sequentially checking the list of rules until a match is found or the end of the list is reached. Given the complexity of FW rules in some environments, this latency could become relevant. This problem is addressed by ordering the list of FW rules to minimize the classification latency, where the rules with higher activation frequencies are placed accordingly starting from the top of the list. This is not always feasible because dependency constraints between rules could exist: swapping the positions of dependent rules results in a loss of the integrity of the implemented security policy. For this reason, the FW rule ordering problem belongs to the realm of constrained combinatorial optimization. This paper proposes a two-stage algorithm to address this problem. The first stage performs an innovative topological sorting algorithm aimed at finding an optimal ordering for the constrained rules, taking into account the fact that rule activation frequencies are influenced by inter-packet arrival time, which typically obeys Zipf's law. The second stage employs a genetic algorithm to find the optimal ordering of all rules within the list. The proposed approach is evaluated using different filtering lists of different complexity provided by ClassBench. A comparison with other state-of-the-art algorithms addressing the same problem is performed. Furthermore, the performance analysis is extended employing an exact optimization method. The results obtained show the effectiveness of the proposed algorithm in minimizing packet classification latency, while a short reordering time is required.

Recognition of known malicious patterns through signature-based systems is unsuccessful against malware for which no known signature exists to identify them. These include not only zero-day but also known malicious software able to self-replicate rewriting its own code leaving unaffected its execution, namely metamorphic malware. YARA is a popular malware analysis tool that uses the so-called YARA-rules, which are built to match malicious contents within files or network packets analyzed by an Anti-Virus engine. Sometimes such content is expressed in the form of a byte-signature, i.e., a sequence of operational machine-level code. However, these can be bypassed since malware obfuscation techniques can change these sequences, rewriting them in several equivalent forms. This paper presents YAMME, a YARA-byte-signatures Metamorphic Mutation Engine to strengthen rules against some malware obfuscation techniques deployed in metamorphic mutation engines. First, it rewrites YARA-bye-signatures in several equivalent ways, as a metamorphic mutation engine would do. Second, an optimization phase exploits the YARA-rules syntax constructs to provide several rules formats, making them suitable for different real-world application requirements. YAMME rules have been evaluated on MWOR, G2, NGVCK, and MetaNG datasets, resulting in a better detection rate than that achieved by YARA-rules generated through AutoYara. Furthermore, an analysis of computational overhead required by different YAMME rules formats validates the low impact introduced by the mutation engine at the YARA-rules level.