In recent years there has been a shift from heuristics based malware detection towards machine learning, which proves to be more robust in the current heavily adversarial threat landscape. While we acknowledge machine learning to be better equipped to mine for patterns in the increasingly high amounts of similar-looking files, we also note a remarkable scarcity of the data available for similarity targeted research. Moreover, we observe that the focus in the few related works falls on quantifying similarity in malware, often overlooking the clean data. This one-sided quantification is especially dangerous in the context of detection bypass. We propose to address the deficiencies in the space of similarity research on binary files, starting from EMBER — one of the largest malware classification datasets. We enhance EMBER with similarity information as well as malware class tags, to enable further research in the similarity space. Our contribution is threefold: (1) we publish EMBERSim, an augmented version of EMBER, that includes similarity informed tags; (2) we enrich EMBERSim with automatically determined malware class tags using the open-source tool AVClass on VirusTotal data and (3) we describe and share the implementation for our class scoring technique and leaf similarity method.

Research, especially in Education, is hampered by scale and our aim is to help shift this dynamic by tracking relevant studies from major scientific venues. The current version of our dataset considers top conferences and journals from the domain (N = 33,531). Several heuristics using advanced text searches and regular expressions, together with NLP techniques ranging from part of speech tagging, syntactic dependency parsing, to semantic models, are employed to extract relevant studies. Our method achieves an F1 score of .633 when employed on a manually annotated subset of 1,000 articles. When applied to the entire dataset, the total number of articles with large N was around 10%, with a positive trend in the last years. [email protected] was by far the venue with the highest density of identified articles, thus arguing for its emphasis on scale. Further filtering of the articles is required when focusing only on learning, as the articles span across multiple domains and have multiple interests. Nonetheless, the order of magnitude raises current problems, namely that these studies are scarce and future endeavors should emphasize the importance of scale. Our dataset, including the validation subset, and the corresponding code for data crawling, PDF processing, and large N extraction mechanisms, have been open-sourced to further support the initiative and stimulate new analyses.