Android security has received a lot of attention over the last decade, especially malware investigation. Researchers attempt to highlight applications’ security-relevant characteristics to better understand malware and effectively distinguish malware from benign applications. The accuracy and the completeness of their proposals are evaluated experimentally on malware and goodware datasets. Thus, the quality of these datasets is of critical importance: if the datasets are outdated or not representative of the studied population, the conclusions may be flawed. We specify different types of experimental scenarios. Some of them require unlabeled but representative datasets of the entire population. Others require datasets labeled with valuable characteristics that may be difficult to compute, such as malware datasets. We discuss the irregularities of datasets used in experiments, questioning the validity of the performances reported in the literature. This article focuses on providing guidelines for designing debiased datasets. First, we propose guidelines for building representative datasets from unlabeled ones. Second, we propose and experiment a debiasing algorithm that, given a biased labeled dataset and a target representative dataset, builds a representative and labeled dataset. Finally, from the previous debiased datasets, we produce datasets for experiments on Android malware detection or classification with machine learning algorithms. Experiments show that debiased datasets perfom better when classifying with machine learning algorithms.

Experimenting with Android malware requires to manipulate a large amount of samples and to chain multiple analyses. Scripting such a sequence of analyses on a large malware dataset becomes a challenge: the analysis has to handle fails on the computer and crashes on the used smartphone, in case of dynamic analyses. We present a new tool, PyMaO, for handling such experiments on a regular desktop PC with the highest performance throughput. PyMaO helps to write sequences of analyses and handle partial experiments that should be restarted after a crash or continued with new unknown analyses. The tool also offers a post processing capability for generating number tables or bar graphs from the analyzed datasets.