This paper investigates the problem of classifying Android applications into malicious and benign. We analyze the performance of a popular malware detection tool, Drebin, and show that its correct classification decisions often stem from using benign rather than malicious features for making predictions. That, effectively, turns the classifier into a benign app detector rather than a malware detector. While such behavior allows the classifier to achieve a high detection accuracy, it also makes it vulnerable to attacks, e.g., by a malicious app pretending to be benign by using features similar to those of benign apps. In this paper, we propose an approach for deprioritizing benign features in malware detection, focusing the detection on truly malicious portions of the apps. We show that our proposed approach makes a classifier more resilient to attacks while still allowing it to maintain a high detection accuracy.