Along with gaining popularity of Real-Time Bidding (RTB) based programmatic advertising, the click farm based invalid traffic, which leverages massive real smartphones to carry out large-scale ad fraud campaigns, is becoming one of the major threats against online advertisement. In this study, we take an initial step towards the detection and large-scale measurement of the click farm based invalid traffic. Our study begins with a measurement on the device's features using a real-world labeled dataset, which reveals a series of features distinguishing the fraudulent devices from the benign ones. Based on these features, we develop EvilHunter, a system for detecting fraudulent devices through ad bid request logs with a focus on clustering fraudulent devices. EvilHunter functions by 1) building a classifier to distinguish fraudulent and benign devices; 2) clustering devices based on app usage patterns; and 3) relabeling devices in clusters through majority voting. EvilHunter demonstrates 97% precision and 95% recall on a real-world labeled dataset. By investigating a super click farm, we reveal several cheating strategies that are commonly adopted by fraudulent clusters. We further reduce the overhead of EvilHunter and discuss how to deploy the optimized EvilHunter in a real-world system. We are in partnership with a leading ad verification company to integrate EvilHunter into their industrial platform.