Fast and accurate visual search is an enabler for many applications of drones. Prior works use POMDPs to produce effective search strategies. As the observation models are from heuristics, the robustness of these approaches on the field is unclear. This work builds a testbed that combines latest developments in related areas, including mobile CNNs for inference on mobile platforms and policy search with point based methods, in a POMDP framework. A dataset for a simple but realistic application, search for a single basketball, is collected to train the perception modules, investigate their error characteristics and validate the control algorithm. From simulation using realistic parameters, we found the significant role persistent factors in the environment can play in designing a fast search strategy. Failure to taking these factors into account results in up to 60% longer search time at the same success rate. Our empirical tests using mobile CNN and real data reveals that prior assumptions on error rates as functions of heights are wrong. The errors grows non-linearly, and there is significant between false positive and false negative rates. Our findings shed new lights on what to consider in designing visual search strategies in a drone platform and is one step towards a fast and robust algorithm.