In this paper, we propose a novel tracking algorithm, i.e., the discriminative ranking list-based tracker (DRLTracker). The DRLTracker models the target object and its local background by using ranking lists of patches of different scales within object bounding boxes. The ranking list of each of such patches is its K nearest neighbors. Patches of the same scale with ranking lists of high purity values (meaning high probabilities to be on the target object) and some confusable background patches constitute the object model under that scale. A pair of object models of two different scales collaborate to determine which patches may belong to the target object in the next frame. The DRLTracker can effectively alleviate the distraction problem, and its superior ability over several representative and state-of-the-art trackers is demonstrated through extensive experiments.