Visual tracking algorithms have important robotic applications such as mobile robot guidance and servoed wide area surveillance systems. These applications ideally require vision algorithms which are robust to camera motion and scene change but are cheap and fast enough to run on small, low power embedded systems. Unfortunately most robust visual tracking algorithms are either computationally expensive or are restricted to a stationary camera. This paper describes a new color based tracking algorithm, the Adaptive Background CAMSHIFT (ABCshift) tracker and an associated technique, mean shift servoing, for efficient pan-tilt servoing of a motorized camera platform. ABCshift achieves robustness against camera motion and other scene changes by continuously relearning its background model at every frame. This also enables robustness in difficult scenes where the tracked object moves past backgrounds with which it shares significant colors. Despite this continuous machine learning, ABCshift needs minimal training and is remarkably computationally cheap. We first demonstrate how ABCshift tracks robustly in situations where related algorithms fail, and then show how it can be used for real time tracking with pan-tilt servo control using only a small embedded microcontroller.