Background subtraction is the first step of many video surveillance applications. What is considered background varies by application, and may include regular, systematic, or complex motions. The paper explores the use of several different local spatio-temporal models of a background, defined at each pixel in the image. We present experiments with real image data and conclude that appropriate local representations are sufficient to make background models of complicated real world motions. Empirical studies illustrate, for example, that an optical flow-based model is able to detect emergency vehicles whose motion is different from those typically observed in traffic scenes. We conclude that "different models are appropriate for different scenes", but give criteria by which one can choose which model will be best.