We present a background model that differentiates between background motion and foreground objects. Unlike most models that represent the variability of pixel intensity at a particular location in the image, we model the underlying warping of pixel locations arising from background motion. The background is modeled as a set of warping layers, where at any given time, different layers may be visible due to the motion of an occluding layer. Foreground regions are thus defined as those that cannot be modeled by some composition of some warping of these background layers. We illustrate this concept by first reducing the possible warps to those where the pixels are restricted to displacements within a spatial neighborhood, and then learning the appropriate size of that spatial neighborhood. Then, we show how changes in intensity/color histograms of pixel neighborhoods can be used to discriminate foreground and background regions. We find that this approach compares favorably with the state of the art, while requiring less computation.

Imaging sensors, or “imagers,” embedded in the natural environment enable remote collection of large quantities of data, thus easing the design and deployment of sensing systems in a variety of application domains. Yet, the data collected from such imagers are difficult to interpret due to a variety of “nuisance factors” in the data formation process, such as illumination, vantage point, partial occlusions, etc. These are especially severe in natural environments, where the objects of interest (e.g., plants, animals) have evolved to blend with their habitat, exhibit complex variability in shape and appearance, and perform rapid motions against dynamic backgrounds with rapid illumination changes. We describe three applications that exemplify these problems and the solutions we developed. First, we show how temporal oversampling can simplify the analysis of a slow process such as the avian nesting cycle. Then, we show how to overcome temporal undersampling in order to detect birds at a feeder station. Finally, we show how to exploit temporal consistency to reliably detect pollinators as they visit flowers in the field.

We present a scalable end-to-end system for vision-based monitoring of natural environments, and illustrate its use for the analysis of avian nesting cycles. Our system enables automated analysis of thousands of images, where manual processing would be infeasible. We automate the analysis of raw imaging data using statistics that are tailored to the task of interest. These “features” are a representation to be fed to classifiers that exploit spatial and temporal consistencies. Our testbed can detect the presence or absence of a bird with an accuracy of 82%, count eggs with an accuracy of 84%, and detect the inception of the nesting stage within a day. Our results demonstrate the challenges and potential benefits of using imagers as biological sensors. An exploration of system performance under varying image resolution and frame rate suggest that an in situ adaptive vision system is technically feasible.

This paper presents our experience using a domain-specific functional language, WaveScript, to build embedded sensing applications used in scientific research. We focus on a recent computervision application for detecting birds in their natural environment. The application was ported from a prototype in C++. In reimplementing the application, we gained a much cleaner factoring of its functionality (through higher-order functions and better interfaces to libraries) and a near-linear parallel speed-up with no additional effort. These benefits are offset by one substantial downside: the lack of familiarity with the language of the original vision researchers, who understandably tried to use the language in the familiar way they use C++ and thus ran into various problems.

Networked embedded acoustic sensors and imagers allow scientists to observe biological and environmental phenomena at high sampling rates and multiple scales. Such sampling can create large data sets that often require some form of automated processing to extract useful information. However, to guarantee the accuracy of the data, the scientist must be included in the processing, rather than treating it as a black box, an approach we call interactive environmental sensing. In this paper we describe the challenges of such an approach and motivate it with several examples from bioacoustics, plant phenology and avian biology.

Our paper addresses the problem of enforcing constraints in human body tracking. A projection technique is derived to impose kinematic constraints on independent multibody motion: we show that for small motions the multibody articulated motion space can be approximated by a linear manifold estimated directly from the previous body pose. We propose a learning approach to model nonlinear constraints; we train a support vector classifier from motion capture data to model the boundary of the space of valid poses. Linear and nonlinear body pose constraints are enforced by first projecting unconstrained motions onto the articulated motion space and then optimizing to find points on this linear manifold that lie within the non-linear constraint surface modeled by the SVM classifier.

Humans use a combination of gesture and speech to convey meaning, and usually do so without holding a device or pointer. We present a system that incorporates body tracking and gesture recognition for an untethered human-computer interface. This research focuses on a module that provides parameterized gesture recognition, using various machine learning techniques. We train the support vector classifier to model the boundary of the space of possible gestures, and train Hidden Markov Models on specific gestures. Given a sequence, we can find the start and end of various gestures using a support vector classifier, and find gesture likelihoods and parameters with a HMM. Finally multimodal recognition is performed using rank-order fusion to merge speech and vision hypotheses.