Hidden Markov models (HMMs) have become the workhorses of the monitoring and event recognition literature because they bring to time-series analysis the utility of density estimation and the convenience of dynamic time warping. Once trained, the internals of these models are considered opaque; there is no effort to interpret the hidden states. We show that by minimizing the entropy of the joint distribution, an HMM's internal state machine can be made to organize observed activity into meaningful states. This has uses in video monitoring and annotation, low bit-rate coding of scene activity, and detection of anomalous behavior. We demonstrate with models of office activity and outdoor traffic, showing how the framework learns principal modes of activity and patterns of activity change. We then show how this framework can be adapted to infer hidden state from extremely ambiguous images, in particular, inferring 3D body orientation and pose from sequences of low-resolution silhouettes.

We show how to learn a concise, interpretable model of scene activity directly from optical flow. The model represents the principal routes and modes of movement in complex scenes such as pedestrian plazas and traffic intersections, and supports a variety of inferences about the observed activities, including annotation, prediction, and anomaly detection. The model takes the form of a novel hidden Markov model generalization that observes a variable number of datapoints per frame (time step). A monotonic entropy-optimizing algorithm determines the parameters and structure of this model, exploiting the duality between learning and compression to produce highly predictive and interpretable models. This approach discovers minimal models of coherent motions and their switching dynamics-without tracking or prior knowledge about the spatial or temporal structure of the scene.

The task of multicamera surveillance is to reconstruct the paths taken by all moving objects that are temporally visible from multiple non-overlapping cameras. We present a Bayesian formalization of this task, where the optimal solution is the set of object paths with the highest posterior probability given the observed data. We show how to efficiently approximate the maximum a posteriori solution by linear programming and present initial experimental results.