We present a novel approach to the causal temporal analysis of event data from video content. Our key observation is that the sequence of visual words produced by a space-time dictionary representation of a video sequence can be interpreted as a multivariate point-process. By using a spectral version of the pairwise test for Granger causality, we can identify patterns of interactions between words and group them into independent causal sets. We demonstrate qualitatively that this produces semantically-meaningful groupings, and we demonstrate quantitatively that these groupings lead to improved performance in retrieving and classifying social games from unstructured videos.

A new problem of retrieving social games from unstructured videos is proposed. Social games are characterized by repetitions (with variations) of alternating turns between two players. We define games as quasi-periodic motion patterns in video based on their repetitiveness property. We have developed an algorithm to extract such patterns from video. The patterns extracted by our method, from video clips of social games taken from YouTube, are shown to correspond to meaningful stages of the games. We demonstrate promising results in retrieving social games from unstructured, lab-recorded footage of children's play, and identifying social interactions in a dataset of approximately 3.75 hours of home movies.

This paper presents the first systematic empirical study of the particle filter (PF) algorithms for human figure tracking in video. Our analysis and evaluation follows a modular approach which is based upon the underlying statistical principles and computational concerns that govern the performance of PF algorithms. Based on our analysis, we propose a novel PF algorithm for figure tracking with superior performance called the Optimized Unscented PF. We examine the role of edge and template features, introduce computationally-equivalent sample sets, and describe a method for the automatic acquisition of reference data using standard motion capture hardware. The software and test data are made publicly-available on our project website.