Quickly accessing the contents of a video is challenging for users, particularly for unstructured video, which contains no intentional shot boundaries, no chapters, and no apparent edited format. We approach this problem in the domain of lecture videos using machine learning and semantic display techniques. We extend an existing video browser, through a display of these machine-learned semantic labelings to provide the user with a multi-timeline semantic view. Each timeline corresponds to one semantic label and indicates the label's probable presence or absence in the associated frames. We also provide a full Boolean algebra over these labels, in order to accommodate more complex queries, such as 'text or code, but no instructor'. Finally, we quantify the effectiveness of our features and our browser through user studies on various tasks. We find that users follow a nearly fixed pattern of access, alternating between the use of these tags and keyframes, and also between the use of 'word bubbles' and the player. We show that the tag algebra is integral to the time efficient use of tag timelines, saving up to 27% of the time for various retrieval tasks.

We explore the problem of rapid automatic semantic tagging of video frames of unstructured (unedited) videos. We apply the sort-merge algorithm for feature selection on a large (>1000) heterogeneous feature set for videos showing lectures, to quickly locate low-level image features most predictive for concepts such as "key frame with text" or "key frame with computer source code". For evaluation, we introduce a "keeper" heuristic for feature retention, which provides a baseline comparison. We then compare early fusion and late fusion of diverse feature types; based on experiments on 12,395 frames, we find that in general late computation cost, compared to early fusion. However, mergers of redundant feature types do not necessarily improve performance over single feature types; exploration of both merged and unmerged performance is necessary.