This paper presents a model for automatic summarization of videos recorded by wearable cameras. The proposed model detects various user activities by computing the transform of matching image features among video frames. Four basic types of user activities are proposed, including "moving closer /farther", "panning", "making a turn", and "rotation". Different summarization techniques are provided for different activity types, and a wearable video sequence can be summarized as a compact set of panoramic images. The user activity analysis is solely based on the analysis of images, without resorting to the information of other sensors. Experimental results on a 19- minute video sequence demonstrate the effectiveness of our proposed model.

In instructional videos of chalk board presentations, the visual content refers to the text and figures written on the boards. Existing methods on video summarization are not effective for this video domain because they are mainly based on low-level image features such as color and edges. In this work, we present a novel approach to summarizing the visual content in instructional videos using middle-level features. We first develop a robust algorithm to extract content text and figures from instructional videos by statistical modelling and clustering. This algorithm addresses the image noise, nonuniformity of the board regions, camera movements, occlusions, and other challenges in the instructional videos that are recorded in real classrooms. Using the extracted text and figures as the middle level features, we retrieve a set of key frames that contain most of the visual content. We further reduce content redundancy and build a mosaicked summary image by matching extracted content based on K-th Hausdorff distance and connected component decomposition. Performance evaluation on four full-length instructional videos shows that our algorithm is highly effective in summarizing instructional video content.

Video key frame extraction is one of the most important research problems for video summarization, indexing, and retrieval. For a variety of applications such as ubiquitous media access and video streaming, the temporal boundaries between video key frames are required for synchronizing visual content with audio. In this article, we define temporal video sampling as a unified process of extracting video key frames and computing their temporal boundaries, and formulate it as an optimization problem. We first provide an optimal approach that minimizes temporal video sampling error using a dynamic programming process. The optimal approach retrieves a key frame hierarchy and all temporal boundaries in O ( n 4 ) time and O ( n 2 ) space. To further reduce computational complexity, we also provide a suboptimal greedy algorithm that exploits the data structure of a binary heap and uses a novel “look-ahead” computational technique, enabling all levels of key frames to be extracted with an average-case computational time of O ( n log n ) and memory usage of O ( n ). Both the optimal and the greedy methods are free of parameters, thus avoiding the threshold-selection problem that exists in other approaches. We empirically compare the proposed optimal and greedy methods with several existing methods in terms of video sampling error, computational cost, and subjective quality. An evaluation of eight videos of different genres shows that the greedy approach achieves performance very close to that of the optimal approach while drastically reducing computational cost, making it suitable for processing long video sequences in large video databases.

Image segmentation and its performance evaluation are very difficult but important problems in computer vision. A major challenge in segmentation evaluation comes from the fundamental conflict between generality and objectivity: For general-purpose segmentation, the ground truth and segmentation accuracy may not be well defined, while embedding the evaluation in a specific application, the evaluation results may not be extended to other applications. We present in this paper a new benchmark for evaluating image segmentation. Specifically, we formulate image segmentation as identifying the single most perceptually salient structure from an image. We collect a large variety of test images that conforms to this specific formulation, construct unambiguous ground truth for each image, and define a reliable way to measure the segmentation accuracy. We then present two special strategies to further address two important issues: (a) the most salient structures in some real images may not be unique or unambiguously defined, and (b) many available image-segmentation methods are not developed to directly extract a single salient structure. Finally, we apply this benchmark to evaluate and compare the performance of several state-of-the-art image-segmentation methods, including the normalized-cut method, the level-set method, the efficient graph-based method, the mean-shift method, and the ratio-contour method.

This paper presents a robust approach to extracting and summarizing the textual content of instructional videos for handwritten recognition, indexing and retrieval, and other e-learning applications. Content extraction from instructional videos is challenging due to image noise, light condition changes, camera movements, and unavoidable occlusions by instructors. In this paper, we develop a probabilistic model to accurately detect board regions and an adaptive thresholding technique to extract the written chalk pixels on blackboards. We farther compute instructional video key frames by analyzing the fluctuation of the number of chalk pixels. By matching the textual content of video frames using a Hausdorff-distance-based technique, we reduce the content redundancy among the key frames. Performance evaluation on three full-length instructional videos shows that our algorithm is highly effective in summarizing instructional video content and achieves very low content missing rates.

This paper presents an efficient three-step algorithm to compute key frames for summarization and indexing of digital videos. In the preprocessing step, we remove the video frames that constitute the gradual transitions of video shots using an entropy-based energy-minimization method. The second step measures the content dissimilarity of video frames and removes the ones that are similar in content. In the postprocessing step, the video frames are processed using a dynamic clustering technique. The first and the second steps are implemented as queues, allowing efficient temporal filtering of video frames. Our algorithm greatly reduces the difficulty of parameter selection with content-adaptive parameters. Experimental results on four videos show that our method retrieves the least number of transitional and near-duplicate key frames, compared with three other existing methods.

With the popularity of e-learning systems, there is an increasing demand on effectively utilizing instructional videos to augment distance learning experiences. This paper addresses some special issues in processing and streaming lecture videos, and provides a new approach for real-time content analysis and adaptive transmission of these videos over wireless networks. A content-based analysis method and a buffer-based model are provided to detect content regions and to select content significant key frames, and an adaptive feedback control scheme is applied to transmit properly compressed key frames to accommodate dynamically changing bandwidths in wireless environments, with consideration of the limitations of resolution and computing capability of mobile devices. We demonstrate the effectiveness and scalability of our system in experimental results of adaptive transmission of instructional videos over 802.11b wireless networks to mobile devices.

Communication between a pair of nodes in the network may get disrupted due to failures of links/nodes resulting in zero effective bandwidth between them during the recovery period. It has been observed that such disruptions are not too uncommon and may last from tens of seconds to minutes. Even an occasional such disruption can drastically degrade the viewing experience of a participant in a video streaming session particularly when a sequence of frames central to the story are lost during the disruption. The conventional approach of prefetching video frames and patching lost ones with retransmissions is not always viable when disruptions are localized and experienced only by a few among many receivers. Error spreading approaches that distribute the losses across the video work well only when the disruptions are quite short. As a better alternative, we propose a disruption-tolerant content-aware video streaming approach that combines the techniques of content summarization and error spreading to enhance viewers experience even when the disruptions are long. We introduce the notion of "substitutable content summary frames" and provide a method to select these frames and also their transmission order to mitigate the impact of a disruption. In the event of a disruption, the already received summary frames are played by the client during disruption and near normal playback is resumed after the disruption. We evaluate our approach and demonstrate that it provides acceptable viewing experience with minimal startup latency and client buffer.

A new approach for content analysis and semantic compression of instructional videos is presented. Cameras often only capture a portion of a hand-drawn slide or blackboard panel that dominate this genre. We therefore have designed a novel semantic technique that retrieves the teaching content (text lines, figures, emphasis marks, etc.) visible in a frame, de-skews them and enhances their contrast, and stitches them back together into "virtual" slides that maintain the relative spatial relations of the captured fragments. Since this semantic mosaicing does not need precise pixel-wise matching, it has low computational complexity. These virtual slides maintain the useful teaching content of their real-world counterparts, and we further stitch them together to form a content summary slide to summarizes an extended video segment. By detecting content and recording its temporal development, we can then semantically compress a video into several such higher quality mosaics, and reconstruct the original instructional video by displaying the mosaics instead at the appropriate times. Our design runs in one pass and in real time. We apply the system to a 17-minute instructional video, and show the results of its semantic compression into virtual slide mosaics.

We present a new content-based approach to summarize instructional videos. We first redefine "scene" in instructional videos. Focusing on one dominant scene type, that of handwritten lecture notes, we define semantic content as "ink pixels", and present a low-level retrieval technique to extract this content from each frame with consideration of various occlusion and illumination effects. "Key frames" in this video genre are redefined as a set of frames that cover the semantic content, and the fluctuating amount of visible ink is used to drive a heuristic real-time key frame extraction method. A rule-based method is also provided to synchronize key frames with audio. We extend our method to the extraction of key frame hierarchies. We show its application to a 17-minute (30K frames) instructional video sequence, resulting in seven key frames. These techniques create tunable instructional summaries over a wide and dynamically varying range of compression factors.

This paper presents a new approach to indexing videos of presentations which use electronic slides. By determining the images of slides in the video frames, and then matching the video sequences to the original electronic slides, the video can be indexed and searched, and the visual appearance of the segments can be improved. We first detect the "content area" in video frames using a color similarity weighted least square method. By monitoring the frame-to-frame "content difference", we then temporally segment the video into sequences which display the same slide. Since this differencing removes low-level imaging effects, we can then relate the video segments to slides by matching the content differences of adjacent video segments to the content differences of all possible slide pairs. By defining the transition probabilities of video segments, we are able to solve this matching efficiently in two steps, first by finding high likelihood matches, and then by using dynamic programming for the unmatched remainder. Once matched, the correspondence to the original slides can be used to enhance the presentation video quality in the content area, and can also be used for indexing and summarization. Experiments show high performance of segmentation and matching on several presentation videos which vary considerably in color and background style.

We provide a novel algorithm for selecting key frames from a video at all possible temporal sampling densities. We define a measure for video reconstruction error (VRE), and use VRE to evaluate the representational quality of the selected key frames. The algorithm uses a heap-based greedy algorithm to build a hierarchy of increasingly sparsely sampled temporal sequences, each consisting of key frames with minimal VRE. By exploiting the heap and a novel computation technique of "forward computing", the complete hierarchy is constructed in O(nlog(n)) time and O(n) space. The algorithm also ranks all frames according to their importance in recovering the original video, potentially useful for applications in temporally scalable video coding and video streaming. Experiments show that our algorithm outperforms other existing methods in VRE, as compared via peak signal-noise ratio, in computation time and in guaranteed convergence.

We define and present a new model for semantic video compression and searching, and apply this model to several video genres, with special emphasis on instructional videos. In this model, a semantic distance based on the video genre is computed between adjacent frames in a dynamic video buffer of predetermined size, and redundant "unkey" frames leak from the buffer interior into a data structure "leakage history directed acyclic graph" (LHDAG) which records the relative importance of these frames. What exits the buffer forms a highly compressed video stream consisting of only the most semantically significant video frames, whereas the LHDAG permits efficient semantic exploration of the video interior. This novel hierarchical but context-sensitive data structure LHDAG permits the searching of the video at display rates that are proportional to visual significance, at levels of semantic density selectable by the user. The data structure is simple to create and query, and appears to be more psychologically plausible than more straightforward fixed sampling indexing schemes. We empirically display and mathematically analyze the relationships of the video buffer's leaking rate, buffer size, and exit delay, and demonstrate its performance on several extended videos. The flexibility of the method is indicated by its demonstration on two very different definitions of semantic significance: color similarity, and content ("ink pixel") similarity.