This paper presents an automatic procedure for constructing a compact synthesized collage from a video sequence. The synthesized image called video collage, is a kind of static video summary - to select most representative images from video, to extract salient regions of interest (ROI) from these images and resize them according to their saliency, and to seamlessly arrange ROI on a given canvas with the temporal structure of video content preserved. We formulate the generation of video collage as an energy minimization problem in which each of above desirability is represented by an energy term. Unlike most existing video presentation schemes, video collage is more compact and visually appealing. We have applied video collage on several home videos and report superior performance in a user study compared with key existing approaches to video presentation.

In this work, we present Video Collage system, which automatically constructs a compact and visually appealing synthesized collage from a video sequence for efficient video browsing. Given a video, Video Collage is able to select the most representative images, extract salient regions of interest (ROI) from these images and resize ROI according to their saliencies, and seamlessly arrange them on a given canvas while preserving the temporal structure of video content. Furthermore, Video Collage provides a novel user interface that enables users to browse video content in a variety of more efficient ways in contrast to many existing approaches to video browsing.

Compared with the video programs taken by professionals, home videos are always with low quality content resulted from non-professional capture skills. In this paper, we present a novel spatiotemporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, subshot, is selected as the basic unit for quality assessment. A set of spatiotemporal visual artifacts, regarded as the key factors affecting the overall perceived quality (i.e., unstableness and jerkiness as temporal factors; infidelity, blurring, brightness, and orientation as spatial factors), are mined from each subshot based on particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three different methods, including user study-based, rule-based and learning-based. To validate the proposed scheme, we present a scalable quality-based home video summarization system from a novel perspective-achieving the best visual quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model-based video skimming approach demonstrated the effectiveness of the proposed quality assessment scheme

With the rapid adoption of consumer digital video recorders and an increase of home video data, content analysis has become an interesting and key research issue to provide personalized experiences and services for both camcorder users and viewers. In this paper, we present a novel view to tackle this issue, which aims at modeling and mining of the capture intention of camcorder users. Based on the study of intention mechanism in psychology, a set of domain-specific capture intention concepts is defined. A comprehensive and extensible scheme consisting of video structure decomposition, intention-oriented feature analysis, as well as singular-value-decomposition-based intention segmentation and learning-based intention classification is proposed to mine the users' capture intention. Experiments were carried on home video sequences of 90 h in total, taken by 16 persons over the past 20 years. Both the user study and objective evaluations indicate that our proposed intention-based approach is an effective complement to existing home video content analysis schemes

This paper presents a novel probabilistic approach to fusing multimodal metadata for event based home photo clustering. Photo events are characterized by the coherence of multimodality including time, content and camera settings. We incorporate these multimodal metadata into a unified probabilistic framework, in which event is taken as a latent semantic concept and discovered by fitting a generative model through an expectation-maximization (EM) algorithm. This approach is general and unsupervised, without any training procedure or predefined threshold. The experimental evaluations on 14 k photos taken by 10 amateur photographers have indicated the effectiveness and efficiency of the proposed framework in browsing and searching personal photo collections

Presenting more comprehensive information than key frame and any subset of frames, mosaic has attracted a growing attention in recent years as a useful element for a variety of vision tasks. In this paper, we present an efficient approach to video mosaicing based on motion analysis with investigating two important issues that significantly affect the performance of mosaicing. First, given a video shot, a method based on motion entropy analysis is introduced to determine whether the visual content in a shot or video segment is suited to be represented by a mosaic. The second is to decide which subset of frames in the shot is the best to be selected for efficient and effective mosaicing by constructing a global motion path. Experimental comparisons with typical mosaicing method approve that with this approach, the visual quality of mosaic is significantly improved, as well as the computation time is remarkably reduced.

In this paper, we present a novel view to home video content analysis, which aims at tracking the capture intention of camcorder users. Based on the study of intention mechanism in psychology, a set of domain-specific capture intention concepts are defined. A comprehensive and extensible scheme consisting of video structuring, intention oriented feature analysis, as well as intention unit segmentation and classification is proposed to mine the users' capture intention. Experiments were carried on home video sequences of 90 hours in total, taken by 16 persons in recent 20 years. Both the user study and objective evaluations indicate that our proposed intention-based approach is an effective complement to existing home video content analysis schemes.

Compared with the video programs taken by professionals, home videos are always with low-quality content resulted from lack of professional capture skills. In this paper, we present a novel spatio-temporal quality assessment scheme in terms of low-level content features for home videos. In contrast to existing frame-level-based quality assessment approaches, a type of temporal segment of video, sub-shot, is selected as the basic unit for quality assessment. A set of spatio-temporal artifacts, regarded as the key factors affecting the overall perceived quality (i.e. unstableness, jerkiness, infidelity, blurring, brightness and orientation), are mined from each sub-shot based on the particular characteristics of home videos. The relationship between the overall quality metric and these factors are exploited by three different methods, including user study, factor fusion, and a learning-based scheme. To validate the proposed scheme, we present a scalable quality-based home video summarization system, aiming at achieving the best quality while simultaneously preserving the most informative content. A comparison user study between this system and the attention model based video skimming approach demonstrated the effectiveness of the proposed quality assessment scheme.

Video is an information-intensive media with much redundancy. Therefore, it is desirable to be able to mine structure or semantics of video data for efficient browsing, summarization and highlight extraction. In this paper, we propose a generic approach to key-event as well as structure mining for sports video analysis. Mosaic is generated for each shot as the representative image of shot content. Based on mosaic, sports video is mined by the method with prior knowledge and without prior knowledge. Without prior knowledge, our system may locate plays by discriminating those segments without essential content, such as breaks. If prior knowledge is available, the key-events in plays are detected using robust features extracted from mosaic. Experimental results have demonstrated the effectiveness and robustness of this sports video mining approach.

Images have become more and more common in mobile communications. People now can easily take and exchange pictures on the move using their mobile devices and digital cameras. However, a crucial challenge is to provide a better user experience for browsing large images on limited and heterogeneous screen sizes of mobile devices. In this paper, we propose a novel image viewing technique based on an adaptive attention shifting model. A presentation technique named rapid serial visual presentation (RSVP), borrowed from the UI community, is used to simulate the attention shifting process. We show a prototype image viewer developed for pocket PC and conduct some evaluations to demonstrate the effectiveness of our approach.

With the growing popularity of personal digital assistants and smart phones, people have become enthusiastic to watch videos through these mobile devices. However, a crucial challenge is to provide a better user experience for browsing videos on the limited and heterogeneous screen sizes. In this paper, we present a novel approach which allows users to overcome the display constraints by zooming into video frames while browsing. An automatic approach for detecting the focus regions is introduced to minimize the amount of user interaction. In order to improve the quality of output stream, virtual camera control is employed in the system. Preliminary evaluation shows that this approach is an effective way for video browsing on small displays.