Stereoscopic video quality assessment has become a major research topic in recent years. Existing stereoscopic video quality metrics are predominantly based on stereoscopic image quality metrics extended to the time domain via for example temporal pooling. These approaches do not explicitly consider the motion sensitivity of the Human Visual System (HVS). To address this limitation, this paper introduces a novel HVS model inspired by physiological findings characterising the motion sensitive response of complex cells in the primary visual cortex (V1 area). The proposed HVS model generalises previous HVS models, which characterised the behaviour of simple and complex cells but ignored motion sensitivity, by estimating optical flow to measure scene velocity at different scales and orientations. The local motion characteristics (direction and amplitude) are used to modulate the output of complex cells. The model is applied to develop a new type of full-reference stereoscopic video quality metrics which uniquely combine non-motion sensitive and motion sensitive energy terms to mimic the response of the HVS. A tailored two-stage multi-variate stepwise regression algorithm is introduced to determine the optimal contribution of each energy term. The two proposed stereoscopic video quality metrics are evaluated on three stereoscopic video datasets. Results indicate that they achieve average correlations with subjective scores of 0.9257 (PLCC), 0.9338 and 0.9120 (SRCC), 0.8622 and 0.8306 (KRCC), and outperform previous stereoscopic video quality metrics including other recent HVS-based metrics.

Radio production involves editing speech-based audio using tools that represent sound using simple waveforms. Semantic speech editing systems allow users to edit audio using an automatically generated transcript, which has the potential to improve the production workflow. To investigate this, we developed a semantic audio editor based on a pilot study. Through a contextual qualitative study of five professional radio producers at the BBC, we examined the existing radio production process and evaluated our semantic editor by using it to create programmes that were later broadcast. We observed that the participants in our study wrote detailed notes about their recordings and used annotation to mark which parts they wanted to use. They collaborated closely with the presenter of their programme to structure the contents and write narrative elements. Participants reported that they often work away from the office to avoid distractions, and print transcripts so they can work away from screens. They also emphasised that listening is an important part of production, to ensure high sound quality. We found that semantic speech editing with automated speech recognition can be used to improve the radio production workflow, but that annotation, collaboration, portability and listening were not well supported by current semantic speech editing systems. In this paper, we make recommendations on how future semantic speech editing systems can better support the requirements of radio production.

Quality of Experience has recently become a paramount research topic in multimedia systems, especially in the emerging areas of high-definition and 3D video content. There has been a significant amount of research focusing on 3D content production, compression and delivery. However very little research has been dedicated to the emerging challenges in assessing user experience when interacting with the 3D video content. Interaction tasks such as pointing and selection are critical to the consumer's experience of the 3D video technology. This paper studies the impact of pointing modalities on the quality of interaction experience with stereoscopic 3D television. The conducted user study compares and evaluates three pointing modalities: standard mouse-based interaction, virtual laser pointer and hand gesture modality, using the ISO 9241-9 standard for multi-directional tapping task. The results suggest that the virtual laser pointer modality can provide better quality of interaction experience than other modalities in terms of user performance and user satisfaction.

The recent proliferation of stereoscopic three dimensional (3D) video technology has fostered a large body of research into 3D video capture, production, compression and delivery. However, little research has been dedicated to the design practices of stereoscopic 3D video interaction. Interaction tasks such as pointing and selection are critical to the consumer's experience of the 3D video technology. This paper presents investigation of pointing modalities in the context of stereoscopic 3D television (TV). Adopting the ISO 9241-9 standard for multi-directional tapping task, the conducted user study compares and evaluates three pointing modalities: standard mouse-based interaction, virtual laser pointer implemented using Wiimote, and hand movement modality using Kinect. The results suggest that the virtual laser pointer modality is more advantageous than other modalities in terms of user performance and user comfort. In addition, this paper discusses the impact of disparity levels on the pointing tasks.

Compressed-domain content analysis enables fast content-based video applications. However, conventional encoder implementation, limited to optimising data compression, does not necessarily result in content representative compressed features. In this paper, we present a novel method for video encoder optimisation that enhances reliability of content representation in the compressed video. We propose a configurable encoder model to jointly optimise bit-rate, distortion and content representation of the encoded video. In encoder optimisation for motion estimation, we analyse the behaviour of motion representation and compression efficiency over a range of content complexities. A mathematical model to control the extent of noise in selected motion vectors is presented. We demonstrate that the compressed domain motion information can be enhanced without incurring a rate-distortion overhead. The computational overheads are marginal and can be eliminated by exploiting the inherently parallel nature of the proposed optimisation model.

This paper describes an efficient system for scalable video summarization that exploits comic-like summaries and multi-scale representations to facilitate interactivity and balance between content coverage and compactness. Due to the layout disturbance induced by the transitions between scales, a new heuristic algorithm is proposed to restrict changes to bounded summary segments. Conducted user evaluations show that the proposed methodology improves usability while keeping the summaries compact and informative.

In order to jointly optimise the quality of video coding on one hand and video analysis on the other, this paper proposes a novel approach to enhance the reusable information content in compressed video domain. By introducing a hierarchical content driven motion estimation mechanism at the encoder, complemented by a statistical prediction of region-of-interest, this approach reduces the complexity and yet increases robustness of the compressed domain vision analysis applications. Taking the object tracking application as an example, we demonstrate that the motion vectors generated by the proposed method can be directly used to extract object information, achieving tracking performance comparable with a pixel domain approach. In addition, we show that the incurred rate distortion (RD) overheads and the effect on encoder complexity are minimal, especially when compared to the reduction of processing required for video analysis targeting a wide spectrum of computer vision applications.

Intuitive interfaces have become increasingly important multimedia applications, from personal photo collection to professional management systems. This paper presents a novel intuitive interactive interface for browsing of large image collections that visualizes underlying structure of the dataset by its size and spatial relations. In order to achieve this, images are initially clustered using an unsupervised graph-based clustering algorithm. By selecting images in a hierarchical layout of the screen, user can intuitively navigate through the collection. The experimental results demonstrate a significant speed-up in a content search scenario compared to a standard browsing interface, as well as inherent intuitiveness of the system.

This paper presents user evaluation of the FreeEye tool for intuitive browsing and summarization of large-scale photo collections. The tool was tested in three different personal photo selection scenarios: a short-time event, a vacation and a yearbook. The experiments were conducted with five participants, evaluating their satisfaction with the summarization result and the overall process. The results demonstrate good usability of the FreeEye tool and improvement when compared to the standard methods of the participants for selection from large personal photo collections.

The work presented in this paper introduces a method for efficient adaptability of video summaries to spatial requirements defined by display size, user’s needs and channel limitations. By utilising compressed domain features and an efficient contour evolution algorithm, a scale space of temporal video descriptors is generated, enabling dynamic video summarisation in real-time. The summary is laid out utilising an unsupervised robust spectral clustering technique and a fast discrete optimisation algorithm. Results show excellent scalability of the video summarisation interface and highly improved efficiency of summary generation.

Due to a high demand for efficient video summarisation and video adaptation technologies, this paper focuses on utilisation of compressed domain feature extraction and hierarchical analysis of motion information in scalable video in order to generate intuitive visual summaries. By combining the analysis of inherently hierarchical motion activity measure and a fast geometrical curve simplification algorithm, a set of the most representative key-frames is generated in a very fast and robust manner. The experimental results show good subjective representation, while the method efficiency enables fast generation of summaries of large-scale video repositories.

In order to represent large amounts of information in the form of a video key-frame summary, this paper studies narrative grammar of comics, and using its universal and intuitive rules, lays out visual summaries in an efficient and user centered way. The system ranks importance of key-frame sizes in the final layout by balancing the dominant visual representability and discovery of unanticipated content utilizing a specific cost function and an unsupervised robust spectral clustering technique. A final layout is created using an optimization algorithm based on dynamic programming. Algorithm efficiency and robustness are demonstrated by comparing the results with the optimal panelling solutions.

A generic system for automatic annotation of videos is introduced. The proposed approach is based on the premise that the rules needed to infer a set of high-level concepts from low-level descriptors cannot be defined a priori. Rather, knowledge embedded in the database and interaction with an expert user is exploited to enable system learning. Underpinning the system at the implementation level is preannotated data that dynamically creates signification links between a set of low-level features extracted directly from the video dataset and high-level semantic concepts defined in the lexicon. The lexicon may consist of words, icons, or any set of symbols that convey the meaning to the user. Thus, the lexicon is contingent on the user, application, time, and the entire context of the annotation process. The main system modules use fuzzy logic and rule mining techniques to approximate human-like reasoning. A rule-knowledge base is created on a small sample selected by the expert user during the learning phase. Using this rule-knowledge base, the system automatically assigns keywords from the lexicon to nonannotated video clips in the database. Using common low-level video representations, the system performance was assessed on a database containing hundreds of broadcasting videos. The experimental evaluation showed robust and high annotation accuracy. The system architecture offers straightforward expansion to relevance feedback and autonomous learning capabilities.

The extensive amount of media coverage today, generates difficulties in identifying and selecting desired information. Browsing and retrieval systems become more and more necessary in order to support users with powerful and easy-to-use tools for searching, browsing and summarization of information content. The starting point for these tasks in video browsing and retrieval systems is the low level analysis of video content, especially the segmentation of video content into shots. This paper presents a fast and efficient way to detect shot changes using only the temporal distribution of macroblock types in MPEG compressed video. The notion of a dominant reference frame is introduced here. A dominant frame denotes the reference frame (I or P) used as prediction reference for most of the macroblocks from a subsequent B frame.

This paper presents a novel approach to multiresolution analysis and scalability in video indexing and retrieval. A scalable algorithm for video parsing and key-frame extraction is introduced. The technique is based on real-time analysis of MPEG motion variables and scalable metrics simplification by discrete contour evolution. Furthermore, a hierarchical key-frame retrieval method using scalable colour histogram analysis is presented. It offers customisable levels of detail in the descriptor space, where the relevance order is determined by degradation of the image, and not by degradation of the image histogram. To assess the performance of the approach, several experiments have been conducted. Selected results are reported.