Determining whether multimedia content related to a specific retrieval topic or a developing news story contains novel information is a crucial task in media monitoring and production. While visual content gains importance, most of the existing work focuses on text documents. We propose a method for determining novel visual information in videos, embedded in a workflow using multimodal topic threading and video to text, and compare two approaches to measure novelty. We present a web application for determining the novelty in multimedia content being ingested into a collection, and evaluate the approach on a subset of the FIVR-200K dataset.

The ACM Lifelog Search Challenge (LSC) is an annual comparative benchmarking exercise that brings together researchers in the field of multimedia retrieval to evaluate interactive search systems using a large-scale multimodal lifelog dataset. This paper presents an overview of the ninth edition of the challenge (LSC’26), held as a workshop during the ACM International Conference on Multimedia Retrieval (ICMR ’26) in Amsterdam. To broaden its scope, the workshop now features three submission tracks: the traditional Challenge Track for real-time search performance, a new General Lifelog Research Track for theoretical and architectural advancements, and an additional Open Source Track aimed at enhancing reproducibility and reducing barriers to entry for new participants.

This paper presents a comprehensive image labelling workflow that integrates the labelme software with Grounded-SAM2. Our approach aims to enhance the efficiency and precision of semantic- and instance segmentation annotation processes. First, labelme is used to create new annotations, forming a foundational dataset for fine-tuning a SAM2.1 model. Then, after training has been completed, a more specialised model is ready to be used for annotation. This iterative process enables us to refine the model's performance and generate high-quality annotations for a diverse range of images. Our approach is specifically designed to operate locally on decent GPUs, ensuring users can efficiently execute the whole workflow inside a single application without relying on cloud-based resources. This enhances accessibility and reduces latency in the annotation process. Furthermore, our workflow keeps annotation data under the user's control, ensuring data privacy and enabling users to securely manage their sensitive information throughout the labelling process.

For the eighth time since 2018, the ACM Lifelog Search Challenge (LSC) was run to compare interactive lifelog search systems in a live metrics-based challenge. The goal of the LSC workshop is to comparatively evaluate the capabilities of systems accessing a large multimodal lifelog. LSC'25 attracted eleven participating teams, each of which had developed an innovative interactive lifelog retrieval system. The benchmark was organised in a hybrid manner (due to political issues in 2025) at the LSC workshop at ACM ICMR'25 in Chicago, USA. This short paper summarises the LSC workshop setting and presents the participating lifelog search systems.

Egocentric video has seen increased interest in recent years, as it is used in a range of areas. However, most existing datasets are limited to a single perspective. In this paper, we present the CASTLE 2024 dataset, a multimodal collection containing ego- and exo-centric (i.e., first- and third-person perspective) video and audio from 15 time-aligned sources, as well as other sensor streams and auxiliary data. The dataset was recorded by volunteer participants over four days in a common location and includes the point of view of 10 participants, with an additional 5 fixed cameras providing an exocentric perspective. The entire dataset contains over 600 hours of UHD video recorded at 50 frames per second. In contrast to other datasets, CASTLE 2024 does not contain any partial censoring, such as blurred faces or distorted audio. The dataset is available via https://castle-dataset.github.io/.

The inaugural edition of the CASTLE grand challenge was held at ACM Multimedia 2025. The focus of the CASTLE challenge is to advance the state-of-the-art in analysis and understanding of multimodal data, especially centered around multistream ego- and exo-centric video. In this first instance of the challenge, participants had to solve three types of tasks - event instance search, object instance search, and question answering - in a fully automatic or interactive setting.

Diminished reality (DR) refers to the removal of real objects from the environment by virtually replacing them with their background. Modern DR frameworks use inpainting to hallucinate unobserved regions. While recent deep learning-based inpainting is promising, the DR use case is complicated by the need to generate coherent structure and 3D geometry (i.e., depth), in particular for advanced applications, such as 3D scene editing. In this paper, we propose Deep DR, a first RGB-D inpainting framework fulfilling all requirements of DR: Plausible image and geometry inpainting with coherent structure, running at real-time frame rates, with minimal temporal artifacts. Our structure-aware generative network allows us to explicitly condition color and depth outputs on the scene semantics, overcoming the difficulty of reconstructing sharp and consistent boundaries in regions with complex backgrounds. Experimental results show that the proposed framework can outperform related work qualitatively and quantitatively.

The creation of digital twins of cityscapes requires the understanding the semantics of relevant objects encountered in the scene, with classes possibly not well covered in standard datasets. In this paper, we assess the generalization capabilities of models for semantic segmentation of street scenes trained on synthetic data to real images. In order to perform assessment on real data, we create a dataset from images obtained from a vehicle-based capture setup, and propose a highly automated annotation pipeline for generating ground truth. We train three state of the art segmentation models on our own synthetic dataset, and evaluate the methods on real data.

Artificial intelligence-based face recognition solutions are becoming increasingly popular. Therefore, it is crucial to fully understand and explain how these technologies work in order to make them more effective and acceptable to society. This is the goal of the CHIST-ERA project XAIface, the final results of which are reported in this article: a framework and toolkit for improving AI decision explainability, in the context of automated face recognition, through several novel methods are presented. These methods are integrated into an end-to-end face recognition demonstrator system, which facilitates studying the impact of various influencing factors and system processes on recognition performance. By doing so, we can visually explain the decisions made by the face verification pipeline for specific instances in our test set using heatmaps and locally interpretable features. Furthermore, we offer a comprehensive explanation of the end-to-end model by examining the relationship between verification failures and misclassifications of soft biometric facial traits.

A web-based application for incremental training of landmark recognition, suitable for domain experts without machine learning expertise is presented. Its backend uses the fine-grained image classification network API-Net in a few-shot setting, making use of the two-stage fine-tuning paradigm. The aim is to enable rapid training of a classifier for recognizing new landmarks in video content, supporting needs in media production. Using base models trained on two different datasets, we demonstrate the speed and effectiveness of the application in training the networks to detect new landmarks. In addition, our application provides the ability to retrain a previously learned landmark with further data, e.g. to improve performance for views or imaging conditions not well supported by the model. This feature ensures that the model remains up-to-date with evolving datasets and environmental conditions, thereby improving its accuracy and adaptability over time.

Augmented Reality (AR) applications rely heavily on the accurate detection and segmentation of objects in order to seamlessly integrate virtual content into real world environments. However, achieving robust object detection and segmentation in AR scenes remains challenging, in particular when specific classes need to be covered. Due to the lack of sufficient real annotated data, training can rely on synthetic data, which has the issue of the domain gap between synthetic and real-world images. This paper addresses the challenge of generalisation in object detection/instance segmentation models trained with synthetic data. It presents techniques to improve the generalisation capability of object detection models trained on synthetic data for real-world applications. Our method is based on the model soup approach, where models trained with different data subsets and hyperparameters are combined to achieve better generalisation performance. The experimental results on the validation dataset of the ADAPT sim2real Object Detection Challenge 2023 demonstrate the effectiveness of our approach in achieving good generalisation performance. This marks a significant step towards more reliable and adaptable object detection and segmentation approaches for AR applications.

For the seventh time since 2018, the Lifelog Search Challenge (LSC) benchmarked interactive lifelog search systems in a live challenge. The LSC goal is to comparatively evaluate system capabilities to access large multimodal lifelogs comprising hundreds of thousands of records. LSC'24 attracted an unprecedented record number of twenty-one participating teams, where each team proposes innovative ideas implemented to new or already established interactive lifelog retrieval systems. The benchmark was organised in front of a live audience at the LSC workshop at ACM ICMR'24 in Phuket, Thailand. This short paper summarises the LSC workshop setting and presents the participating lifelog search systems.

This tutorial provides an overview of multimedia retrieval in the context of eXtended Reality (XR), including using virtual and augmented/mixed reality as a user interface for multimedia retrieval, as well as multimedia search tasks addressing content needs for the creation of XR experiences.It will discuss the opportunities and limitations of XR-based search, the evaluation of XR-based multimedia retrieval systems, the demonstration of selected research systems, and open research challenges.

The way we create, consume and interact with multimedia content has changed significantly in recent years with the advent of affordable recording devices and easy sharing and access in the form of mobile phones. With the imminent wave of affordable devices that enable mixed reality experiences and the large variety of devices on the market, interaction with multimedia content is expected to continue to evolve rapidly. This will also drastically affect the entire area of multimedia information retrieval in eXtended Reality (XR), for instance by novel ways to express user needs in VR, result presentation that takes the specific capabilities of XR devices into account, and/or result feedback. This tutorial on Multimedia Retrieval in XR discusses and demonstrates existing solutions and highlights key challenges in this evolving field.

In this paper we propose ÖWF-OD, a new dataset for object detection in archival film content. The dataset enables the evaluation of object detection methods on image data with very different image qualities. 1,000 selected keyframes from 100 hours of video material have been annotated, 4,480 bounding boxes are labeled. In addition to these annotations, image quality measures were calculated for the keyframes in order to assess the influence of image quality on object detection. We evaluate different versions of the YOLO object detector to provide a baseline of object detection results for this dataset. The annotation data and the code to extract the keyframes are provided at https://github.com/TailoredMediaProject/OEWF_ObjectDetection.

Different task interpretations are a highly undesired element in interactive video retrieval evaluations. When a participating team focuses partially on a wrong goal, the evaluation results might become partially misleading. In this paper, we propose a process for refining known-item and open-set type queries, and preparing the assessors that judge the correctness of submissions to open-set queries. Our findings from recent years reveal that a proper methodology can lead to objective query quality improvements and subjective participant satisfaction with query clarity.

For the sixth time since 2018, the Lifelog Search Challenge (LSC) was organized as a comparative benchmarking exercise for various interactive lifelog search systems. The goal of this international competition is to test system capabilities to access large multimodal lifelogs. LSC’23 attracted twelve participanting teams, each of whom had developed a competitive interactive lifelog retrieval system. The benchmark was organized in front of live audience at the LSC workshop at ACM ICMR’23. As in previous editions, this introductory paper presents the LSC workshop and introduces the participating lifelog search systems.

This paper addresses the issue of dealing with few-shot learning settings in which different classes are annotated on different datasets. Each part of the data has exhaustive annotations for only one or a small set of classes, but not for others used in training. It is likely, that unannotated samples of a class exist, potentially impacting the gradient as negative samples. Because of this fact, we argue that few-shot learning is essentially a semi-supervised learning problem. We analyze how approaches from semi-supervised learning can be applied. In particular, the use of soft-sampling to weight the gradient based on overlap of detections and ground truth, and creating missing annotations using a preliminary detector are studied. The use of soft-sampling provides small but consistent improvements, at much lower computational effort than predicting additional annotations.

During the last 10 years of Video Browser Showdown (VBS), there were many different approaches tested for known-item search and ad-hoc search tasks. Undoubtedly, teams incorporating state-of-the-art models from the machine learning domain had an advantage over teams focusing just on interactive interfaces. On the other hand, VBS results indicate that effective means of interaction with a search system is still necessary to accomplish challenging search tasks. In this tutorial, we summarize successful deep models tested at the Video Browser Showdown as well as interfaces designed on top of corresponding distance/similarity spaces. Our broad experience with competition organization and evaluation will be presented as well, focusing on promising findings and also challenging problems from the most recent iterations of the Video Browser Showdown.

The retrieval of multimedia content remains a difficult problem where a high accuracy or specificity can often only be achieved interactively, with a user working closely and iteratively with a retrieval system. While there exist several venues for the exchange of insights in the area of information retrieval in general and multimedia retrieval specifically, there is little discussion on such interactive retrieval approaches. The Workshop on Interactive Multimedia Retrieval offers such a venue. Held for the 2nd time in 2022, it attracted a diverse set of contributions, six of which were accepted for presentation. The following provides a brief overview of the workshop itself as well as the contributions of 2022.

Neural Network Coding and Representation (NNR) is the first international standard for efficient compression of neural networks (NNs). The standard is designed as a toolbox of compression methods, which can be used to create coding pipelines. It can be either used as an independent coding framework (with its own bitstream format) or together with external neural network formats and frameworks. For providing the highest degree of flexibility, the network compression methods operate per parameter tensor in order to always ensure proper decoding, even if no structure information is provided. The NNR standard contains compression-efficient quantization and deep context-adaptive binary arithmetic coding (DeepCABAC) as core encoding and decoding technologies, as well as neural network parameter pre-processing methods like sparsification, pruning, low-rank decomposition, unification, local scaling and batch norm folding. NNR achieves a compression efficiency of more than 97% for transparent coding cases, i.e. without degrading classification quality, such as top-1 or top-5 accuracies. This paper provides an overview of the technical features and characteristics of NNR.

Amateur film and video collections are often only sparsely documented and the contents of the material are only vaguely known. The progress in deep learning based computer vision methods over the last decade provides an opportunity to automatically generate additional descriptions that would help exploring and finding content, even if they do not meet the level of professional archive documentation. While the automatically generated metadata can facilitate interactive search and exploration, it will always need to be used with a human user in the loop, as it will not be perfect. In this paper, we describe a framework including visual content analysis tools and a web-based visualisation of the extracted metadata.

Communication with a customer or future user during the planning and design phase is crucial in applications such as interior design and furniture retailing. Augmented Reality (AR) has the potential to make these communication processes highly effective and provide a better experience for the customer. Current AR authoring solutions are quite complex and require manually creating scenes or rely on objects prepared with even more complex applications such as CAD tools. However, both design experts and their customers often lack the IT skills to use these tools. In addition, many practical cases involve changing reality rather than just adding to it, thus requiring the use of Diminished Reality (DR) technologies. This paper presents a comprehensive analysis of the requirements of both professionals and consumers (gathered using user surveys and individual interviews) for a lightweight and automated authoring process of AR and DR experiences, deriving a set of requirements that can be aligned with state of the art technologies and identifying a number of challenges for AR research.

Despite the fact that automatic content analysis has made remarkable progress over the last decade - mainly due to significant advances in machine learning - interactive video retrieval is still a very challenging problem, with an increasing relevance in practical applications. The Video Browser Showdown (VBS) is an annual evaluation competition that pushes the limits of interactive video retrieval with state-of-the-art tools, tasks, data, and evaluation metrics. In this paper, we analyse the results and outcome of the 8th iteration of the VBS in detail. We first give an overview of the novel and considerably larger V3C1 dataset and the tasks that were performed during VBS 2019. We then go on to describe the search systems of the six international teams in terms of features and performance. And finally, we perform an in-depth analysis of the per-team success ratio and relate this to the search strategies that were applied, the most popular features, and problems that were experienced. A large part of this analysis was conducted based on logs that were collected during the competition itself. This analysis gives further insights into the typical search behavior and differences between expert and novice users. Our evaluation shows that textual search and content browsing are the most important aspects in terms of logged user interactions. Furthermore, we observe a trend towards deep learning based features, especially in the form of labels generated by artificial neural networks. But nevertheless, for some tasks, very specific content-based search features are still being used. We expect these findings to contribute to future improvements of interactive video search systems.

Comprehensive and fair performance evaluation of information retrieval systems represents an essential task for the current information age. Whereas Cranfield-based evaluations with benchmark datasets support development of retrieval models, significant evaluation efforts are required also for user-oriented systems that try to boost performance with an interactive search approach. This article presents findings from the 9th Video Browser Showdown, a competition that focuses on a legitimate comparison of interactive search systems designed for challenging known-item search tasks over a large video collection. During previous installments of the competition, the interactive nature of participating systems was a key feature to satisfy known-item search needs, and this article continues to support this hypothesis. Despite the fact that top-performing systems integrate the most recent deep learning models into their retrieval process, interactive searching remains a necessary component of successful strategies for known-item search tasks. Alongside the description of competition settings, evaluated tasks, participating teams, and overall results, this article presents a detailed analysis of query logs collected by the top three performing systems, SOMHunter, VIRET, and vitrivr. The analysis provides a quantitative insight to the observed performance of the systems and constitutes a new baseline methodology for future events. The results reveal that the top two systems mostly relied on temporal queries before a correct frame was identified. An interaction log analysis complements the result log findings and points to the importance of result set and video browsing approaches. Finally, various outlooks are discussed in order to improve the Video Browser Showdown challenge in the future.

In this paper we address unsupervised machine learning for two use cases in satellite communication, which are scheduling problems: (i) Ka-band frequency plan optimization and (ii) dynamic configuration of an active antenna array satellite. We apply approaches based on the Expectation-Maximization (EM) framework to both of them. We compare against baselines of currently deployed solutions, and show that they can be significantly outperformed by the EM-based approach. In addition, the approaches can be applied incrementally, thus supporting fast adaptation to small changes in the input configuration.

The Video Browser Showdown (VBS) has influenced the Multimedia community already for 10 years now. More than 30 unique teams from over 21 countries participated in the VBS since 2012 already. In 2021, we are celebrating the 10th anniversary of VBS, where 17 international teams compete against each other in an unprecedented contest of fast and accurate multimedia retrieval. In this tutorial we discuss the motivation and details of the VBS contest, including its history, rules, evaluation metrics, and achievements for multimedia retrieval. We talk about the properties of specific VBS retrieval systems and their unique characteristics, as well as existing open-source tools that can be used as a starting point for participating for the first time. Participants of this tutorial get a detailed understanding of the VBS and its search systems, and see the latest developments of interactive video retrieval.

Replacing faces in image and video content with generated ones (e.g., using generative adversarial networks, GANs) has gained attention recently, as it enables resolving privacy issues in image and video data used for visualization purposes or training data in multimedia analysis and retrieval systems. Privacy issues should be addressed when visual content enters the system, as identifying and removing content later (which may be necessary due to the shifts in legislation and users' increased awareness) is a tedious and costly task. This paper proposes two improvements of face generation: First, we propose the use of portrait segmentation on the training data of the GAN, in order to generate images that are not only cropped to the face region, which may cause artifacts during insertion. Second, we add a face detection term to the loss function, in order to better guide the training process. The results show that these modifications enable creating uncropped face images achieving the same or better performance than for closely cropped images. We use the detectability of the generated faces as an evaluation metric, discuss the limitations of such a metric and propose enhancements for better comparability. We also demonstrate that the aim of anonymization is achieved by running face recognition on the modified images from the LFW data set.

The detection of objects in image and video has made huge progress in recent years due to the use of deep convolutional neural networks (DNNs), with some network architectures becoming de-facto standards. This paper addresses the problem of sharing a backbone CNN for different tasks, for example, to enable detection of additional classes when an already trained network is available. When using multiple such neural networks, sharing a backbone can save inference time and memory consumption. We study sharing a common backbone between neural networks trained for different tasks (logoness and text block detection) based on Yolo v3. We provide results on the impact of different lengths of the shared backbone on performance and resource efficiency.

We present a tutorial focusing on video retrieval tasks, where state-of-the-art deep learning approaches still benefit from interactive decisions of users. The tutorial covers general introduction to the interactive video retrieval research area, state-of-the-art video retrieval systems, evaluation campaigns and recently observed results. Moreover, a significant part of the tutorial is dedicated to a practical exercise with three selected state-of-the-art systems in the form of an interactive video retrieval competition. Participants of this tutorial will gain a practical experience and also a general insight of the interactive video retrieval topic, which is a good start to focus their research on unsolved challenges in this area.

This work summarizes the findings of the 7th iteration of the Video Browser Showdown (VBS) competition organized as a workshop at the 24th International Conference on Multimedia Modeling in Bangkok. The competition focuses on video retrieval scenarios in which the searched scenes were either previously observed or described by another person (i.e., an example shot is not available). During the event, nine teams competed with their video retrieval tools in providing access to a shared video collection with 600 hours of video content. Evaluation objectives, rules, scoring, tasks, and all participating tools are described in the article. In addition, we provide some insights into how the different teams interacted with their video browsers, which was made possible by a novel interaction logging mechanism introduced for this iteration of the VBS. The results collected at the VBS evaluation server confirm that searching for one particular scene in the collection when given a limited time is still a challenging task for many of the approaches that were showcased during the event. Given only a short textual description, finding the correct scene is even harder. In ad hoc search with multiple relevant scenes, the tools were mostly able to find at least one scene, whereas recall was the issue for many teams. The logs also reveal that even though recent exciting advances in machine learning narrow the classical semantic gap problem, user-centric interfaces are still required to mediate access to specific content. Finally, open challenges and lessons learned are presented for future VBS events.

In order to enable efficient instance search in video, compact descriptors for video segments have been proposed. They exploit the temporal redundancy within a video segment to obtain smaller descriptors, and the segment-based representation can be exploited to enable more efficient matching. In this paper we analyze the performance of different visual features when applying both lossless and lossy compression to the set of descriptors of one video segment. We consider both handcrafted and deep features, i.e., visual features obtained from training a deep convolutional neural network. We also propose optimizations to the extraction and matching procedure. Both the compression methods and the optimizations are experimentally evaluated on a large video data set.

Automatic quality assessment is an essential step in the professional audiovisual media production process. In this paper we propose a novel sharpness metric taking the specific properties of video into account, and having a higher robustness against variations in image content, interlacing artifacts and noise. Furthermore, a comprehensive user study is presented, where we obtain subjective scores to validate the sharpness metric. We ask 28 viewers for both absolute and relative judgments of sharpness in two separate experiment settings, supported with an eye tracker to obtain the locations used for judgements. Experimental results show that the objective sharpness metric under test is well correlated with human perception. Both results, the absolute and relative subjective ratings confirm the good correlation of the proposed metric with human perception. The analysis of the eye tracking data highlights differences between experts and consumers.

In this tutorial we discuss interactive video search tools and methods, review their need in the age of deep learning, and explore video and multimedia search challenges and their role as evaluation benchmarks in the field of multimedia information retrieval. We cover three different campaigns (TRECVID, Video Browser Showdown, and the Lifelog Search Challenge), discuss their goals and rules, and present their achieved findings over the last half-decade. Moreover, we talk about datasets, tasks, evaluation procedures, and examples of interactive video search tools, as well as how they evolved over the years. Participants of this tutorial will be able to gain collective insights from all three challenges and use them for focusing their research efforts on outstanding problems that still remain unsolved in this area.

The last decade has seen innovations that make video recording, manipulation, storage, and sharing easier than ever before, thus impacting many areas of life. New video retrieval scenarios emerged as well, which challenge the state-of-the-art video retrieval approaches. Despite recent advances in content analysis, video retrieval can still benefit from involving the human user in the loop. We present our experience with a class of interactive video retrieval scenarios and our methodology to stimulate the evolution of new interactive video retrieval approaches. More specifically, the video browser showdown evaluation campaign is thoroughly analyzed, focusing on the years 2015-2017. Evaluation scenarios, objectives, and metrics are presented, complemented by the results of the annual evaluations. The results reveal promising interactive video retrieval techniques adopted by the most successful tools and confirm assumptions about the different complexity of various types of interactive retrieval scenarios. A comparison of the interactive retrieval tools with automatic approaches (including fully automatic and manual query formulation) participating in the TRECVID 2016 ad hoc video search task is discussed. Finally, based on the results of data analysis, a substantial revision of the evaluation methodology for the following years of the video browser showdown is provided.

Interactive video retrieval tools developed over the past few years are emerging as powerful alternatives to automatic retrieval approaches by giving the user more control as well as more responsibilities. Current research tries to identify the best combinations of image, audio and text features that combined with innovative UI design maximize the tools performance. We present the last installment of the Video Browser Showdown 2015 which was held in conjunction with the International Conference on MultiMedia Modeling 2015 (MMM 2015) and has the stated aim of pushing for a better integration of the user into the search process. The setup of the competition including the used dataset and the presented tasks as well as the participating tools will be introduced . The performance of those tools will be thoroughly presented and analyzed. Interesting highlights will be marked and some predictions regarding the research focus within the field for the near future will be made.

We present a novel dataset for multi-view video and spatial audio. An ensemble of ten musicians from the BBC Philharmonic Orchestra performed in the orchestra's rehearsal studio in Salford, UK, on 25th March 2014. This presented a controlled environment in which to capture a dataset that could be used to simulate a large event, whilst allowing control over the conditions and performance. The dataset consists of hundreds of video and audio clips captured during 18 takes of performances, using a broad range of professional-and consumer-grade equipment, up to 4K video and high-end spatial microphones. In addition to the audiovisual essence, sensor metadata has been captured, and ground truth annotations, in particular for temporal synchronization and spatial alignment, have been created. A part of the dataset has also been prepared for adaptive content streaming. The dataset is released under a Creative Commons Attribution Non-Commercial Share Alike license and hosted on a specifically adapted content management platform.

Information or feedback about users' information needs beyond a brief query is crucial for improving the effectiveness of video search. One paradigm that addresses this issue is search and linking, i.e., after an initial search the user selects an item and requests a set of items that share properties with the one selected. We propose an approach for the linking step using both textual and visual features. We first evaluate the proposed method on the MediaEval 2013 Search and Hyperlinking data set. The video segment from which linking starts cannot be assumed to be relevant and to have well-defined boundaries in a real-world setting, in contrast to the benchmarking data set. We thus investigate the performance of the proposed linking method on actual search results of different systems and using only a coarse estimate of a contextual segment (e.g., scene, story). The linking results starting from a related segment among the top-ranked search results are nearly as good as when using the correct segment as starting point. The results show that contextual information surrounding the segment used for generating the linking results plays a crucial role. When starting from a relevant or at least related item in the search result, the use of context information significantly improves results. For segments related to the query, the linking results are similar in terms of performance to those starting from relevant segments (without context), but contextual information does not improve the results.

This paper describes a content set that has been made available by the European Broadcasting Union (EBU). The content in the set consists of broadcast media content collected from different broadcasters around the world. This content set is made available to the research community in order to evaluate automatic information extraction tools on this broadcast media. The set also contains ground truth data and annotations for several automatic information extraction tasks.

Kernel based methods are widely applied to concept and event detection in video. Recently, kernels working on sequences of feature vectors of a video segment have been proposed for this problem, rather than treating feature vectors of individual frames independently. It has been shown that these sequence-based kernels (based e.g., on the dynamic time warping or edit distance paradigms) outperform methods working on single frames for concepts with inherently dynamic features. Existing work on sequence-based kernels either uses a single type of feature or a fixed combination of the feature vectors of each frame. However, different features (e.g., visual and audio features) may be sampled at different (possibly even irregular) rates, and the optimal alignment between the sequences of features may be different. Multiple kernel learning (MKL) has been applied to similarly structured problems, and we propose MKL for combining different sequence-based kernels on different features for video concept detection. We demonstrate the advantage of the proposed method with experiments on the TRECVID 2011 Semantic Indexing data set.

Automatic quality control for audiovisual media is an important tool in the media production process. In this paper we present tools for assessing the quality of audiovisual content in order to decide about the reusability of archive content. We first discuss automatic detectors for the common impairments noise and grain, video breakups, sharpness, image dynamics and blocking. For the efficient viewing and verification of the automatic results by an operator, three approaches for user interfaces are presented. Finally, we discuss the integration of the tools into a service oriented architecture, focusing on the recent standardization efforts by EBU and AMWA's Joint Task Force on a Framework for Interoperability of Media Services in TV Production (FIMS).

We present a C++ library implementing part 2, 3, 4 and 5 of the MPEG 7 multimedia content description standard, including the updated version finalized in 2004. The library supports handling of MPEG-7 descriptions as trees of typed objects, supporting (de)serialization from/to XML. It has convenient and powerful features such as creation of subtrees by XPath statements and is extensible at runtime. The library is available for Windows, Linux/Unix and Mac OS X. It has been provided under a free use license for several years, downloaded more than 5,200 times and used in a large number of projects. It has been published under GNU LGPL in 2009. This paper discusses the key functionalities of the library as well as some exemplary applications.

A significant amount of work in film and video preservation is dedicated to quality assessment of the content to be archived or reused out of the archive. This paper proposes automatic content analysis algorithms which reduce manual inspection time in software based preservation environments. We list the requirements for such algorithms and tools and show exemplarily on a freeze frame impairment detector how analysis algorithms need to be designed for meeting the requirements. The evaluation has shown that robustness against other impairments (noise and flickering) is an essential part of the algorithm. Successful detection of freeze frame impairments with a minimum length of three frames is achieved. The consideration of human perception is important to achieve low false detection rates. Analysis results are represented in a MPEG-7 standard compliant way. The proposed defect summary visualization tool enables efficient human exploration of visually impaired content.

There are still no established methods for the evaluation of browsing and exploratory search tools. In the (multimedia) information retrieval community evaluations following the Cranfield paradigm (as e.g. used in TRECVID) have been widely adopted. We have applied two TRECVID style fact finding approaches (retrieval and question answering tasks) and a user survey to the evaluation of a video browsing tool. We analyze the correlation between the results of the different methods, whether different aspects can be evaluated independently with the survey, and if a learning effect can be measured with the different methods. The results show that the retrieval task correlates better with the user experience according to the survey than the question answering tasks. It turns out that the survey rather measures the general user experience while different aspects of the usability cannot be analyzed independently.

Video summarization is a useful tool which allows a user to grasp rapidly the essence of a video. In the development of this research topic we propose a new method based on different individual content segmentation and selection tools in a collaborative system. The main innovation of this work is to merge results from different approaches, so as to benefit from their respective qualities. Our system is organized in two phases: first segmentation of the video, second identification of relevant and redundant segments. The final list of selected segments is used to concatenate the video segments and build the final summary. In order to assess the effectiveness of this organization, we evaluate our system with a method based on the TRECVID 2007 BBC rushes summarization evaluation pilot and compare our performance with existing systems.

A considerable amount of work in larger film and video restoration projects is dedicated to manually exploring the audiovisual content in order to estimate the costs for restoration and to plan the restoration. Manual exploration is a significant cost factor. In this paper we propose automatic content analysis algorithms and summarization techniques which allow the reduction of manual inspection time in a software based restoration environment. The throughput requirement for analysis of dust and other defects is reached by sparse application of the detectors in the image sequence while retaining sufficient detection accuracy. Analysis result metadata are represented in a MPEG-7 standard compliant way. The proposed defect summary visualization tools facilitate efficient exploration of visually impaired content by the user.

MPEG-7 is an excellent choice for the description of audiovisual content due to its flexibility and comprehensiveness. The drawback is that these properties also increase the complexity of descriptions and cause ambiguities which hinder interoperability. In order to partly solve these problems, profiles and levels have been proposed, but the definitions of the adopted profiles lack semantic constraints which are necessary for interoperability. We propose a profile for detailed description of audiovisual content that can be used in a broad range of applications. The profile aims at describing single multimedia content entities, allowing a comprehensive structural description of the content, including also audio and visual feature descriptions. The profile defines a set of semantic constraints on the selected tools, which resolve ambiguities in modeling the description and support system interoperability. The proposed profile has been successfully used in a wide range of applications