Multimedia data, such as images and videos, con-tinue to be recorded and created in ever increasing quantities. Multimedia collections on social media, but also private collec-tions of personal vacation and home videos are being collected at a higher rate than ever. This rapid growth makes it difficult, and in many cases impossible, to effectively use this kind of data without the use of automated analysis methods. While many methods have been developed to automate the analysis of multimedia data, users are unable to access the information extracted through this analysis without appropriate interfaces. Although purpose-built user interfaces are important even for use by trained experts, they are essential in facilitating the interaction with large multimedia data collections for laypersons and novices. In this paper, we describe a new and improved version of vitrivr- VR, a prototype multimedia analytics system built for immersive interfaces, in the form in which it will participate in the Video Retrieval for Beginners (VR4B) evaluation campaign. vitrivr- VR aims to improve the video browsing experience for novices through the additional affordances provided by immer-sive interfaces. We describe the current state of the system with a focus on features and adjustments made to support accessibility of large scale multimedia analytics for novices.

Egocentric video has seen increased interest in recent years, as it is used in a range of areas. However, most existing datasets are limited to a single perspective. In this paper, we present the CASTLE 2024 dataset, a multimodal collection containing ego- and exo-centric (i.e., first- and third-person perspective) video and audio from 15 time-aligned sources, as well as other sensor streams and auxiliary data. The dataset was recorded by volunteer participants over four days in a common location and includes the point of view of 10 participants, with an additional 5 fixed cameras providing an exocentric perspective. The entire dataset contains over 600 hours of UHD video recorded at 50 frames per second. In contrast to other datasets, CASTLE 2024 does not contain any partial censoring, such as blurred faces or distorted audio. The dataset is available via https://castle-dataset.github.io/.

Reproducibility is indispensable for transferring explainable-AI algorithms from academic prototypes to production systems. This companion paper documents the artefacts, procedures, and outcomes that reproduce the empirical claims of ''Enhancing Model Interpretability with Local Attribution over Global Exploration'' (ACM MM 2024). We release a containerised archive containing source code, data-serialisation scripts, one-click executables, and a detailed README, all conforming to the ACM Multimedia reproducibility guidelines. The regenerated Insertion and Deletion scores deviate by only 2.2% on average. In addition, an exhaustive 10, 20, 30 3 grid-search over key hyper-parameters reveals a new configuration, (30, 20, 30), that improves the Insertion score of three convolutional backbones by 7.51% without additional code changes. These artefacts provide a rigorous, extensible foundation for future research on local attribution methods. Our code is available at: https://github.com/LMBTough/LA/

With the rapidly increasing amount of multimedia content encountered in every day life, interactive multimedia retrieval continues to rise in importance. While a lot of progress has been made in making interactive multimedia retrieval accessible even to novice users using conventional desktop interfaces, novel interface technologies, such as immersive interfaces, have not received enough attention. With the rise in popularity and the increasing affordability of mixed and virtual reality (VR) devices, new methods to enable interactive multimedia retrieval, especially for novice users, are needed. In this paper, we describe a version of the vitrivr-VR virtual reality multimedia retrieval system enhanced to increase usability by novices. By providing simple and intuitive interactions and interfaces with the most common functionality prominently accessible, and more advanced functionality available for those who want to use it, we expect to be able to provide an effective and enjoyable search experience for novice users and experts alike. Specifically, we simplify the search interface to a single search bar, provide convenient and intuitive VR text input methods, and support simple and familiar results browsing interfaces ideal for novice users.

Within the domain of multimedia formats, 3D models – alongside images, videos, and texts – are rapidly gaining prominence in applications and research. As the uses of 3D models increase and it becomes easier and more accessible to capture and digitally create 3D models, methods are required to allow automatic analysis and retrieval within large collections. While a lot of previous work has focused on geometry-based retrieval, these methods usually require an example 3D model to query, and often struggle to capture appearance information expressed through textures and materials.In this paper, we propose a novel view-based approach that enables embedding of 3D models into multi-modal embedding spaces, by rendering 2D planar projections. Furthermore, our approach addresses the challenge of viewpoint selection for 3D model rendering, in order to maximize its semantic recognition. We demonstrate the capability of our approach using two pre-trained multi-modal embedding models applied to a large collection of modern 3D models.

Virtual reality (VR) applications are increasingly permeating our lives. The immersion provided by VR enables novel interactions with data that would be impossible in conventional environments. Especially regarding multimedia data, VR could overcome existing limitations when browsing videos to find specific scenes. This paper introduces a virtual multimedia drawer, tailored to VR environments, to enable novel ways to interact with videos. A within-subjects design experiment (N=24) was conducted to evaluate the multimedia drawer on user experience, efficiency, and effectiveness. Results show that the multimedia drawer, while taking slightly longer to locate a particular scene within a video for certain types of tasks (i.e., sequence tasks), provides statistically significantly higher levels of enjoyment, novelty, and stimulation and is preferred over conventional timeline-based approaches. Implications of quantitative and qualitative results regarding the design and features of the multimedia drawer as a video browsing method in VR are critically discussed.

This tutorial provides an overview of multimedia retrieval in the context of eXtended Reality (XR), including using virtual and augmented/mixed reality as a user interface for multimedia retrieval, as well as multimedia search tasks addressing content needs for the creation of XR experiences.It will discuss the opportunities and limitations of XR-based search, the evaluation of XR-based multimedia retrieval systems, the demonstration of selected research systems, and open research challenges.

The way we create, consume and interact with multimedia content has changed significantly in recent years with the advent of affordable recording devices and easy sharing and access in the form of mobile phones. With the imminent wave of affordable devices that enable mixed reality experiences and the large variety of devices on the market, interaction with multimedia content is expected to continue to evolve rapidly. This will also drastically affect the entire area of multimedia information retrieval in eXtended Reality (XR), for instance by novel ways to express user needs in VR, result presentation that takes the specific capabilities of XR devices into account, and/or result feedback. This tutorial on Multimedia Retrieval in XR discusses and demonstrates existing solutions and highlights key challenges in this evolving field.

Interactive retrieval with user-friendly and performant interfaces remains a necessity for video retrieval, even in light of significant gains in retrieval performance through multi-modal encoders. In recent years, novel interaction modalities such as virtual reality (VR) and augmented reality (AR) have gained popularity, but the best way to adapt paradigms from traditional retrieval interfaces, especially for result browsing and interaction, remains an open research question. In this paper, we compare two video retrieval interfaces in a controlled setting to gain insight into the differences in video browsing between VR and desktop interfaces. We formulate hypotheses explaining why there might be performance differences between the two interfaces, define metrics to test the hypotheses, and show results based on data gathered at an evaluation campaign. Our results show that VR interfaces can be competitive in browsing performance and indicate that there can even be an advantage when browsing larger result sets in VR.

The past decades have seen an exponential growth in the amount of data which is produced by individuals. Smartphones which capture images, videos and sensor data have become commonplace, and wearables for fitness and health are growing in popularity. Lifelog retrieval systems aim to aid users in finding and exploring their personal history. We present two systems for lifelog retrieval: vitrivr and vitrivr-VR, which share a common retrieval model and backend for multi-modal multimedia retrieval. They differ in the user interface component, where vitrivr relies on a traditional desktop-based user interface and vitrivr-VR has a Virtual Reality user interface. Their effectiveness is evaluated at the Lifelog Search Challenge 2021, which offers an opportunity for interactive retrieval systems to compete with a focus on textual descriptions of past events. Our results show that the conventional user interface outperformed the VR user interface. However, the format of the evaluation campaign does not provide enough data for a thorough assessment and thus to make robust statements about the difference between the systems. Thus, we conclude by making suggestions for future interactive evaluation campaigns which would enable further insights.