Recent advances in video generation models enable visually compelling single clips. However, real-world video creation is inherently continuous and iterative: creators refine content over multiple rounds while maintaining narrative, style, and entity consistency. Existing standalone generators are largely stateless and lack memory of previously generated segments, making it difficult to produce a coherent and consistent video project. To address this gap, we present VideoCreator, a unified video agent that integrates generation and understanding with a project-level memory system. VideoCreator leverages understanding capabilities to perform fine-grained analysis of newly produced content and uses persistent memory to retain and reuse prior context across turns, enabling continuous multi-round creation with consistency throughout the project. The code is in https://github.com/chrisx599/VideoCreator.

The ACM Lifelog Search Challenge (LSC) is an annual comparative benchmarking exercise that brings together researchers in the field of multimedia retrieval to evaluate interactive search systems using a large-scale multimodal lifelog dataset. This paper presents an overview of the ninth edition of the challenge (LSC’26), held as a workshop during the ACM International Conference on Multimedia Retrieval (ICMR ’26) in Amsterdam. To broaden its scope, the workshop now features three submission tracks: the traditional Challenge Track for real-time search performance, a new General Lifelog Research Track for theoretical and architectural advancements, and an additional Open Source Track aimed at enhancing reproducibility and reducing barriers to entry for new participants.

The growing popularity of life-logging technologies presents privacy-related challenges, both for users and bystanders alike. However, traditional anonymisation techniques such as redaction and blurring do not fully protect sensitive visual information while keeping the semantic meaning of lifelog images. In this work, we aim to address this problem by proposing a new approach, which uses a combination of Diffusion Models and Large Language Models to generate synthetic yet semantically similar lifelog images. Our method applies a combination of Text- to-Image and Image-to-Image pipelines to replace original images with generated ones, keeping coherence across events. A user study was also carried out to evaluate the approach and showed promising results in the method's ability to anonymise personal and environmental details while still maintaining meaningful content.

Composed Image Retrieval (CIR) involves retrieving a target image based on a query composed of a reference image and a textual modification. Zero-Shot CIR extends this task by removing the need for labeled triplets during training. Most state-of-the-art (SOTA) methods share a common structure: a vision-language encoder followed by a matching module using Transformers or contrastive learning. Instead of increasing data or model complexity, we wonder that: Can we improve retrieval performance at inference time? To answer this, we propose the Vision Projector (VP)-a lightweight, plug-and-play module that enhances visual representations without retraining. Integrated directly into MagicLens, VP consistently improves performance across CIRR, FashionIQ, and CIRCO. Notably, it boosts MagicLens by 18% on CIRCO, despite not using its strongest variant. Code is available at: https://github.com/baohl00/VisionProjector_ZSCIR.

Lifelog question-answering (QA) involves seeking answers to users' questions from within their personal lifelog. A lifelog consists of passively collected multimodal personal information from the owner's life experiences, including images, biometrics, geolocation data, and textual descriptions. Data in a lifelog can become vast, spanning the user's lifetime. QA for such large collections presents significant challenges: finding the most relevant lifelog events (contexts) that may contain the answer before generating a response. We propose a reranking model designed to improve retrieval accuracy by effectively ranking the relevant contexts. Our model integrates multimodal information from images and text, employing a combination of visual extractors and language models. We experiment with three visual extractor models, Vision Transformer, BLIP2, and CLIP, as well as three language models, namely BERT, MiniLM, and ModernBERT. Compared with a retrieval baseline using cosine similarity ranking from Stella-1.5B embeddings, our experimental results on two lifelog QA datasets demonstrate a substantial improvement. Recall@1 is observed to increase from 37.65 % to 65.57% and Precision@1 from 52.65% to 85.88% when using the ModernBERT +ViT reranking model for the OpenLifelogQA task. These findings show the robustness of multimodal reranking in context selection for lifelog QA and provide a mechanism for accurate and efficient retrieval in lifelog applications.

For the eighth time since 2018, the ACM Lifelog Search Challenge (LSC) was run to compare interactive lifelog search systems in a live metrics-based challenge. The goal of the LSC workshop is to comparatively evaluate the capabilities of systems accessing a large multimodal lifelog. LSC'25 attracted eleven participating teams, each of which had developed an innovative interactive lifelog retrieval system. The benchmark was organised in a hybrid manner (due to political issues in 2025) at the LSC workshop at ACM ICMR'25 in Chicago, USA. This short paper summarises the LSC workshop setting and presents the participating lifelog search systems.

Lifelogging is the passive collection, storage and analysis of daily data through wearable sensors. Question Answering (QA) for lifelog data enables natural language interactions with personal daily life records, providing insights into individual routines and behaviours. While this task has great potential for personal analytics and memory augmentation, progress has been limited due to the challenges of lifelog management, since they can comprise of enormous multi-modal data sets spanning a lifetime. We introduce a Retrieval-Augmented Generation (RAG) approach for addressing the lifelog QA task. A RAG approach first includes a retrieval model finding the correct lifelog events containing answers and then a large language model (LLM) generating answers from the questions. In addition, we construct an open-ended lifelog QA benchmark with 14,187 QA pairs to examine the RAG approach to lifelog QA. Using an embedding-based retrieval approach, our lifelog context retriever achieves a performance of 77.67% Recall@5 and 94.35% Recall@20 using an embedding-based retrieval approach with the Stella 1.5B model. Combined with the Mistral 7B model, the model achieves scores of 39.54% ROUGE-L and 3.475 Accuracy on a scale of 5 scored by GPT-4o. This approach potentially provides an effective approach to lifelog QA with high performance that does not require fine-tuning.

The Second ACM Workshop on AI-Powered Question & Answering Systems for Multimedia (AIQAM'25) was held on 27 October 2025 in Dublin, Ireland, co-located with ACM Multimedia 2025. The workshop's main objective is to create a collaborative and inclusive space for researchers at the intersection of Artificial Inteligence (AI), large language models (LLMs), multimodal information retrieval, and question answering systems. Building on the success of its first edition (AIQAM'24) at ICMR 2024, AIQAM'25 provided a forum for presenting novel methods, applications, and surveys, which address the challenges of integrating text, image, audio, and video data into QA systems. The programme featured contributions ranging from methodological advances in reasoning and evaluation frameworks, domain-specific applications in education and sustainability, to a survey work reviewing the state of multimedia retrieval-augmented QA. This summary paper outlines the objectives and scope of the workshop, describes its format, highlights the keynote and accepted contributions, and acknowledges the efforts of the organising committee.

Egocentric video has seen increased interest in recent years, as it is used in a range of areas. However, most existing datasets are limited to a single perspective. In this paper, we present the CASTLE 2024 dataset, a multimodal collection containing ego- and exo-centric (i.e., first- and third-person perspective) video and audio from 15 time-aligned sources, as well as other sensor streams and auxiliary data. The dataset was recorded by volunteer participants over four days in a common location and includes the point of view of 10 participants, with an additional 5 fixed cameras providing an exocentric perspective. The entire dataset contains over 600 hours of UHD video recorded at 50 frames per second. In contrast to other datasets, CASTLE 2024 does not contain any partial censoring, such as blurred faces or distorted audio. The dataset is available via https://castle-dataset.github.io/.

The inaugural edition of the CASTLE grand challenge was held at ACM Multimedia 2025. The focus of the CASTLE challenge is to advance the state-of-the-art in analysis and understanding of multimodal data, especially centered around multistream ego- and exo-centric video. In this first instance of the challenge, participants had to solve three types of tasks - event instance search, object instance search, and question answering - in a fully automatic or interactive setting.

We present SLIVeR (Someone else's Lifelog in Virtual Reality), an interactive system that reimagines lifelog data through narrative-based Virtual Reality (VR). Instead of passively viewing chronological data, users navigate personal history through cinematic scenes and existential prompts embedded in a memory-reconstruction storyline. Guided by life-oriented questions, users progress from disorientation to recollection using curated lifelog clips. Built in Unity and deployed on Meta Quest 3, SLIVeR transforms lifelogging into a reflective, game-like journey that blends storytelling, gamification, and immersive visuals to enhance user engagement.

The demand for identity-preserving Facial Aesthetic Enhancement (FAE) has surged in social media and digital entertainment. However, existing methods based on deep generative models encounter difficulties in striking a balance between fine-grained detail enhancement and preserving the unique identities of individuals from diverse ethnic and gender backgrounds. To tackle this issue, this paper proposes a novel tuning-based framework that integrates prototype-based hierarchical prompt learning within a CLIP model and a StyleGAN-based inversion model. Our approach first adapts a pre-trained StyleGAN to the input face via pivotal tuning, optimizing around pivotal latent codes to minimize reconstruction distortion while retaining editability. Then, a prototype-based hierarchical prompt learning module is designed for learning multigrained facial features to achieve comprehensive and fine-grained facial descriptions for FAE. Specifically, we propose a prototypical similarity measure based on a multi-ethnic dataset to select geometrically similar faces with high aesthetic scores as reference faces. This selection is guided by ArcFace regularization within categorized gender and ethnic groups to minimize identity loss. Additionally, we design a novel aesthetic attribute selection algorithm to generate generic fine-grained aesthetic attributes from these reference faces for detailed facial descriptions. These components work synergistically through dynamic weight modulation, prioritizing features with high aesthetic contributions (such as enhancing lip fullness) while ensuring semantic consistency through CLIP-driven optimization for pivotal latent codes. Extensive experiments demonstrate that our method outperforms state-of-the-art techniques in both aesthetic quality and identity preservation, especially for out-of-domain faces.

We present the VideoEase interactive video retrieval system, which we used to participate in VBS2025. This is the first time that VideoEase has taken part in the VBS challenge. VideoEase is built on the Milvus vector database, which supports vector searches on massive datasets with high-dimensional vectors. The CLIP, BLIP2, and OpenCLIP models play a crucial role in encoding keyframe images and queries into embeddings. A new user interface with simple yet effective components is also introduced. We experiment with VBS24 queries to evaluate the performance of the VideoEase system. Our results show that answer rank is improved by merging outputs from several models with appropriate weights.

Document layout analysis, a critical process in automated document processing, traditionally relies on object detection techniques, primarily focusing on the structural segmentation of documents. However, these approaches often fall short in comprehensively understanding the semantic content within the text, leading to a disjointed analysis of document structure and content. To address this, we propose a novel methodology that combines text clustering with multi-modal graph convolution networks, aiming to integrate structural detection with semantic understanding. Our approach starts with text detection, followed by encoding using a large language model. Subsequently, we integrate visual and positional data using Graph Neural Networks to perform clustering, creating a synergy between the textual and structural aspects of documents. Extensive experiments on mainstream datasets demonstrate that our method significantly outperforms existing approaches, especially in understanding text-centric document layouts. This paper contributes to the field by offering a novel, semantically-enriched approach to document layout analysis, enhancing the capabilities of automated document processing systems in handling diverse and complex document formats.

Efficient extraction of critical information from receipt is essential for automating financial processes and supporting timely decision-making in businesses. However, this process faces significant challenges, starting with variations in the quality of scanned receipt images due to differences in scanning equipment, followed by the complexity of diverse receipt formats, and further complicated by handwritten elements and noise, making accurate extraction particularly difficult. Therefore, to address these issues, we propose a model framework called DocExtractNet, based on LayoutLMv3, designed for extracting key information from receipt. Firstly, we introduce the ImageEnhance method to process image modality features, enhancing image clarity and significantly improving recognition accuracy for low-quality images. Then, we implement the PrecisionHints strategy to supplement missing key–value pairs in the text modality, improving data integrity and the model’s overall performance. Furthermore, we apply the CrossModalFusion method to combine both image and text features, allowing the model to better understand and extract receipt information. The experimental results on the Finance-Receipts, FUNSD, and CORD datasets show that DocExtractNet significantly improves F1 scores compared to other models, with F1 scores reaching 97.07% for Finance-Receipts, 91.80% for FUNSD, and 97.38% for CORD, highlighting its superior performance in receipt information extraction. • Introduced a novel model for key information extraction from invoices. • Enhanced multimodal information extraction by integrating data from different modalities. • Achieved significant accuracy improvements in complex invoice datasets through innovative fine-tuning strategies. • Optimized model structure and parameters, leading to increased processing efficiency for invoice information extraction tasks.

Understanding user engagement and behavior during reading activities is important for multimedia system design as it can provide valuable insights into developing more effective interactive systems. This paper proposes a novel approach to predicting multiple reading tasks based on eye movement measures. We conducted experimental research involving participant trials across various reading tasks, collecting time-series eye movement data. Our method employs Convolutional Neural Networks (CNNs) with Bidirectional Long Short-Term Memory (BiLSTM) to identify patterns associated with four reading tasks: skimming, reading, scanning, and proofreading. Importantly, this method doesn't require experts to label local windows manually and requires minimal data pre-processing, as the model autonomously extracts features. The proposed model demonstrates promising performance, achieving a multiclass classification accuracy of$\mathbf{7 0. 0 4 \%}$. The experimental results indicate that the proposed model surpasses the performance of the other methods examined in the study. Furthermore, our proposed method holds the potential for real-time reading task classification, which can deepen our understanding of user behavior and interaction patterns. By enabling accurate prediction of reading tasks, our approach facilitates the development of enhanced multimedia browsing and search systems tailored to user preferences and goals.

Our paper introduces a novel middleware designed to enhance the user experience of novice users of video event retrieval systems. The middleware simplifies the search process while maintaining advanced semantic information-seeking features. It achieves this by refining sentence representation, rewriting the original query based on user intent and dataset metadata knowledge, reducing ambiguities, and increasing relevance with domain-specific terms. Our approach allows users to provide simplified descriptions, while the system automates query refinement and parameterizes these for different search functions. This reduces the difficulty of navigating complex user interfaces and selecting from available functionalities. Additionally, the multilingual feature supports non-native English speakers by helping them describe their queries more effectively. In general, our system is suitable for people who are not experienced in advanced retrieval tasks, specifically focusing on its applicability in the special session Interactive Video Retrieval for Beginners (IVR4B).

In this paper, we present the VidBasys interactive video retrieval system for novice users, an upgraded version of VideoCLIP 2.0 that participated in the Video Browser Showdown 2024. While the novel user interface is designed in a more user-friendly way for newbies to accommodate the target of the Interactive Video Retrieval for Beginner (IVR4B), the core search engine is enhanced with the advance of the recent CLIP model to bridge the gap in semantics between image and text. This version is designed to focus on novice users with a simple, easy-to-use but effective user interface. The system supports freetext search to enhance the user experience and minimise the number of actions required for filtering. The new user interface supports simple search and filters with clearly designed freetext search boxes. In addition, the retrieved results are displayed in an optimised layout to maximise image display space and minimise user interactions. The improvements are expected to support novice users in accurately retrieving the desired videos.

For the seventh time since 2018, the Lifelog Search Challenge (LSC) benchmarked interactive lifelog search systems in a live challenge. The LSC goal is to comparatively evaluate system capabilities to access large multimodal lifelogs comprising hundreds of thousands of records. LSC'24 attracted an unprecedented record number of twenty-one participating teams, where each team proposes innovative ideas implemented to new or already established interactive lifelog retrieval systems. The benchmark was organised in front of a live audience at the LSC workshop at ACM ICMR'24 in Phuket, Thailand. This short paper summarises the LSC workshop setting and presents the participating lifelog search systems.

The advent of large language models (LLMs) has energised research in Question-Answering (QA) tasks, enabling responses across varied domains like economics and mathematics. Despite their capabilities, LLMs often lack explainability due to their complex parameter embeddings. Additionally, integrating multimedia data into QA systems introduces challenges in processing and interpreting diverse data types such as text, images, audio, and video. This necessitates sophisticated algorithms for accurate information retrieval across media while ensuring the reliability of the data and responses remains a significant challenge. The AIQAM workshop aims to bring together researchers and practitioners to address these challenges and enhance QA systems with multimedia data. The focus is on promoting innovations that improve the accuracy, explainability, and trustworthiness of QA systems, contributing to the development of the field.

In the realm of video understanding, Video Moment Retrieval (VMR) is an important yet challenging task that aims to locate the boundary of a moment of interest within a long untrimmed video. Existing VMR methods often focus on the visual content extracted from the video only (or frame sequences), however, the rich semantic information at the object level that describes the image's content has not been explored yet. To overcome those limitations, we propose PaTF, an attention-based Parallel Transformer Framework that enriches the feature representations by exploring both low-level visual cues and high-level relational contexts of video-query pairs. Our framework consists of two parallel transformers: one for the visual-textual stream and the other for the semantic-textual stream. The visual-textual stream extracts the links between global visual features and textual information, while the semantic-textual stream emphasises the relations between objects via scene graph representations. Furthermore, our comprehensive experiment conducted on the Charades-STA dataset demonstrates that the proposed framework outperforms the state-of-the-art methods by a large margin, roughly 5% and 7% at Recall@1 with IoU = 0.5 and IoU = 0.7, respectively.

Lifelogging is the process of automatically logging data about an individual's daily life, which can then be used in various domains, such as behavior analysis and health monitoring. Various technological devices, including wearable cameras and smartwatches, can help record lifelog data, but getting access to lifelog cameras has proven difficult in recent years, due to a lack of such devices on the market. Creating a lifelog camera that is not only easy to use and cost-efficient but also provides comprehensive functions to log all images about life is challenging due to the lack of hardware and software. This paper introduces MemoriLens, a low-cost camera that efficiently collects, organizes, and stores lifelog data using a readily available custom-designed Raspberry Pi Zero board. The camera is designed to capture images automatically and send them to a private account in cloud services for storage. We open-source the implementing of the camera at: https://github.com/linh222/raspberry_lifelog_camera and we encourage lifelog researchers to use our designs and software as required.

Supporting Question Answering (QA) tasks is the next step for lifelog retrieval systems, similar to the progression of the parent field of information retrieval. In this paper, we propose a new pipeline to tackle the QA task in the context of lifelogging, which is based on the open-domain QA pipeline. We incorporate this pipeline into a multimodal lifelog retrieval system, which allows users to submit questions prevalent to a lifelog and then suggests possible text answers based on multimodal data. A test collection is developed to facilitate the user study, the aim of which is to evaluate the effectiveness of the proposed system compared to a conventional lifelog retrieval system. The results show that the proposed system is more effective than the conventional system, in terms of both effectiveness and user satisfaction. The results also suggest that the proposed system is more valuable for novice users, while both systems are equally effective for experienced users.

In this paper, we present the second release of VideoCLIP, an interactive CLIP-based video retrieval system that participated in the Video Browser Showdown 2023. While we continue to use the underlying architecture to map the content between image and text, we concentrate on improving the user experience for novice users. Specifically, we have implemented three different query modalities and redesigned the user interface in order to adapt to the context of the Interactive Video Retrieval for Beginners (IVR4B) workshop. These modifications ultimately aim to provide newcomers with a simple and efficient user experience to locate the desired videos.

Wearable cameras provide valuable new sources of data for health and wellness monitoring, however, such visual data brings privacy concerns. This paper proposes a prototype egocentric face de-identification system for wearable camera images by swapping the original faces with synthetic faces. The motivation of this paper is to: (1) de-identify faces in egocentric images and (2) preserve the existence of each identity in images where the source identity is altered. The system incorporates our proposed method, which promises a privacy-aware and cost-effective approach. We evaluated the system on the Ego4D audio-visual PoV diarization training set by analysing six activities where faces are visible in wearable camera data. The results show promising de-identification on the source faces while most existences remain.

For the sixth time since 2018, the Lifelog Search Challenge (LSC) was organized as a comparative benchmarking exercise for various interactive lifelog search systems. The goal of this international competition is to test system capabilities to access large multimodal lifelogs. LSC’23 attracted twelve participanting teams, each of whom had developed a competitive interactive lifelog retrieval system. The benchmark was organized in front of live audience at the LSC workshop at ACM ICMR’23. As in previous editions, this introductory paper presents the LSC workshop and introduces the participating lifelog search systems.

Recently, there has been an increased interest in cross-data research problems, such as predicting air quality using life logging images, predicting congestion using weather and tweets data, and predicting sleep quality using daily exercises and meals. Although several research focusing on multimodal data analytics have been performed, few studies have been conducted on cross-data research (e.g., cross-modal data, cross-domain, cross-platform). The article collection “Intelligent Cross-Data Analysis and Retrieval” aims to encourage research in intelligent cross-data analytics and retrieval and contribute to the creation of a sustainable society. Researchers from diverse domains such as well-being, disaster prevention and mitigation, mobility, climate, tourism and healthcare are welcome to contribute to this Research Topic.

Video-language learning has attracted significant attention in the fields of multimedia, computer vision and natural language processing in recent years. One of the key challenges in this area is how to effectively integrate visual and linguistic information to enable machines to understand video content and query information. In this work, we leverage graph-based representations and multi-grained audio-visual alignment to address this challenge. First, our approach starts by transforming video and query inputs into visual-scene graphs and semantic role graphs using a visual-scene parser and semantic role labeler respectively. These graphs are then encoded using graph neural networks to obtain enriched representations and combined to obtain a video-query joint representation that enhances the semantic expressivity of the inputs. Second, to achieve accurate matching of relevant parts of audio and visual features, we propose a multi-grained alignment module that aligns the audio and visual features at multiple scales. This enables us to effectively fuse the audio and visual information in a way that is consistent with the semantic-level information captured by the graph-based representations. Experiments on five representative datasets collected for Video Retrieval and Video Question Answering tasks show that our approach outperforms the literature on several metrics. Our extensive ablation studies demonstrate the effectiveness of graph-based representation and multi-grained audio-visual alignment.

Abstract Lifelogging is a form of personal data collection which seeks to capture the totality of one’s experience through intelligent technology and sensors. Yet despite notable advancement in such technologies, there remain persistent challenges to developing interactive systems to analyse the types of large-scale personal collections often generated by lifelogging. In response to this, we present the Lifelog Application Design (LAD) model which is intended to address these challenges and support the design of more novel interactive systems that may target a broader range of application use cases. The model is deliberately structured to remain impartial to the specific personal data, technology platform, or application criterion, to provide maximum utility across the domain. We demonstrate this utility by exploring two case studies and a retrospective analysis of VRLE, a real-world application prototype developed to examine the potential of large-scale personal data retrieval in virtual reality. This work is based on the accumulation of insights garnered from involvement in a number of collaborative lifelogging projects over the past decade. It is our goal to encourage future researchers to utilise the LAD model to support the design and development of their own application prototypes and further solidify the model’s contribution to the domain as a whole.

In this paper, we attempt to fine-tune the CLIP (Contrastive Language-Image Pre-Training) model on the Lifelog Question Answering dataset (LLQA) to investigate retrieval performance of the fine-tuned model over the zero-shot baseline model. We train the model adopting a weight space ensembling approach using a modified loss function to take into account the differences in our dataset (LLQA) when compared with the dataset the CLIP model was originally pretrained on. We further evaluate our fine-tuned model using visual as well as multimodal queries on multiple retrieval tasks, demonstrating improved performance over the zero-shot baseline model.

We have witnessed the rise of cross-data against multimodal data problems recently. The cross-modal retrieval system uses a textual query to look for images; the air quality index can be predicted using lifelogging images; the congestion can be predicted using weather and tweets data; daily exercises and meals can help to predict the sleeping quality are some examples of this research direction. Although vast investigations focusing on multimodal data analytics have been developed, few cross-data (e.g., cross-modal data, cross-domain, cross-platform) research has been carried on. In order to promote intelligent cross-data analytics and retrieval research and to bring a smart, sustainable society to human beings, the specific article collection on "Intelligent Cross-Data Analysis and Retrieval" is introduced. This Research Topic welcomes those who come from diverse research domains and disciplines such as well-being, disaster prevention and mitigation, mobility, climate change, tourism, healthcare, and food computing

For the fifth time since 2018, the Lifelog Search Challenge (LSC) facilitated a benchmarking exercise to compare interactive search systems designed for multimodal lifelogs. LSC'22 attracted nine participating research groups who developed interactive lifelog retrieval systems enabling fast and effective access to lifelogs. The systems competed in front of a hybrid audience at the LSC workshop at ACM ICMR'22. This paper presents an introduction to the LSC workshop, the new (larger) dataset used in the competition, and introduces the participating lifelog search systems.

Recollecting details from lifelog data involves a higher level of granularity and reasoning than a conventional lifelog retrieval task. Investigating the task of Question Answering (QA) in lifelog data could help in human memory recollection, as well as improve traditional lifelog retrieval systems. However, there has not yet been a standardised benchmark dataset for the lifelog-based QA. In order to provide a first dataset and baseline benchmark for QA on lifelog data, we present a novel dataset, LLQA, which is an augmented 85-day lifelog collection and includes over 15,000 multiple-choice questions. We also provide different baselines for the evaluation of future works. The results showed that lifelog QA is a challenging task that requires more exploration. The dataset is publicly available at https://github.com/allie-tran/LLQA .

Letters, diaries, postcards, photo albums, home videos, and lifelogs! These are artefacts of our personal history, they represent how we cherish and preserve memories, re-engage with our past and share our experiences with others. In this demonstration paper, we explore an Virtual Reality (VR) approach to help people reminisce about the past through lifelogs. Our user study found that most participants enjoyed the experience, although for some, the VR environment was overwhelming.

Cross-data analytics and retrieval have gained significant improvement recently. People can now extract more data insights precisely and quickly towards having many excellent applications serving human lives. Since people create multimedia and other types of data that reflect the diverse perspectives of human lives, these data are just pieces of the puzzle of the world's pictures. Hence, it is necessary to assembly all these pieces towards having a better solution for human-centered problems. Hence, the workshop welcomes those who work with multimedia and others and come from diverse research domains and disciplines to work on intelligent cross-data analytics and retrieval to bring a smart, sustainable society to human beings. The research domain can vary from well-being, disaster prevention and mitigation, mobility to food computing, to name a few.

The Lifelog Search Challenge (LSC) is an annual benchmarking challenge for comparing approaches to interactive retrieval from multi-modal lifelogs. LSC'21, the fourth challenge, attracted sixteen participants, each of which had developed interactive retrieval systems for large multimodal lifelogs. These interactive retrieval systems participated in a comparative evaluation in front of an online live-audience at the LSC workshop at ACM ICMR'21. This overview presents the motivation for LSC'21, the lifelog dataset used in the competition, and the participating systems.

The rising popularity of Artificial Intelligence (AI) has brought considerable public interest as well faster and more direct transfer of research ideas into practice. One of the aspects of AI that still trails behind considerably is the role of machines in interpreting, enhancing, modeling, generating, and influencing social behavior. Such behavior is captured as social signals, usually by sensors recording multiple modalities, making it classic multimedia data. Such behavior can also be generated by an AI system when interacting with humans. Using AI techniques in combination with multimedia data can be used to pursue multiple goals, two of which are high-lighted here. First, supporting people during social interactions and helping them to fulfil their social needs either actively or passively.Second, improving our understanding of how people collaborate, build relationships, and process self identity. Despite the rise of fields such as Social Signal Processing, a similar panel organised at ACM Multimedia 2014, and an area on social and emotional signal sat the ACM MM since 2014, we argue that we have yet to truly fulfil the potential of the combining social signals and multimedia. This panel asks where we have come far enough and what remaining challenges there are in light of recent global events.

Conventional approaches to image-text retrieval mainly focus on indexing visual objects appearing in pictures but ignore the interactions between these objects. Such objects occurrences and interactions are equivalently useful and important in this field as they are usually mentioned in the text. Scene graph presentation is a suitable method for the image-text matching challenge and obtained good results due to its ability to capture the inter-relationship information. Both images and text are represented in scene graph levels and formulate the retrieval challenge as a scene graph matching challenge. In this paper, we introduce the Local and Global Scene Graph Matching (LGSGM) model that enhances the state-of-the-art method by integrating an extra graph convolution network to capture the general information of a graph. Specifically, for a pair of scene graphs of an image and its caption, two separate models are used to learn the features of each graph’s nodes and edges. Then a Siamese-structure graph convolution model is employed to embed graphs into vector forms. We finally combine the graph-level and the vector-level to calculate the similarity of this image-text pair. The empirical experiments show that our enhancement with the combination of levels can improve the performance of the baseline method by increasing the recall by more than 10% on the Flickr30k dataset. Our implementation code can be found at https://github.com/m2man/LGSGM.

Identifying stress level can provide valuable data for mental health analytics as well as labels for annotation systems. Although much research has been conducted into stress detection models using heart rate variability at a higher cost of data collection, there is a lack of research on the potential of using low-resolution Electrodermal Activity (EDA) signals from consumer-grade wearable devices to identify stress patterns. In this paper, we concentrate on performing statistical analyses on the stress detection capability of two popular approaches of training stress detection models with stress-related biometric signals: user-dependent and user-independent models. Our research manages to show that user-dependent models are statistically more accurate for stress detection. In terms of effectiveness assessment, the balanced accuracy (BA) metric is employed to evaluate the capability of distinguishing stress and non-stress conditions of the models trained on either low-resolution or high-resolution Electrodermal Activity (EDA) signals. The results from the experiment show that training the model with (comparatively low-cost) low-resolution EDA signal does not affect the stress detection accuracy of the model significantly compared to using a high-resolution EDA signal. Our research results demonstrate the potential of attaching the user-dependent stress detection model trained on personal low-resolution EDA signal recorded to collect data in daily life to provide users with personal stress level insight and analysis.

Comprehensive and fair performance evaluation of information retrieval systems represents an essential task for the current information age. Whereas Cranfield-based evaluations with benchmark datasets support development of retrieval models, significant evaluation efforts are required also for user-oriented systems that try to boost performance with an interactive search approach. This article presents findings from the 9th Video Browser Showdown, a competition that focuses on a legitimate comparison of interactive search systems designed for challenging known-item search tasks over a large video collection. During previous installments of the competition, the interactive nature of participating systems was a key feature to satisfy known-item search needs, and this article continues to support this hypothesis. Despite the fact that top-performing systems integrate the most recent deep learning models into their retrieval process, interactive searching remains a necessary component of successful strategies for known-item search tasks. Alongside the description of competition settings, evaluated tasks, participating teams, and overall results, this article presents a detailed analysis of query logs collected by the top three performing systems, SOMHunter, VIRET, and vitrivr. The analysis provides a quantitative insight to the observed performance of the systems and constitutes a new baseline methodology for future events. The results reveal that the top two systems mostly relied on temporal queries before a correct frame was identified. An interaction log analysis complements the result log findings and points to the importance of result set and video browsing approaches. Finally, various outlooks are discussed in order to improve the Video Browser Showdown challenge in the future.

Nowadays, recommendation systems play an indispensable role in many fields, including e-commerce, finance, economy, and gaming. There is emerging research on publication venue recommendation systems to support researchers when submitting their scientific work. Several publishers such as IEEE, Springer, and Elsevier have implemented their submission recommendation systems only to help researchers choose appropriate conferences or journals for submission. In this work, we present a demo framework to construct an effective recommendation system for paper submission. With the input data (the title, the abstract, and the list of possible keywords) of a given manuscript, the system recommends the list of top relevant journals or conferences to authors. By using state-of-the-art techniques in natural language understanding, we combine the features extracted with other useful handcrafted features. We utilize deep learning models to build an efficient recommendation engine for the proposed system. Finally, we present the User Interface (UI) and the architecture of our paper submission recommendation system for later usage by researchers.

The Lifelog Search Challenge (LSC) is an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multi-modal lifelogs. LSC'20, the third such challenge, attracts fourteen participants with their interactive lifelog retrieval systems. These systems are comparatively evaluated in front of a live-audience at the LSC workshop at ACM ICMR'20 in Dublin, Ireland. This overview motivates the challenge, presents the dataset and system configuration used in the challenge, and briefly presents the participating teams.

With the rapid growth of users' data in SaaS (Software-as-a-service) platforms using micro-services, it becomes essential to detect duplicated entities for ensuring the integrity and consistency of data in many companies and businesses (primarily multinational corporations). Due to the large volume of databases today, the expected duplicate detection algorithms need to be not only accurate but also practical, which means that it can release the detection results as fast as possible for a given request. Among existing algorithms for the deduplicate detection problem, using Siamese neural networks with the triplet loss has become one of the robust ways to measure the similarity of two entities (texts, paragraphs, or documents) for identifying all possible duplicated items. In this paper, we first propose a practical framework for building a duplicate detection system in a SaaS platform. Second, we present a new active learning schema for training and updating duplicate detection algorithms. In this schema, we not only allow the crowd to provide more annotated data for enhancing the chosen learning model but also use the Siamese neural networks as well as the triplet loss to construct an efficient model for the problem. Finally, we design a user interface of our proposed deduplicate detection system, which can easily apply for empirical applications in different companies.

The automatic recognition of risk situations for frail people is an urgent research topic for the interdisciplinary artificial intelligence and multimedia community. Risky situations can be recognized from lifelog data recorded with wearable devices. In this paper, we present a new approach for the detection of semantic risk situations for frail people in lifelog data. Concept matching between general lifelog and risk taxonomies was realized and tuned AlexNet was deployed for detection of two semantic risks situations such as risk of domestic accident and risk of fraud with promising results.

Lifelogging is a phenomenon whereby an individual digitally records his/her personal life experiences, for a variety of purposes. Activity recognition and segmentation is fundamental to many of the use cases in lifelogging. However, detecting sufficiently robust user activity boundaries that could be deployed with confidence in a subjective real-world setting remains a challenge. In this paper, we extend our previous work on identifying a better activity recognition and segmentation approach to multimodal lifelog data, primarily through the introduction of automatic thresholding techniques, but also through revising the criteria for selecting the most appropriate size of sliding window when evaluating the proposed algorithms. We use an open and publicly available lifelog test collection over a time period of 27 days with manual annotations and manually groundtruthed activities.

Notwithstanding recent advances in machine vision, video activity recognition from multiple cameras still remains a challenging task as many real-world interactions cannot be automatically recognised for many reasons, such as partial occlusion or coverage black-spots. In this paper we propose a new technique that infers the unseen relationship between two individuals captured by different cameras and use it to retrieve relevant video clips if there is a likely interaction between the two individuals. We introduce a human object interaction (HOI) model integrating the causal relationship between the humans and the objects. For this we first extract the key frames and generate the labels or annotations using the state-of-the-art image captioning models. Next, we extract SVO (subject, verb, object) triples and encode the descriptions into a vector form for HOI inference using the Stanford CoreNLP parser. In order to calculate the HOI co-existence and the possible causality score we use transfer entropy. From our experimentation, we found that integrating casual relations into the content indexing process and using transfer entropy to calculate the causality score leads to improvement in retrieval performance.

We present a tutorial focusing on video retrieval tasks, where state-of-the-art deep learning approaches still benefit from interactive decisions of users. The tutorial covers general introduction to the interactive video retrieval research area, state-of-the-art video retrieval systems, evaluation campaigns and recently observed results. Moreover, a significant part of the tutorial is dedicated to a practical exercise with three selected state-of-the-art systems in the form of an interactive video retrieval competition. Participants of this tutorial will gain a practical experience and also a general insight of the interactive video retrieval topic, which is a good start to focus their research on unsolved challenges in this area.

The multimedia and multi-modal community is witnessing an explosive transformation in the recent years with major societal impact. With the unprecedented deployment of multimedia devices and systems, multimedia research is critical to our abilities and prospects in advancing state-of-the-art technologies and solving real-world challenges facing the society and the nation. To respond to these challenges and further advance the frontiers of the field of multimedia, this panel will discuss the challenges and visions that may guide future research in the next ten years.

The task intelligence workshop at the 2019 ACM Web Search and Data Mining (WSDM) conference comprised a mixture of research paper presentations, reports from data challenge participants, invited keynote(s) on broad topics related to tasks, and a workshop-wide discussion about task intelligence and its implications for system development.

In this paper, we propose a deep learning based system for food recognition from personal life archive im- ages. The system first identifies the eating moments based on multi-modal information, then tries to focus and enhance the food images available in these moments, and finally, exploits GoogleNet as the core of the learning process to recognise the food category of the images. Preliminary results, experimenting on the food recognition module of the proposed system, show that the proposed system achieves 95.97% classification accuracy on the food images taken from the personal life archive from several lifeloggers, which potentially can be extended and applied in broader scenarios and for different types of food categories.

Nowadays, almost everyone holds some form or other of a personal life archive. Automatically maintaining such an archive is an activity that is becoming increasingly common, however without automatic support the users will quickly be overwhelmed by the volume of data and will miss out on the potential benefits that lifelogs provide. In this paper we give an overview of the current status of lifelog research and propose a concept for exploring these archives. We motivate the need for new methodologies for indexing data, organizing content and supporting information access. Finally we will describe challenges to be addressed and give an overview of initial steps that have to be taken, to address the challenges of organising and searching personal life archives.

In this tutorial we discuss interactive video search tools and methods, review their need in the age of deep learning, and explore video and multimedia search challenges and their role as evaluation benchmarks in the field of multimedia information retrieval. We cover three different campaigns (TRECVID, Video Browser Showdown, and the Lifelog Search Challenge), discuss their goals and rules, and present their achieved findings over the last half-decade. Moreover, we talk about datasets, tasks, evaluation procedures, and examples of interactive video search tools, as well as how they evolved over the years. Participants of this tutorial will be able to gain collective insights from all three challenges and use them for focusing their research efforts on outstanding problems that still remain unsolved in this area.

Lifelog, Human Activity Recognition, Smartphones, Embedded Sensors, Smart-City, Heterogeneous Sensory Data Analytics.\nThis paper introduces a new idea for sensor data analytics, named PHASOR, that can recognize and stream individual human activities online. The proposed sensor concept can be utilized to solve some emerging problems in smartcity domain such as health care, urban mobility, or security by creating a lifelog of human activities. PHASOR is created from three ‘components’: ID, model, and Sensor. The first component is to identify which sensor is used to monitor which object (e.g., group of users, individual users, type of smart- phone). The second component decides suitable classifiers for human activities recognition. The last one includes two types: (1) physical sensors that utilize embedded sensors in smartphones to recognize human activities, (2) human factors that uses human interaction to personally increase the accuracy of the detection. The advantage of PHASOR is the error signal is inversely proportional to its lifetime, which is well-suited for lifelogging applications. The proposed concept is evaluated and compared to de-facto datasets as well as state-of-the-art of Human Activity Recognition (HAR) using smartphones, confirming that applying PHASOR can improves the accuracy of HAR.

Ever since the emergence of digitization, we've used the term multimedia to represent a combination of different kinds of media types, such as images, audio, and videos. As new sensing technologies emerge and are now becoming omnipresent in daily lives, the definition, role and significance of multimedia is changing. Multimedia now represents the means for communicating, cooperating, and also for monitoring numerous aspects of daily life, at various levels of granularity and application, ranging from personal to societal. With this shift, we have since moved from comprehending single media and its state toward comprehending media in terms of its use context.

The organisation of personal data is receiving increasing research attention due to the challenges we face in gathering, enriching, searching, and visualising such data. Given the increasing ease with which personal data being gathered by individuals, the concept of a lifelog digital library of rich multimedia and sensory content for every individual is fast becoming a reality. The LTA 2017 workshop aims to bring together academics and practitioners to discuss approaches to lifelog data analytics and applications; and to debate the opportunities and challenges for researchers in this new and challenging area.

Ever since the emergence of digitization, the term multimedia has been used to represent a combination of different kinds of media types, such images, audio, and videos. As new sensing technologies emerge and are now becoming omnipresent in daily lives, the definition, role and significance of multimedia is changing. Multimedia now represents the means for communicating, cooperating, and also for monitoring numerous aspects of daily life, at various levels of granularity and application, ranging from personal to societal. With this shift, we have since moved from comprehending single media and its state toward comprehending media in terms of its use context. Multimedia is thus no longer confined to documentation and preservation, entertainment or personal media collections; rather, it has become an integral part of the tools and systems that are providing solutions to today's societal challenges-including challenges related to health care and personal health , aging, education, societal participation, sustainable energy, and intelligent transportation. Multimedia has thus evolved into a core enabler for future interactive and cooperative applications at the heart of society. In this workshop we explore the relevance and contribution of multimedia to health care and personal media.

The organisation of personal data is receiving increasing research attention due to the challenges we face in gathering, enriching, searching, and visualising such data. Given the increasing ease with which personal data being gathered by individuals, the concept of a lifelog digital library of rich multimedia and sensory content for every individual is fast becoming a reality. The LTA~2016 workshop aims to bring together academics and practitioners to discuss approaches to lifelog data analytics and applications; and to debate the opportunities and challenges for researchers in this new and challenging area.

Health care has a long history of adopting technology to save lives and improve the quality of living. Visual information is frequently applied for disease detection and assessment, and the established fields of computer vision and medical imaging provide essential tools. It is, however, a misconception that disease detection and assessment are provided exclusively by these fields and that they provide the solution for all challenges. Integration and analysis of data from several sources, real-time processing, and the assessment of usefulness for end-users are core competences of the multimedia community and are required for the successful improvement of health care systems. We have conducted initial investigations into two use cases surrounding diseases of the gastrointestinal (GI) tract, where the detection of abnormalities provides the largest chance of successful treatment if the initial observation of disease indicators occurs before the patient notices any symptoms. Although such detection is typically provided visually by applying an endoscope, we are facing a multitude of new multimedia challenges that differ between use cases. In real-time assistance for colonoscopy, we combine sensor information about camera position and direction to aid in detecting, investigate means for providing support to doctors in unobtrusive ways, and assist in reporting. In the area of large-scale capsular endoscopy, we investigate questions of scalability, performance and energy efficiency for the recording phase, and combine video summarization and retrieval questions for analysis.

Test collections have a long history of supporting repeatable and comparable evaluation in Information Retrieval (IR). However, thus far, no shared test collection exists for IR systems that are designed to index and retrieve multimodal lifelog data. In this paper we introduce the first test collection for personal lifelog data, which has been employed for the NTCIR12-Lifelog task. In this paper, the requirements for the test collection are motivated, the process of creating the test collection is described, along with an overview of the test collection. Finally suggestions are given for possible applications of the test collection.

In this paper, we present a novel application for elucidating all kind of videos that require expert knowledge, e.g., sport videos, medical videos etc., focusing on endoscopic surgery and video capsule endoscopy. In the medical domain, the knowledge of experts for tagging and interpretation of videos is of high value. As a result of the stressful working environment of medical doctors, they often simply do not have time for extensive annotations. We therefore present a semi-supervised method to gather the annotations in a very easy and time saving way for the experts and we show how this information can be used later on.

Wearable devices such as Google Glass are receiving increasing attention and look set to become part of our technical landscape over the next few years. At the same time, lifelogging is a topic that is growing in popularity with a host of new devices on the market that visually capture life experience in an automated manner. In this paper, we describe a visual lifelogging solution for Google Glass that is designed to capture life experience in rich visual detail, yet maintain the privacy of unknown bystanders.

With the increasing availability of passive, wearable sensor devices, digital lifelogs can now be captured for individuals. Lifelogs contain a digital trace of a person's life, and are characterised by large quantities of rich contextual data. In this paper, we propose a content-based recommender system to leverage such lifelogs to suggest activities to users. We model lifelogs as timelines of chronological sequences of activity objects, and describe a recommendation framework in which a two-level distance metric is proposed to measure the similarity between current and past timelines. An initial evaluation of our activity recommender performed using a real-world lifelog dataset demonstrates the utility of our approach.

Dementia is one of the most common diseases in the elderly people. Experience shows that Microsoft's SenseCam can be an effective memory-aid device, as it helps users to improve recollecting an experience by creating visual lifelogs. Given the vast amount of images that are maintained in a visual lifelog, it is a significant challenge to deconstruct a sizeable collection of images into meaningful events for users. In this paper, random matrix theory (RMT) is applied to a cross-correlation matrix C, constructed using SenseCam lifelog data streams to identify such events. The analysis reveals a number of eigenvalues that deviate from the spectrum suggested by RMT. The components of the deviating eigenvectors are found to correspond to “distinct significant events” in the visual lifelogs. Finally, the cross-correlation matrix C is cleaned by separating the noisy part from the non-noisy part. Overall, the RMT technique is shown to be useful to detect major events in SenseCam images.

Smart-phones are becoming our constant companions, they are with us all of the time, being used for calling, web surfing, apps, music listening, TV viewing, social networking, buying, gaming, and a myriad of other uses. Smart-phones are a technology that knows us much better than most of us could imagine. Based on our usage and the fact that we are never far away from our smart phones, they know where we go, who we interact with, what information we consume, and with a little clever software, they can know what we are doing and even why we are doing it. They are beginning to know us better than we know ourselves. In this work we present SenseSeer a generic mobile-cloud-based mobile Lifelogging framework. This framework supports customisable analytic services for sensing the person, understanding the semantics of life activities and the easy deployment of analytic tools and novel interfaces. At present, SenseSeer supports services in many domains, such as personal health monitoring, location tracking, lifestyle analysis and tourism focused applications. This work demonstrate the design principles of SenseSeer and three of its services: My Health, My Location and My Social Activity.

People have a natural tendency to collect things about themselves, their experiences and their shared experiences with people important to them, especially family. Similar to traditional objects such as photographs, lifelogs have been shown to support reminiscence. A lifelog is a digital archive of a person's experiences and activities and lifelog devices such as wearable cameras can automatically and continuously record events throughout a whole day. We were interested in investigating what would motivate people to lifelog. Due to the importance of shared family reminiscence between family members we focused our study on comparing shared or personal motivations with ten older and ten younger family members. We found from our results that both older and younger adults were more likely to lifelog for the purposes of information sharing and that reviewing lifelog images supported family reminiscence, reflection and story-telling. Based on these findings, recommendations are made for the design of a novel intergenerational family lifelog system.

As the increasing affordability for capturing and storing video and the proliferation of Web 2.0 applications, video content is no longer necessarily created and supplied by a limited number of professional producers; any amateur can produce and publish his/her video quickly. Therefore, the amount of both professional-produced as well as amateur-produced video on the web is ever increasing. In this work, we propose a question; whether we can automatically classify an Internet video clip as being either professional-produced or amateur-produced? Hence, we investigate features and classification methods to answer this question. Based on the differences in the production processes of these two video categories, four features including camera motion, structure, audio feature and combined feature are adopted and studied along with with four popular classifiers KNN, SVM GMM and C4.5. Extensive experiments over representative datasets, evaluate these features and classifiers under different settings and compare to existing techniques. Experimental results demonstrate that SVMs with multimodal features from multi-sources are more effective at classifying video type. Finally, for answering the proposed question, results also show that automatically classifying a clip as professional-produced video or amateur-produced video can be achieved with good accuracy.

With the increasing use of mobile devices as personal recording, communication and sensing tools, extracting the semantics of life activities through sensed data (photos, accelerometer, GPS etc.) is gaining widespread public awareness. A person who engages in long-term personal sensing is engaging in a process of lifelogging. Lifelogging typically involves using a range of (wearable) sensors to capture raw data, to segment into discrete activities, to annotate and subsequently to make accessible by search or browsing tools. In this paper, we present an intuitive lifelog activity recording and management system called ZhiWo. By using a supervised machine learning approach, sensed data collected by mobile devices are automatically classified into different types of daily human activities and these activities are interpreted as life activity retrieval units for personal archives.

In this demo paper, we describe our user-generated video search system, compromising of an iPad interface communicating with a remote server. The goal of this system is to provide an easy access to video content lacking textual annotations by clustering key frames. Moreover, the graphical user interface allows users to filter video content based on various semantic concepts.

Human memory is a dynamic system that makes accessible certain memories of events based on a hierarchy of information, arguably driven by personal significance. Not all events are remembered, but those that are tend to be more psychologically relevant. In contrast, lifelogging is the process of automatically recording aspects of one's life in digital form without loss of information. In this article we share our experiences in designing computer-based solutions to assist people review their visual lifelogs and address this contrast. The technical basis for our work is automatically segmenting visual lifelogs into events, allowing event similarity and event importance to be computed, ideas that are motivated by cognitive science considerations of how human memory works and can be assisted. Our work has been based on visual lifelogs gathered by dozens of people, some of them with collections spanning multiple years. In this review article we summarize a series of studies that have led to the development of a browser that is based on human memory systems and discuss the inherent tension in storing large amounts of data but making the most relevant material the most accessible.

Locating content in existing video archives is both a time and bandwidth consuming process since users might have to download and manually watch large portions of superfluous videos. In this paper, we present two novel prototypes using an Internet based video composition and streaming system with a keyword-based search interface that collects, converts, analyses, indexes, and ranks video content. At user requests, the system can automatically sequence out portions of single videos or aggregate content from multiple videos to produce a single, personalized video stream on-the-fly.

Semantic concept detection is a very useful technique for developing powerful retrieval or filtering systems for multimedia data. To date, the methods for concept detection have been converging on generic classification schemes. However, there is often imbalanced dataset or rare class problems in classification algorithms, which deteriorate the performance of many classifiers. In this paper, we adopt three “under-sampling” strategies to handle this imbalanced dataset issue in a SVM classification framework and evaluate their performances on the TRECVid 2007 dataset and additional positive samples from TRECVid 2010 development set. Experimental results show that our well-designed “under-sampling” methods (method SAK) increase the performance of concept detection about 9:6% overall. In cases of extreme imbalance in the collection the proposed methods worsen the performance than a baseline sampling method (method SI), however in the majority of cases, our proposed methods increase the performance of concept detection substantially. We also conclude that method SAK is a promising solution to address the SVM classification with not extremely imbalanced datasets.

In this paper we describe the design considerations for a touchscreen visual lifelog browser. Visual lifelogs are large collections of photographs which represent a person's experiences. Lifelogging devices, such as the wearable camera known as SenseCam, can record thousands of images per day. Utilizing the approach of event segmentation to organize and present these images, we have designed an interface to present lifelog collections for touchscreen interaction, thus increasing accessibility for users.

Lifelogging is the capture and storage of everyday experiences and the act of reviewing lifelog data can significantly support episodic memory, which is particularly vulnerable to the effects of ageing. To design an accessible lifelogging application for older users we firstly need to explore what lifelogging features the application should include. We carried out a questionnaire study to investigate what lifelogging items people from different age groups are currently collecting. Also of interest to us was what items the participants would like to collect and the differences in lifelogging choices between age groups. The results from this questionnaire will contribute to the design of a lifelogging application which focuses on older users' preferences, motivations and abilities.

Much effort has focused in recent years on developing more life-like robots. In this paper we propose a model of memory for robots, based on human digital memories, though our model incorporates an element of forgetting to ensure that the robotic memory appears more human and therefore can address some of the challenges for human-robot interaction.

We conjecture that composition of video events from various sources into personalized video playouts will become an important part of next generation streaming systems. Here, video search is a key component since it enables users to retrieve candidate video events based on their interests. One of the main challenges, however, is to analyze the videos in order to correctly identify the various events used to annotate and index the video data. Key problems with current video analysis solutions include that they 1) are complex and therefore require a lot of processing time resulting in large delays; 2) that they can only identify a limited set of events; and 3) that they are still too inaccurate, both giving false positives and failing to find all events. In our Davvi prototype, we therefore extract metadata for our video search engine by combining existing automatic video analysis tools with currently untapped textual information available in the Internet. This provides an end-user experience where textual query results can be combined dynamically into seamless, highly personalized video playouts using an adaptive torrent-like HTTP streaming solution.

In this work we discuss a new, but highly relevant, topic to the multimedia community; systems to inform individuals of their carbon footprint, which could ultimately effect change in community carbon footprint-related activities. The reduction of carbon emissions is now an important policy driver of many governments, and one of the major areas of focus is in reducing the energy demand from the consumers i.e. all of us individually.

In this paper, we present prototype interactive TV software that incorporates visual content analysis tools and social networking in the home TV. We present the challenges of working with the living room TV environment and outline how we have utilized visual processing and search technologies to address these challenges and create a novel prototype interactive TV system.

Colour permeates our daily lives, yet we rarely take notice of it. In this work we utilise the SenseCam (a visual lifelogging tool), to investigate the predominant colours in one million minutes of human life that a group of 20 individuals encounter throughout their normal daily activities. We also compare the colours that different groups of people are exposed to in their typical days. This information is presented in using a novel colour-wheel visualisation which is a new means of illustrating that people are exposed to bright colours over longer durations of time during summer months, and more dark colours during winter months.

In this paper we describe how content-based analysis techniques can be used to provide much greater functionality to the users of an interactive TV (iTV) device. We describe several content-based multimedia analysis techniques and how some of these can be exploited in the iTV domain, resulting in the provision of a set of powerful functions for iTV users. To validate our ideas, we introduce an iTV application we developed which incorporates some of these techniques into a simple set of user features, in order to demonstrate the usefulness of content-based techniques for iTV. The contribution of this paper is not to provide an in-depth discussion on each of the individual content-based techniques, but rather to show how many of these powerful technologies can be incorporated into an interactive TV system.

In this demo, we present DAVVI, a prototype of the next generation multimedia entertainment platform. It delivers multi-quality video content in a torrent-similar way like known systems from Move Networks, Microsoft and Apple do. However, it also provides a brand new, personalized user experience. Through applied search, personalization and recommendation technologies, end-users can efficiently search and retrieve highlights and combine arbitrary events in a customized manner using drag and drop. The created playlists of video segments are then delivered back to the system to improve future search and recommendation results. Here, we demonstrate this system using a soccer example.

The Fischlar-TRECVid-2004 system was developed for Dublin City University's participation in the 2004 TRECVid video information retrieval benchmarking activity. The system allows search and retrieval of video shots from over 60 hours of content. The shot retrieval engine employed is based on a combination of query text matched against spoken dialogue combined with image-image matching where a still image (sourced externally), or a keyframe (from within the video archive itself), is matched against all keyframes in the video archive. Three separate text retrieval engines are employed for closed caption text, automatic speech recognition and video OCR. Visual shot matching is primarily based on MPEG-7 low-level descriptors. The system supports relevance feedback at the shot level enabling augmentation and refinement using relevant shots located by the user. Two variants of the system were developed, one that supports both text- and image-based searching and one that supports image only search. A user evaluation experiment compared the use of the two systems. Results show that while the system combining text- and image-based searching achieves greater retrieval effectiveness, users make more varied and extensive queries with the image only based searching version.

In this paper we present a technique for supporting mobile access to digital video archives without requiring explicit user queries. The idea is to infer the interests and needs of users from their WWW browsing history and represent those needs as persistent queries to the archive. An experiment, which we present here, suggests that this technique is effective for recommending video content to users on mobile devices. We also describe how to apply these findings to a mobile interface for a digital video archive.

Handheld computing devices are becoming highly connected devices with high capacity storage. This has resulted in their being able to support storage of, and access to, personal photo archives. However the only means for mobile device users to browse such archives is typically a simple one-by-one scroll through image thumbnails in the order that they were taken, or by manually organising them based on folders. In this paper we describe a system for context-based browsing of personal digital photo archives. Photos are labeled with the GPS location and time they are taken and this is used to derive other context-based metadata such as weather conditions and daylight conditions. We present our prototype system for mobile digital photo retrieval, and an experimental evaluation illustrating the utility of location information for effective personal photo retrieval.

In recent years digital cameras have seen an enormous rise in popularity, leading to a huge increase in the quantity of digital photos being taken. This brings with it the challenge of organising these large collections. We preset work which organises personal digital photo collections based on date/time and GPS location, which we believe will become a key organisational methodology over the next few years as consumer digital cameras evolve to incorporate GPS and as cameras in mobile phones spread further. The accompanying video illustrates the results of our research into digital photo management tools which contains a series of screen and user interactions highlighting how a user utilises the tools we are developing to manage a personal archive of digital photos.

The I-SPY meta-search engine uses a technique called collaborative Web search to leverage the past search behaviour (queries and selections) of a community of users in order to promote search results that are relevant to the community. In this paper we describe recent studies to clarify the benefits of this approach in situations when the behaviour of users cannot be relied upon in terms of their ability to consistently select relevant results during search sessions.

In this poster we describe alternative inverted index structures that reduce the time required to process queries, produce a higher query throughput and still return high quality results to the end user. We give results based upon the TREC Terabyte dataset showing improvements that these indices give in terms of effectiveness and efficiency.

In this paper we give an outline of the Fischlar system developed to enable participation in the interactive search task within TRECVID 2003. TRECVID is an annual benchmarking exercise which measures the effectiveness of various video information retrieval tasks, including interactive retrieval. The accompanying video provides a usage scenario for our TRECVID2003 system which highlights how a user uses the system in order to perform retrieval of video shots.