The 2026 Grand Challenge on Multimedia Verification at ICMR 2026 invited participants to verify real-world multimedia content across various dimensions, such as verified evidence (source, location, time, entities, motivation) and forensic analysis. Four teams submitted complete solutions, evaluated by professional fact-checkers on a 110-point rubric across ten real-world cases. This paper presents an overview of the challenge, summarizes the participating methods, and reports the official scoreboard together with per-criterion analysis. We observe a clear trend toward multi-agent, tool-augmented LLM pipelines, but also recurring weaknesses in human oversight, reasoning grounding, and resistance to hallucination. As a result, no submission was declared a winner this year, and we outline concrete directions for the next edition.

With the rapid growth of sensors, communication technologies, and social networks, it is now easy to collect large amounts of data from people and their surrounding environments, such as wearable devices, lifelog cameras, and ambient sensors. These data provide both personal and external views of human activities, but most existing work still focuses on analyzing each type of data separately. As a result, there is a clear gap in understanding how to combine and use cross-data across different sources. This workshop provides a platform for researchers from academia and industry to explore cross-data analysis and retrieval, with a focus on practical challenges such as integrating different data types, handling distributed data, and ensuring data security, aiming to support the development of a smart and sustainable society.

Sharing multimodal datasets while protecting privacy remains a challenge in research on social interaction and human behavior. Traditional anonymization often removes useful visual cues, while many generative methods treat each frame independently, leading to inconsistent identities across time and camera views. In this demo, we present an anonymization framework that maintains a stable synthetic identity for each individual throughout a video. By linking observations of people over time and reusing the same generated appearance when they reappear, the system preserves continuity across occlusions and viewpoint changes. This helps retain important contextual signals such as pose, movement, and interactions, while still protecting personal identity. Code repository and video examples available at: https://github.com/Leidiland/TrackAnon.

The ACM Lifelog Search Challenge (LSC) is an annual comparative benchmarking exercise that brings together researchers in the field of multimedia retrieval to evaluate interactive search systems using a large-scale multimodal lifelog dataset. This paper presents an overview of the ninth edition of the challenge (LSC’26), held as a workshop during the ACM International Conference on Multimedia Retrieval (ICMR ’26) in Amsterdam. To broaden its scope, the workshop now features three submission tracks: the traditional Challenge Track for real-time search performance, a new General Lifelog Research Track for theoretical and architectural advancements, and an additional Open Source Track aimed at enhancing reproducibility and reducing barriers to entry for new participants.

The increasing sophistication of cheapfakes and deepfakes poses new challenges for maintaining information integrity in the digital age. While recent advances in deep learning classifiers and chatbot-assisted verification have greatly improved detection capabilities, there remains an opportunity to further strengthen these systems by incorporating contextual understanding from original source material. In this paper, we propose a multimedia verification framework that complements existing approaches by retrieving and leveraging contextual evidence associated with the target media to support more informed assessments of authenticity. For cheapfake detection, we analyze the relationship between image-caption pairs to determine if the caption is out-of-context (OOC). Using the COSMOS dataset, our model achieves a state-of-the-art accuracy of 93.6%. For deepfake detection, we segment videos by speaker using diarization, extract key frames, and apply the GenConViT model for face analysis. We further retrieve related news articles and contextual metadata (speaker identity, topic, time) to support chatbot reasoning. On the DF-W YouTube dataset, our system achieves an F1 score of 89%, surpassing existing benchmarks. Our results demonstrate the importance of integrating contextual retrieval and reasoning into multimedia verification pipelines, especially when verifying AI-generated or manipulated content.

For the eighth time since 2018, the ACM Lifelog Search Challenge (LSC) was run to compare interactive lifelog search systems in a live metrics-based challenge. The goal of the LSC workshop is to comparatively evaluate the capabilities of systems accessing a large multimodal lifelog. LSC'25 attracted eleven participating teams, each of which had developed an innovative interactive lifelog retrieval system. The benchmark was organised in a hybrid manner (due to political issues in 2025) at the LSC workshop at ACM ICMR'25 in Chicago, USA. This short paper summarises the LSC workshop setting and presents the participating lifelog search systems.

The sixth edition of the Intelligent Cross-Data Analysis and Retrieval (ICDAR) workshop continues to serve as a forum for researchers and practitioners addressing the integration, analysis, and retrieval of heterogeneous data sources. While individual modalities such as wearable sensors, lifelogging cameras, and social media have been well studied, analyzing cross-data that incorporates multiple perspectives remains a crucial yet challenging task for advancing human-centered applications. In 2025, the workshop received 19 submissions, of which 7 were accepted following a careful peer-review process, resulting in an acceptance rate of 37%. The accepted papers covered a wide range of topics, including zero-shot composed image retrieval, vision-language scene understanding, adaptive modality fusion, lightweight fine-tuning with truncated SVD, and real-world federated split learning on mobile devices. By fostering interdisciplinary collaboration across domains such as well-being, disaster mitigation, mobility, food computing, and smart cities, the workshop continues to highlight emerging challenges and solutions for building intelligent, sustainable, and human-centric systems driven by cross-modal and multimodal data analytics.

The 2025 Grand Challenge on Multimedia Verification addresses the challenges of verifying the authenticity and context of online multimedia content. Participants analyzed real-world cases of images and videos, assessed their sources, detected any potential manipulations, and submitted detailed verification reports. The main competition was structured into three stages: Training, Validation, and Real-World Verification, which included live test cases and welcomed solutions ranging from manual methods and OSINT practices to automated tools and novel AI techniques. A total of 32 teams from 11 countries registered. The submitted solutions presented diverse pipelines that integrated forensic analysis, multimodal reasoning, and large language models (LLMs). The results demonstrate significant progress toward semi-automated verification workflows, while also exposing challenges in scalability, consistency, and reliability. In this paper, we present the challenge design, datasets, evaluation criteria, and participant outcomes, providing insights to guide future research and practice in multimedia verification.

The Second ACM Workshop on AI-Powered Question & Answering Systems for Multimedia (AIQAM'25) was held on 27 October 2025 in Dublin, Ireland, co-located with ACM Multimedia 2025. The workshop's main objective is to create a collaborative and inclusive space for researchers at the intersection of Artificial Inteligence (AI), large language models (LLMs), multimodal information retrieval, and question answering systems. Building on the success of its first edition (AIQAM'24) at ICMR 2024, AIQAM'25 provided a forum for presenting novel methods, applications, and surveys, which address the challenges of integrating text, image, audio, and video data into QA systems. The programme featured contributions ranging from methodological advances in reasoning and evaluation frameworks, domain-specific applications in education and sustainability, to a survey work reviewing the state of multimedia retrieval-augmented QA. This summary paper outlines the objectives and scope of the workshop, describes its format, highlights the keynote and accepted contributions, and acknowledges the efforts of the organising committee.

The pervasive spread of multimedia misinformation and disinformation presents a significant challenge to information integrity, demanding robust and efficient verification methodologies. Accurately assessing the authenticity and context of complex multimedia content across diverse platforms requires advanced analytical capabilities. This work introduces Ægis, a novel AI-enhanced solution developed as the authors' submission to the ACMMM'25 - Grand Challenge on Multimedia Verification, aimed at improving the efficiency of multimedia verification. The proposed solution integrates both state-of-the-art and traditional verification tools to comprehensively address various aspects of verification tasks, including event summarization, forensic analysis, and evidence validation. In addition to selectively applied verification modules, large language models (LLMs) are leveraged extensively to fuse findings, perform reasoning, and generate structured verification reports-significantly streamlining the verification process. Demonstrated on the competition multimedia dataset, Ægis accurately validates content integrity, extracts geospatial and temporal information, and identifies content origin across platforms, offering reliable and ethical support for real-world fact-checking.

Egocentric video has seen increased interest in recent years, as it is used in a range of areas. However, most existing datasets are limited to a single perspective. In this paper, we present the CASTLE 2024 dataset, a multimodal collection containing ego- and exo-centric (i.e., first- and third-person perspective) video and audio from 15 time-aligned sources, as well as other sensor streams and auxiliary data. The dataset was recorded by volunteer participants over four days in a common location and includes the point of view of 10 participants, with an additional 5 fixed cameras providing an exocentric perspective. The entire dataset contains over 600 hours of UHD video recorded at 50 frames per second. In contrast to other datasets, CASTLE 2024 does not contain any partial censoring, such as blurred faces or distorted audio. The dataset is available via https://castle-dataset.github.io/.

The inaugural edition of the CASTLE grand challenge was held at ACM Multimedia 2025. The focus of the CASTLE challenge is to advance the state-of-the-art in analysis and understanding of multimodal data, especially centered around multistream ego- and exo-centric video. In this first instance of the challenge, participants had to solve three types of tasks - event instance search, object instance search, and question answering - in a fully automatic or interactive setting.

Abstract This paper investigates the use of multimedia verification, in particular, computational tools and Open‐source Intelligence (OSINT) methods, for verifying online multimedia content in the context of the ongoing wars in Ukraine and Gaza. Our study examines the workflows and tools used by several fact‐checkers and journalists working at Faktisk, a Norwegian fact‐checking organization. Our study showcases the effectiveness of diverse resources, including AI tools, geolocation tools, internet archives, and social media monitoring platforms, in enabling journalists and fact‐checkers to efficiently process and corroborate evidence, ensuring the dissemination of accurate information. This research provides an in‐depth analysis of the role of computational tools and OSINT methods for multimedia verification. It also underscores the potentials of currently available technology, and highlights its limitations while providing guidance for future development of digital multimedia verification tools and frameworks.

Information disorder is one of the most typical challenges in the current era of science and technology. The amount of information on the internet is increasing, but its correctness and authenticity are not always guaranteed, leading to false information, fake news, etc. The mentioned problem negatively affects users' reception and use of information. Unlike deepfake, cheapfake is created using simple techniques and does not rely on AI to produce fake multimedia. Cheapfake is becoming increasingly popular due to its ease of creation. Thus, there is a growing need to develop techniques that can detect cheapfake content. Following previous events, the Grand Challenge on Detecting Cheapfakes at ACM ICMR 2024 continues to seek contributions from researchers on cheapfake detection with the goal of improving effectiveness and creativity in approach, and understanding the limitations of the current dataset. This challenge has accepted 6 new proposed methods from participants with the highest private test accuracies achieved at 72.2% for Task 1 and 54.84% for Task 2. The highest public test accuracies for the two tasks are 95.6% and 93% respectively. These new methods focus on incorporating new AI models such as Stable Diffusion, LLM. These new findings represent the latest advancements in cheapfake detection research and introduce new potential approaches for future research.

The explosive growth of social networks this decade has significantly impacted society, with digital technologies enabling anyone to create and disseminate fake news using simple editing tools. Media professionals and fact-checkers are now tasked with the complex challenge of verifying visual content, as both multimedia and associated text can be fabricated. Fact-checkers must thoroughly search for related information, such as the time and place of an image's capture, to uncover inconsistencies for content verification. This process demands expertise in journalism and image analysis and involves navigating vast information spaces with current computational and search tools. While AI can tackle specific tasks, it cannot yet replicate human reasoning and judgment. Our study aims to develop a comprehensive pipeline that employs computer science techniques to assist fact-checkers in working more efficiently and effectively, integrating advanced search tools, deep learning models, and additional data like weather conditions to streamline the verification process.

Our workshop aims to provide a platform for both academic and industrial professionals engaged in the analysis and retrieval of cross-data from diverse perspectives, with a particular emphasis on wearable and ambient sensors, lifelog cameras, social networks, and surrounding sensors.Despite numerous studies exploring individual viewpoints, there remains a significant gap in the analysis and retrieval of cross-data to maximize benefits for humanity.Additionally, challenges such as data security and distributed learning for cross-modal model training and inference arise when dealing with large and distributed datasets.We invite researchers to contribute to this initiative, with the overarching goal of fostering the development of a smart and sustainable society through the efficient utilization of intelligent cross-data analysis and retrieval techniques.

For the seventh time since 2018, the Lifelog Search Challenge (LSC) benchmarked interactive lifelog search systems in a live challenge. The LSC goal is to comparatively evaluate system capabilities to access large multimodal lifelogs comprising hundreds of thousands of records. LSC'24 attracted an unprecedented record number of twenty-one participating teams, where each team proposes innovative ideas implemented to new or already established interactive lifelog retrieval systems. The benchmark was organised in front of a live audience at the LSC workshop at ACM ICMR'24 in Phuket, Thailand. This short paper summarises the LSC workshop setting and presents the participating lifelog search systems.

The recent advancements in Generative Adversarial Networks (GANs) and the emergence of Diffusion models have significantly streamlined the production of highly realistic and widely accessible synthetic content. As a result, there is a pressing need for effective general purpose detection mechanisms to mitigate the potential risks posed by deepfakes. In this paper, we explore the effectiveness of pre-trained vision-language models (VLMs) when paired with recent adaptation methods for universal deepfake detection. Following previous studies in this domain, we employ only a single dataset (ProGAN) in order to adapt CLIP for deepfake detection. However, in contrast to prior research, which rely solely on the visual part of CLIP while ignoring its textual component, our analysis reveals that retaining the text part is crucial. Consequently, the simple and lightweight Prompt Tuning based adaptation strategy that we employ outperforms the previous SOTA approach by 5.01% mAP and 6.61% accuracy while utilizing less than one third of the training data (200k images as compared to 720k). To assess the real-world applicability of our proposed models, we conduct a comprehensive evaluation across various scenarios. This involves rigorous testing on images sourced from 21 distinct datasets, including those generated by GANs-based, diffusion-based and commercial tools. Code and pre-trained models: https://github.com/sohailahmedkhan/CLIPping-the-Deception

The rise of social media enables access to valuable information but also fuels the spread of fake news and misinformation. Cheapfake is a type of misinformation created through simple techniques, often involving the use of unaltered images with misleading captions. To distinguish between Out-of-Context (OOC) and Not-Out-of-Context (NOOC) image-caption pairs, prior research has used text-to-image generative models to generate images from captions and then extract correlations between the generated images and the original images. Despite being unable to identify contradictions in the caption pairs, the aforementioned work has demonstrated promising potential for using generative models in cheapfake detection. In this paper, we introduce a novel framework that leverages a generative model to join the contents of an original image and a caption into a new image, referred to as a context-synthetic image. To convert the quantitative difference between an original image and a context-synthetic image, termed as the contextual deviation value, into OOC and NOOC labels, we train a classification model on a newly curated dataset of 7144 context-synthetic images generated using the Stable Diffusion model. We believe that our work offers valuable insights into the use of generative models for cheapfake detection, paving the way for future advancements in this field.

The advent of large language models (LLMs) has energised research in Question-Answering (QA) tasks, enabling responses across varied domains like economics and mathematics. Despite their capabilities, LLMs often lack explainability due to their complex parameter embeddings. Additionally, integrating multimedia data into QA systems introduces challenges in processing and interpreting diverse data types such as text, images, audio, and video. This necessitates sophisticated algorithms for accurate information retrieval across media while ensuring the reliability of the data and responses remains a significant challenge. The AIQAM workshop aims to bring together researchers and practitioners to address these challenges and enhance QA systems with multimedia data. The focus is on promoting innovations that improve the accuracy, explainability, and trustworthiness of QA systems, contributing to the development of the field.

In our modern world, the manipulation of digital content, especially the usage of out-of-context images known as Cheapfakes, has become a significant challenge for the endorsement of integrity and trustworthiness for information on the Internet. This highlights an urgent need for effective detection of the misuse of images and their accompanied captions. Motivated by this issue, our research presents an innovative approach to the solution of this problem. Participating in the ACM ICMR 2024 Grand Challenge on Detecting Cheapfakes, we leverage a unified end-to-end network, integrated with generative synthetic data for training. After complete evaluation, our proposed network demonstrated a remarkable accuracy of 95.60% on the public test dataset for Task 1, as well as efficiency in Task 2. This paper highlights the notable potential of employing an end-to-end network for Cheapfakes detection, which composes a significant contribution to the advancement of multimedia content integrity. Our source code is publicly available at https://github.com/thanhson28/cheapfakes_detection_icmr2024.git

The growth of misinformation and re-contextualized media in social media and news leads to an increasing need for fact-checking methods. Concurrently, the advancement in generative models makes cheapfakes and deepfakes both easier to make and harder to detect. In this paper, we present a novel approach using generative image models to our advantage for detecting Out-of-Context (OOC) use of images-caption pairs in news. We present two new datasets with a total of 6800 images generated using two different generative models including (1) DALL-E 2, and (2) Stable-Diffusion. We are confident that the method proposed in this paper can further research on generative models in the field of cheapfake detection, and that the resulting datasets can be used to train and evaluate new models aimed at detecting cheapfakes. We run a preliminary qualitative and quantitative analysis to evaluate the performance of each image generation model for this task, and evaluate a handful of methods for computing image similarity.

For the sixth time since 2018, the Lifelog Search Challenge (LSC) was organized as a comparative benchmarking exercise for various interactive lifelog search systems. The goal of this international competition is to test system capabilities to access large multimodal lifelogs. LSC’23 attracted twelve participanting teams, each of whom had developed a competitive interactive lifelog retrieval system. The benchmark was organized in front of live audience at the LSC workshop at ACM ICMR’23. As in previous editions, this introductory paper presents the LSC workshop and introduces the participating lifelog search systems.

Recently, there has been an increased interest in cross-data research problems, such as predicting air quality using life logging images, predicting congestion using weather and tweets data, and predicting sleep quality using daily exercises and meals. Although several research focusing on multimodal data analytics have been performed, few studies have been conducted on cross-data research (e.g., cross-modal data, cross-domain, cross-platform). The article collection “Intelligent Cross-Data Analysis and Retrieval” aims to encourage research in intelligent cross-data analytics and retrieval and contribute to the creation of a sustainable society. Researchers from diverse domains such as well-being, disaster prevention and mitigation, mobility, climate, tourism and healthcare are welcome to contribute to this Research Topic.

Quality journalism has become more important than ever due to the need for quality and trustworthy media outlets that can provide accurate information to the public and help to address and counterbalance the wide and rapid spread of disinformation. At the same time, quality journalism is under pressure due to loss of revenue and competition from alternative information providers. This vision paper discusses how recent advances in Artificial Intelligence (AI), and in Machine Learning (ML) in particular, can be harnessed to support efficient production of high-quality journalism. From a news consumer perspective, the key parameter here concerns the degree of trust that is engendered by quality news production. For this reason, the paper will discuss how AI techniques can be applied to all aspects of news, at all stages of its production cycle, to increase trust.

Deepfake media is becoming widespread nowadays because of the easily available tools and mobile apps which can generate realistic looking deepfake videos/images without requiring any technical knowledge. With further advances in this field of technology in the near future, the quantity and quality of deepfake media is also expected to flourish, while making deepfake media a likely new practical tool to spread mis/disinformation. Because of these concerns, the deepfake media detection tools are becoming a necessity. In this study, we propose a novel hybrid transformer network utilizing early feature fusion strategy for deepfake video detection. Our model employs two different CNN networks, i.e., (1) XceptionNet and (2) EfficientNet-B4 as feature extractors. We train both feature extractors along with the transformer in an end-to-end manner on FaceForensics++, DFDC benchmarks. Our model, while having relatively straightforward architecture, achieves comparable results to other more advanced state-of-the-art approaches when evaluated on FaceForensics++ and DFDC benchmarks. Besides this, we also propose novel face cut-out augmentations, as well as random cut-out augmentations. We show that the proposed augmentations improve the detection performance of our model and reduce overfitting. In addition to that, we show that our model is capable of learning from considerably small amount of data.

We have witnessed the rise of cross-data against multimodal data problems recently. The cross-modal retrieval system uses a textual query to look for images; the air quality index can be predicted using lifelogging images; the congestion can be predicted using weather and tweets data; daily exercises and meals can help to predict the sleeping quality are some examples of this research direction. Although vast investigations focusing on multimodal data analytics have been developed, few cross-data (e.g., cross-modal data, cross-domain, cross-platform) research has been carried on. In order to promote intelligent cross-data analytics and retrieval research and to bring a smart, sustainable society to human beings, the specific article collection on "Intelligent Cross-Data Analysis and Retrieval" is introduced. This Research Topic welcomes those who come from diverse research domains and disciplines such as well-being, disaster prevention and mitigation, mobility, climate change, tourism, healthcare, and food computing

For the fifth time since 2018, the Lifelog Search Challenge (LSC) facilitated a benchmarking exercise to compare interactive search systems designed for multimodal lifelogs. LSC'22 attracted nine participating research groups who developed interactive lifelog retrieval systems enabling fast and effective access to lifelogs. The systems competed in front of a hybrid audience at the LSC workshop at ACM ICMR'22. This paper presents an introduction to the LSC workshop, the new (larger) dataset used in the competition, and introduces the participating lifelog search systems.

Misuse of real photographs with conflicting image captions in news items is one case of out-of-context (OOC) misuse of media. To detect out-of-context given pair of news (i.e., captions) and attached image, people should determine the truthfulness of the statement and evaluate whether the triplet talks to the same event. This paper presents a new method to detect the OOC media challenge introduced in ACMMM'22 Grand Challenge on Detecting Cheapfakes. For $Task_1$ (i.e., detect conflicting image-caption triplets), our approach uses bottom-up attention with visual semantic reasoning to extract global features of the image to perform comprehensive image-text matching, multiple models of natural language processing to extract semantic relation of the caption and utilize boosting to improve the accuracy of the performance. Our method achieved an 85.5% accuracy score on 80% of the testing dataset and has 4.5% higher accuracy than baseline. For Task2 (i.e., determine whether a given (Image, Caption) pair is real or fake), we detect the veracity of captioned image base semantic features and correlation between image/caption and achieve 76% accuracy. Our source code is available at https://github.com/latuanvinh1998/Cheapfakes_detection_acmmm. Docker for submission is available at https://hub.docker.com/repository/docker/latuanvinh1998/acmmmcheapfakes.

The growth of communication has led to misinformation in many different forms. "Cheapfake" is a recently coined term referring to manipulated media generated by non-AI techniques. One of the most prevalent ways to create cheapfakes is by simply altering the context of an image/video using a misleading caption. The "ACMMM 2022 Grand Challenge on Detecting Cheapfakes" has raised the problem of catching the out-of-context misuse to assist fact-checkers, as detecting conflicting image-caption sets helps narrow the search space. To cope with this challenge, we propose a multimodal heuristic method. The proposed method expands the baseline method of the challenge (i.e., COSMOS) with four additional components (i.e., Natural Language Inference, Fabricated Claims Detection, Visual Entailment, and Online Caption Checking) towards overcoming the current weaknesses of the baseline. During runtime, our proposed method has achieved a maximum of 89.1% accuracy on Task 1, which is 7.2% higher than the baseline method, and 73% accuracy on Task 2. The code for our solution is publicly available on Github1 https://github.com/pwnyniche/acmmmcheapfake2022 and the Docker image can be found on DockerHub2 https://hub.docker.com/repository/docker/tqtnk2000/acmmmcheapfakes.

Cross-data analytics and retrieval have gained significant improvement recently. People can now extract more data insights precisely and quickly towards having many excellent applications serving human lives. Since people create multimedia and other types of data that reflect the diverse perspectives of human lives, these data are just pieces of the puzzle of the world's pictures. Hence, it is necessary to assembly all these pieces towards having a better solution for human-centered problems. Hence, the workshop welcomes those who work with multimedia and others and come from diverse research domains and disciplines to work on intelligent cross-data analytics and retrieval to bring a smart, sustainable society to human beings. The research domain can vary from well-being, disaster prevention and mitigation, mobility to food computing, to name a few.

The Lifelog Search Challenge (LSC) is an annual benchmarking challenge for comparing approaches to interactive retrieval from multi-modal lifelogs. LSC'21, the fourth challenge, attracted sixteen participants, each of which had developed interactive retrieval systems for large multimodal lifelogs. These interactive retrieval systems participated in a comparative evaluation in front of an online live-audience at the LSC workshop at ACM ICMR'21. This overview presents the motivation for LSC'21, the lifelog dataset used in the competition, and the participating systems.

Nowadays, recommendation systems play an indispensable role in many fields, including e-commerce, finance, economy, and gaming. There is emerging research on publication venue recommendation systems to support researchers when submitting their scientific work. Several publishers such as IEEE, Springer, and Elsevier have implemented their submission recommendation systems only to help researchers choose appropriate conferences or journals for submission. In this work, we present a demo framework to construct an effective recommendation system for paper submission. With the input data (the title, the abstract, and the list of possible keywords) of a given manuscript, the system recommends the list of top relevant journals or conferences to authors. By using state-of-the-art techniques in natural language understanding, we combine the features extracted with other useful handcrafted features. We utilize deep learning models to build an efficient recommendation engine for the proposed system. Finally, we present the User Interface (UI) and the architecture of our paper submission recommendation system for later usage by researchers.

The Lifelog Search Challenge (LSC) is an annual comparative benchmarking activity for comparing approaches to interactive retrieval from multi-modal lifelogs. LSC'20, the third such challenge, attracts fourteen participants with their interactive lifelog retrieval systems. These systems are comparatively evaluated in front of a live-audience at the LSC workshop at ACM ICMR'20 in Dublin, Ireland. This overview motivates the challenge, presents the dataset and system configuration used in the challenge, and briefly presents the participating teams.

With the rapid growth of users' data in SaaS (Software-as-a-service) platforms using micro-services, it becomes essential to detect duplicated entities for ensuring the integrity and consistency of data in many companies and businesses (primarily multinational corporations). Due to the large volume of databases today, the expected duplicate detection algorithms need to be not only accurate but also practical, which means that it can release the detection results as fast as possible for a given request. Among existing algorithms for the deduplicate detection problem, using Siamese neural networks with the triplet loss has become one of the robust ways to measure the similarity of two entities (texts, paragraphs, or documents) for identifying all possible duplicated items. In this paper, we first propose a practical framework for building a duplicate detection system in a SaaS platform. Second, we present a new active learning schema for training and updating duplicate detection algorithms. In this schema, we not only allow the crowd to provide more annotated data for enhancing the chosen learning model but also use the Siamese neural networks as well as the triplet loss to construct an efficient model for the problem. Finally, we design a user interface of our proposed deduplicate detection system, which can easily apply for empirical applications in different companies.

In this paper, we present PMData: a dataset that combines traditional lifelogging data with sports-activity data. Our dataset enables the development of novel data analysis and machine-learning applications where, for instance, additional sports data is used to predict and analyze everyday developments, like a person's weight and sleep patterns; and applications where traditional lifelog data is used in a sports context to predict athletes' performance. PMData combines input from Fitbit Versa 2 smartwatch wristbands, the PMSys sports logging smartphone application, and Google forms. Logging data has been collected from 16 persons for five months. Our initial experiments show that novel analyses are possible, but there is still room for improvement.

The BioMedia 2019 ACM Multimedia Grand Challenge is the first in a series of competitions focusing on the use of multimedia for different medical use-cases. In this year's challenge, the participants are asked to develop efficient algorithms which automatically detect a variety of findings commonly identified in the gastrointestinal (GI) tract (a part of the human digestive system). The purpose of this task is to develop methods to aid medical doctors performing routine endoscopy inspections of the GI tract. In this paper, we give a detailed description of the four different tasks of this year's challenge, present the datasets used for training and testing, and discuss how each submission is evaluated both qualitatively and quantitatively.

This paper introduces an interactive system that discovers students’ daily moods and classroom emotions to enhance the teaching and learning process using heterogeneous sensors. The system is designed to enable (1) detecting students daily moods and classroom emotions using physiological, physical activities, and event tags data coming from wristband sensors and smart-phones, (2) discovering association/correlation between students’ lifestyle and daily moods, and (3) displaying statistical reports and the distribution of daily moods and classroom emotions of students, both in individual and group modes. A pilot proof-of-concept study was carried out using Empatica E4 wristband sensors and Android smart-phones, and preliminary evaluation and findings showing promising results are reported and discussed.

In this paper, we propose a deep learning based system for food recognition from personal life archive im- ages. The system first identifies the eating moments based on multi-modal information, then tries to focus and enhance the food images available in these moments, and finally, exploits GoogleNet as the core of the learning process to recognise the food category of the images. Preliminary results, experimenting on the food recognition module of the proposed system, show that the proposed system achieves 95.97% classification accuracy on the food images taken from the personal life archive from several lifeloggers, which potentially can be extended and applied in broader scenarios and for different types of food categories.

Nowadays, almost everyone holds some form or other of a personal life archive. Automatically maintaining such an archive is an activity that is becoming increasingly common, however without automatic support the users will quickly be overwhelmed by the volume of data and will miss out on the potential benefits that lifelogs provide. In this paper we give an overview of the current status of lifelog research and propose a concept for exploring these archives. We motivate the need for new methodologies for indexing data, organizing content and supporting information access. Finally we will describe challenges to be addressed and give an overview of initial steps that have to be taken, to address the challenges of organising and searching personal life archives.

Lifelog, Human Activity Recognition, Smartphones, Embedded Sensors, Smart-City, Heterogeneous Sensory Data Analytics.\nThis paper introduces a new idea for sensor data analytics, named PHASOR, that can recognize and stream individual human activities online. The proposed sensor concept can be utilized to solve some emerging problems in smartcity domain such as health care, urban mobility, or security by creating a lifelog of human activities. PHASOR is created from three ‘components’: ID, model, and Sensor. The first component is to identify which sensor is used to monitor which object (e.g., group of users, individual users, type of smart- phone). The second component decides suitable classifiers for human activities recognition. The last one includes two types: (1) physical sensors that utilize embedded sensors in smartphones to recognize human activities, (2) human factors that uses human interaction to personally increase the accuracy of the detection. The advantage of PHASOR is the error signal is inversely proportional to its lifetime, which is well-suited for lifelogging applications. The proposed concept is evaluated and compared to de-facto datasets as well as state-of-the-art of Human Activity Recognition (HAR) using smartphones, confirming that applying PHASOR can improves the accuracy of HAR.

Being able to automatically link social media information and data to remote-sensed data holds large possibilities for society and research. In this paper, we present a system called JORD that is able to autonomously collect social media data about technological and environmental disasters, and link it automatically to remote-sensed data. In addition, we demonstrate that queries in local languages that are relevant to the exact position of natural disasters retrieve more accurate information about a disaster event. To show the capabilities of the system, we present some examples of disaster events detected by the system. To evaluate the quality of the provided information and usefulness of JORD from the potential users point of view we include a crowdsourced user study.

The organisation of personal data is receiving increasing research attention due to the challenges we face in gathering, enriching, searching, and visualising such data. Given the increasing ease with which personal data being gathered by individuals, the concept of a lifelog digital library of rich multimedia and sensory content for every individual is fast becoming a reality. The LTA 2017 workshop aims to bring together academics and practitioners to discuss approaches to lifelog data analytics and applications; and to debate the opportunities and challenges for researchers in this new and challenging area.

Analysis of medical videos for detection of abnormalities and diseases requires both high precision and recall, but also real-time processing for live feedback and scalability for massive screening of entire populations. Existing work on this field does not provide the necessary combination of retrieval accuracy and performance.; [email protected] this paper, a multimedia system is presented where the aim is to tackle automatic analysis of videos from the human gastrointestinal (GI) tract. The system includes the whole pipeline from data collection, processing and analysis, to visualization. The system combines filters using machine learning, image recognition and extraction of global and local image features. Furthermore, it is built in a modular way so that it can easily be extended. At the same time, it is developed for efficient processing in order to provide real-time feedback to the doctors. Our experimental evaluation proves that our system has detection and localisation accuracy at least as good as existing systems for polyp detection, it is capable of detecting a wider range of diseases, it can analyze video in real-time, and it has a low resource consumption for scalability.

Automatic detection of diseases by use of computers is an important, but still unexplored field of research. Such innovations may improve medical practice and refine health care systems all over the world. However, datasets containing medical images are hardly available, making reproducibility and comparison of approaches almost impossible. In this paper, we present KVASIR, a dataset containing images from inside the gastrointestinal (GI) tract. The collection of images are classified into three important anatomical landmarks and three clinically significant findings. In addition, it contains two categories of images related to endoscopic polyp removal. Sorting and annotation of the dataset is performed by medical doctors (experienced endoscopists). In this respect, KVASIR is important for research on both single- and multi-disease computer aided detection. By providing it, we invite and enable multimedia researcher into the medical domain of detection and retrieval.

In this article, we present a novel framework that can produce a visual description of a tourist attraction by choosing the most diverse pictures from community-contributed datasets, which describe different details of the queried location. The main strength of the proposed approach is its flexibility that permits us to filter out non-relevant images and to obtain a reliable set of diverse and relevant images by first clustering similar images according to their textual descriptions and their visual content and then extracting images from different clusters according to a measure of the user’s credibility. Clustering is based on a two-step process, where textual descriptions are used first and the clusters are then refined according to the visual features. The degree of diversification can be further increased by exploiting users’ judgments on the results produced by the proposed algorithm through a novel approach, where users not only provide a relevance feedback but also a diversity feedback. Experimental results performed on the MediaEval 2015 “Retrieving Diverse Social Images” dataset show that the proposed framework can achieve very good performance both in the case of automatic retrieval of diverse images and in the case of the exploitation of the users’ feedback. The effectiveness of the proposed approach has been also confirmed by a small case study involving a number of real users.

In this paper, we propose a novel approach for face spoofing detection using the high-order Local Derivative Pattern from Three Orthogonal Planes (LDP-TOP). The proposed method is not only simple to derive and implement, but also highly efficient, since it takes into account both spatial and temporal information in different directions of subtle face movements. According to experimental results, the proposed approach outperforms state-of-the-art methods on three reference datasets, namely Idiap REPLAY-ATTACK, CASIA-FASD, and MSU MFSD. Moreover, it requires only 25 video frames from each video, i.e., only one second, and thus potentially can be performed in real time even on low-cost devices.

In this paper, we present Heimdallr, a dataset that aims to serve two different purposes. The first purpose is action recognition and pose estimation, which requires a dataset of annotated sequences of athlete skeletons. We employed a crowdsourcing platform where people around the world were asked to annotate frames and obtained more than 3000 fully annotated frames for 42 different sequences with a variety of poses and actions. The second purpose is an improved understanding of crowdworkers, and for this purpose, we collected over 10000 written feedbacks from 592 crowdworkers. This is valuable information for crowdsourcing researchers who explore algorithms for worker quality assessment. In addition to the complete dataset, we also provide the code for the application that has been used to collect the data as an open source software.

In this paper, we present a novel method that can produce a visual description of a landmark by choosing the most diverse pictures that best describe all the details of the queried location from community-contributed datasets. The main idea of this method is to filter out non-relevant images at a first stage and then cluster the images according to textual descriptors first, and then to visual descriptors. The extraction of images from different clusters according to a measure of user's credibility, allows obtaining a reliable set of diverse and relevant images. Experimental results performed on the MediaEval 2014 “Retrieving Diverse Social Images” dataset show that the proposed approach can achieve very good performance outperforming state-of-art techniques.

Digital forensics is a relatively new research area which aims at authenticating digital media by detecting possible digital forgeries. Indeed, the ever increasing availability of multimedia data on the web, coupled with the great advances reached by computer graphical tools, makes the modification of an image and the creation of visually compelling forgeries an easy task for any user. This in turns creates the need of reliable tools to validate the trustworthiness of the represented information. In such a context, we present here RAISE, a large dataset of 8156 high-resolution raw images, depicting various subjects and scenarios, properly annotated and available together with accompanying metadata. Such a wide collection of untouched and diverse data is intended to become a powerful resource for, but not limited to, forensic researchers by providing a common benchmark for a fair comparison, testing and evaluation of existing and next generation forensic algorithms. In this paper we describe how RAISE has been collected and organized, discuss how digital image forensics and many other multimedia research areas may benefit of this new publicly available benchmark dataset and test a very recent forensic technique for JPEG compression detection.

Modern computer graphics technologies brought realism in computer-generated characters, making them achieve truly natural appearance. Besides traditional virtual reality applications such as avatars, games, or cinema, these synthetic characters may be used to generate realistic fakes, which may lead to improper use of the technology. This fact raises the demand for advanced tools able to discriminate real and artificial human faces in digital media. In this paper, we propose a method to distinguish between computer generated and natural faces by modeling and evaluating their dynamic behavior. Because of a 3D-model-based video analysis, the proposed technique allows identifying synthetic characters by detecting their more limited variability over time. Experimental results demonstrate the effectiveness of the proposed approach also on very challenging and realistic video sequences.

Given the recent development of advanced multimedia techniques able to support the creation of realistic computer generated characters, there is the parallel need of automatic tools allowing users to verify the source of the multimedia data they are observing, thus discriminating between artificial and natural information. In this paper, we focus on video representing human beings and we propose a novel method to identify computer generated characters by analysing the evolution of the face model in chronological order. Experimental results show that photorealistic facial animations, which are usually performed following fixed patterns, can be distinguished from natural ones, which follow much more complicated and various geometric distortions.

Median filtering is a well-known non linear denoising filter often used as an harmless post-processing, sometimes also employed to affect the reliability of some forensic techniques. In this work, we present a novel counter-forensic method able to conceal the characteristic traces left by median filtering. By exploiting the knowledge of features used in existing median filtering detectors, we are able to remove the characteristic footprints via suitable random pixel modification, while keeping the quality of the counter-attacked image high. Experimental results show that the proposed method is very effective, computationally efficient and competitive with other state-of-the-art techniques.

Quick reorganizing and draft annotating personal photo albums under event scheme is an emerging trend. In this research, a method has been developed to meet such requirements using the idea of gist and mosaic art so that viewers could understand the meaning of a whole scene without paying much attention in individual details. First, given a photo album, all chronologically ordered images are normalized to a smaller size, and then mosaicked side-by-side to create a signature image representing for that album. Next, by integrating the optimized linear programming with the color descriptor of the signature image, not only the event-type of the album but also all sub-event-types of the sub-sequence photos are decided. More than 19,000 images of five varied event-types have been used to evaluate the proposed method. Experimental results show that the proposed method could detect events towards annotation and re-organization of personal photo albums with high accuracy at a rapid speed.