Verifying the authenticity of media has become an increasingly challenging task. Rapid advances in AI-generated content, spanning modalities like text, images, video, audio have significantly blurred the line between genuine and synthetic information. Nowadays, powerful foundation models can easily be leveraged to create, amplify and disseminate information at scale, enabling disinformation campaigns, defamation, or impersonation. This results in the erosion of trust in online information, which poses a great threat to society. The MAD’26 workshop seeks to address this problem by bringing together researchers and practitioners from diverse disciplines, united by the goal of combating disinformation through AI-driven approaches. Now in its fifth edition, the workshop aims to cultivate a collaborative environment that encourages the exchange of ideas, methodologies, and practical experiences. The workshop focuses on key research directions, including the detection of AI-generated and manipulated content, the analysis of disinformation propagation, and the examination of its broader societal impact.

Fact-Check Retrieval (FCR) plays a crucial role in automated fact-checking by retrieving relevant fact-checked articles for disputed claims. While recent work has explored text-based, multilingual, and multimodal FCR, most efforts remain unimodal or limited to English. To bridge this gap, we introduce M3-Check, the first FCR dataset combining multilingual texts and images from social media posts with fact-check articles from diverse, credible sources. Furthermore, we introduce FACTOR a two-tower Transformer-based architecture that employs cross-tower parameter sharing and modality-wise aligned weight initialization; that outperforms zero-shot baselines, two-tower linear models, and vanilla Transformers, achieving a 17% improvement over the latter. Moreover we conduct modality ablations and compare state-of-the-art encoders, showing that multilingual encoders like multi-E5 can provide an additional 13% in performance without requiring English translations.

2148

EXplainable Artificial Intelligence (XAI) approaches are widely applied for identifying fairness issues in Artificial Intelligence (AI) systems. However, in the context of facial analysis, existing XAI approaches, such as pixel attribution methods, offer explanations for individual images, posing challenges in assessing the overall behavior of a model, which would require labor-intensive manual inspection of a very large number of instances and leaving to the human the task of drawing a general impression of the model behavior from the individual outputs. Addressing this limitation, we introduce FaceX, the first method that provides a comprehensive understanding of face attribute classifiers through summary model explanations. Specifically, FaceX leverages the presence of distinct regions across all facial images to compute a region-level aggregation of model activations, allowing for the visualization of the model's region attribution across 19 predefined regions of interest in facial images, such as hair, ears, or skin. Beyond spatial explanations, FaceX enhances interpretability by visualizing specific image patches with the highest impact on the model's decisions for each facial region within a test benchmark. Through extensive evaluation in various experimental setups, including scenarios with or without intentional biases and mitigation efforts on four benchmarks, namely CelebA, FairFace, CelebAMask-HQ, and Racial Faces in the Wild, FaceX demonstrates high effectiveness in identifying the models' biases.

1339

Integrity 2024 is the fifth edition of the Workshop on Integrity in Social Networks and Media, held in conjunction with the ACM Conference on Web Search and Data Mining (WSDM) since the 2020 edition [1-4]. The goal of the workshop is to bring together academic and industry researchers working on integrity, fairness, trust and safety in social networks to discuss the most pressing risks and cutting-edge technologies to reliably measure and mitigate them. The event consists of invited talks from academic experts and industry leaders as well as peer-reviewed papers and posters through an open call-for-papers.

Abstract Computer vision systems are employed in a variety of high-impact applications. However, making them trustworthy requires methods for the detection of potential biases in their training data, before models learn to harm already disadvantaged groups in downstream applications. Image data are typically represented via extracted features, which can be hand-crafted or pre-trained neural network embeddings. In this work, we introduce a framework for bias discovery given such features that is based on optimal transport theory; it uses the (quadratic) Wasserstein distance to quantify disparity between the feature distributions of two demographic groups (e.g., women vs men). In this context, we show that the Kantorovich potentials of the images, which are a byproduct of computing the Wasserstein distance and act as “transportation prices", can serve as bias scores by indicating which images might exhibit distinct biased characteristics. We thus introduce a visual dataset exploration pipeline that helps auditors identify common characteristics across high- or low-scored images as potential sources of bias. We conduct a case study to identify prospective gender biases and demonstrate theoretically-derived properties with experiments on the CelebA and Biased MNIST datasets.

In this article the statement after Equation 1 had an error in the published version. Please refer the correction as follows: “where ν = T#µ and T# is the push-forward of µ along the function T : X → Y” was incorrectly written as “where T# is the push-forward of µ along the function T : X → Y. Furthermore, Equation 1 itself was incorrectly formulated. Namely, the integral should have been over X and not over X × Y. The original article has been corrected.

With recent advancements in synthetic media manipulation and generation, verifying multimedia content posted online has become increasingly difficult. Additionally, the malicious exploitation of AI technologies by actors to disseminate disinformation on social media, and more generally the Web, at an alarming pace poses significant threats to society and democracy. Therefore, the development of AI-powered tools that facilitate media verification is urgently needed. The MAD ’23 workshop aims to bring together individuals working on the wider topic of detecting disinformation in multimedia to exchange their experiences and discuss innovative ideas, attracting people with varying backgrounds and expertise. The research areas of interest include identifying manipulated and synthetic content in multimedia, as well as examining the dissemination of disinformation and its impact on society. The multimedia aspect is very important since content most often contains a mix of modalities and their joint analysis can boost the performance of verification methods.

Hate speech is a societal problem that has significantly grown through the Internet. New forms of digital content such as image memes have given rise to spread of hate using multimodal means, being far more difficult to analyse and detect compared to the unimodal case. Accurate automatic processing, analysis and understanding of this kind of content will facilitate the endeavor of hindering hate speech proliferation through the digital world. To this end, we propose MemeFier, a deep learning-based architecture for fine-grained classification of Internet image memes, utilizing a dual-stage modality fusion module. The first fusion stage produces feature vectors containing modality alignment information that captures non-trivial connections between the text and image of a meme. The second fusion stage leverages the power of a Transformer encoder to learn inter-modality correlations at the token level and yield an informative representation. Additionally, we consider external knowledge as an additional input, and background image caption supervision as a regularizing component. Extensive experiments on three widely adopted benchmarks, i.e., Facebook Hateful Memes, Memotion7k and MultiOFF, indicate that our approach competes and in some cases surpasses state-of-the-art. Our code is available on GitHub1.

Artificial intelligence can facilitate the management of large amounts of media content and enable media organisations to extract valuable insights from their data. Although AI for media understanding has made rapid progress over the recent years, its deployment in applications and professional sectors poses challenges, especially to organizations with no AI expertise. This motivated the creation of the Media Asset Annotation and Management platform (MAAM) that employs state-of-the-art deep learning models to annotate and facilitate the management of image and video assets. Annotation models provided by MAAM include automatic captioning, object detection, action recognition and moderation models, such as NSFW and disturbing content classifiers. By annotating media assets with these models, MAAM can support easy navigation, filtering and retrieval of media assets. In addition, our platform leverages the power of deep learning to support advanced visual and multi-modal retrieval capabilities. That allows accurately identifying assets that convey a similar idea, or concept even if they are not visually identical, and support a state-of-the-art reverse search facility for images and videos.

Abstract Class imbalance poses a major challenge for machine learning as most supervised learning models might exhibit bias towards the majority class and under-perform in the minority class. Cost-sensitive learning tackles this problem by treating the classes differently, formulated typically via a user-defined fixed misclassification cost matrix provided as input to the learner. Such parameter tuning is a challenging task that requires domain knowledge and moreover, wrong adjustments might lead to overall predictive performance deterioration. In this work, we propose a novel cost-sensitive boosting approach for imbalanced data that dynamically adjusts the misclassification costs over the boosting rounds in response to model’s performance instead of using a fixed misclassification cost matrix. Our method, called AdaCC, is parameter-free as it relies on the cumulative behavior of the boosting model in order to adjust the misclassification costs for the next boosting round and comes with theoretical guarantees regarding the training error. Experiments on 27 real-world datasets from different domains with high class imbalance demonstrate the superiority of our method over 12 state-of-the-art cost-sensitive boosting approaches exhibiting consistent improvements in different measures, for instance, in the range of [0.3–28.56%] for AUC, [3.4–21.4%] for balanced accuracy, [4.8–45%] for gmean and [7.4–85.5%] for recall.

The verification of multimedia content posted online becomes increasingly challenging due to recent advancements in synthetic media manipulation and generation. Moreover, malicious actors can easily exploit AI technologies to spread disinformation across social media at a rapid pace, which poses very high risks for society and democracy. There is, therefore, an urgent need for AI-powered tools that facilitate the media verification process. The objective of the MAD '22 workshop is to bring together those who work on the broader topic of disinformation detection in multimedia in order to share their experiences and discuss their novel ideas, reaching out to people with different backgrounds and expertise. The research domains of interest vary from the detection of manipulated and synthetic content in multimedia to the analysis of the spread of disinformation and its impact on society. The MAD '22 workshop proceedings are available at: https://dl.acm.org/citation.cfm?id=3512732.

In this paper, we address the problem of global-scale image geolocation, proposing a mixed classification-retrieval scheme. Unlike other methods that strictly tackle the problem as a classification or retrieval task, we combine the two practices in a unified solution leveraging the advantages of each approach with two different modules. The first leverages the EfficientNet architecture to assign images to a specific geographic cell in a robust way. The second introduces a new residual architecture that is trained with contrastive learning to map input images to an embedding space that minimizes the pairwise geodesic distance of same-location images. For the final location estimation, the two modules are combined with a search-within-cell scheme, where the locations of most similar images from the predicted geographic cell are aggregated based on a spatial clustering scheme. Our approach demonstrates very competitive performance on four public datasets, achieving new state-of-the-art performance in fine granularity scales, i.e., 15.0% at 1km range on Im2GPS3k.

Personalized PageRank (PPR) is a popular scheme for scoring the relevance of network nodes to a set of seed ones through a random walk with restart process. Calculating the scores of all network nodes often involves the power method, which iterates the PPR formula until convergence to an empirically selected numerical tolerance. However, finding a tolerance that is not so lax as to impact pairwise node comparisons but not so strict as to require a high number of iterations to converge requires time-consuming empirical investigation. In this work we aim to avoid this investigation by stopping power method iterations when node score order is robust against subsequent changes. To do this, we analyse the expected fraction of random walks considered at a given iteration and identify a potential stopping point that depends on a (fixed) confidence level of future iterations preserving node order. Experiments on four real-world networks show that a confidence level of 98% runs in a fraction of the time and yields more than 0.999 Spearman correlation with the node order of 10-20numerical tolerance. Furthermore, that stopping point is comparable to empirically selecting a numerical tolerance that yields robust node order.

Local community detection is an emerging topic in network analysis that aims to detect well-connected communities encompassing sets of priorly known seed nodes. In this work, we explore the similar problem of ranking network nodes based on their relevance to the communities characterized by seed nodes. However, seed nodes may not be central enough or sufficiently many to produce high quality ranks. To solve this problem, we introduce a methodology we call seed oversampling, which first runs a node ranking algorithm to discover more nodes that belong to the community and then reruns the same ranking algorithm for the new seed nodes. We formally discuss why this process improves the quality of calculated community ranks if the original set of seed nodes is small and introduce a boosting scheme that iteratively repeats seed oversampling to further improve rank quality when certain ranking algorithm properties are met. Finally, we demonstrate the effectiveness of our methods in improving community relevance ranks given only a few random seed nodes of real-world network communities. In our experiments, boosted and simple seed oversampling yielded better rank quality than the previous neighborhood inflation heuristic, which adds the neighborhoods of original seed nodes to seeds.

Machine learning bias and fairness have recently emerged as key issues due to the pervasive deployment of data-driven decision making in a variety of sectors and services. It has often been argued that unfair classifications can be attributed to bias in training data, but previous attempts to 'repair' training data have led to limited success. To circumvent shortcomings prevalent in data repairing approaches, such as those that weight training samples of the sensitive group (e.g. gender, race, financial status) based on their misclassification error, we present a process that iteratively adapts training sample weights with a theoretically grounded model. This model addresses different kinds of bias to better achieve fairness objectives, such as trade-offs between accuracy and disparate impact elimination or disparate mistreatment elimination. We show that, compared to previous fairness-aware approaches, our methodology achieves better or similar trades-offs between accuracy and unfairness mitigation on real-world and synthetic datasets.

Location extraction, also called “toponym extraction,” is a field covering geoparsing, extracting spatial representations from location mentions in text, and geotagging, assigning spatial coordinates to content items. This article evaluates five “best-of-class” location extraction algorithms. We develop a geoparsing algorithm using an OpenStreetMap database, and a geotagging algorithm using a language model constructed from social media tags and multiple gazetteers. Third-party work evaluated includes a DBpedia-based entity recognition and disambiguation approach, a named entity recognition and Geonames gazetteer approach, and a Google Geocoder API approach. We perform two quantitative benchmark evaluations, one geoparsing tweets and one geotagging Flickr posts, to compare all approaches. We also perform a qualitative evaluation recalling top N location mentions from tweets during major news events. The OpenStreetMap approach was best (F1 0.90+) for geoparsing English, and the language model approach was best (F1 0.66) for Turkish. The language model was best (F1@1km 0.49) for the geotagging evaluation. The map database was best (R@20 0.60+) in the qualitative evaluation. We report on strengths, weaknesses, and a detailed failure analysis for the approaches and suggest concrete areas for further research.

The publication and spread of misleading content is a problem of increasing magnitude, complexity and consequences in a world that is increasingly relying on user-generated content for news sourcing. To this end, multimedia analysis techniques could serve as an assisting tool to spot and debunk misleading content on the Web. In this paper, we tackle the problem of misleading multimedia content detection on Twitter in real time. We propose a number of new features and a new semi-supervised learning event adaptation approach that helps generalize the detection capabilities of trained models to unseen content, even when the event of interest is of different nature compared to that used for training. Combined with bagging, the proposed approach manages to outperform previous systems by a significant margin in terms of accuracy. Moreover, in order to communicate the verification process to end users, we develop a web-based application for visualizing the results.

This paper describes a multimodal graph-based approach to address the problem of event detection and summarization in large scale image collections. A first version of our system was presented in the Yahoo-Flickr Event Summarization Challenge of ACM Multimedia 2015 [6]. The objective of the approach is to automatically detect events within millions of photos and summarizing them efficiently for user consumption. The presented approach uses a moving time window over the collection of multimedia items to build a same-event image graph and applies graph clustering to detect events. In addition, it makes use of a graph-based diversity-oriented ranking algorithm to summarize instances of the detected events. A demo of the system is online at: http://mklab.iti.gr/acmmm2015-gc/.

Information sharing in online social networks is a daily practice for billions of users. The sharing process facilitates the maintenance of users' social ties but also entails privacy disclosure in relation to other users and third parties. Depending on the intentions of the latter, this disclosure can become a risk. It is thus important to propose tools that empower the users in their relations to social networks and third parties connected to them. As part of USEMP, a coordinated research effort aimed at user empowerment, we introduce a system that performs privacy-aware classification of images. We show that generic privacy models perform badly with real-life datasets in which images are contributed by individuals because they ignore the subjective nature of privacy. Motivated by this, we develop personalized privacy classification models that, utilizing small amounts of user feedback, provide significantly better performance than generic models. The proposed semi-personalized models lead to performance improvements for the best generic model ranging from 4%, when 5 user-specific examples are provided, to 18% with 35 examples. Furthermore, by using a semantic representation space for these models we manage to provide intuitive explanations of their decisions and to gain novel insights with respect to individuals' privacy concerns stemming from image sharing. We hope that the results reported here will motivate other researchers and practitioners to propose new methods of exploiting user feedback and of explaining privacy classifications to users.

research-article Share on Overview of the Special Issue on Trust and Veracity of Information in Social Media Authors: Symeon Papadopoulos Centre for Research and Technology Hellas; Thessaloniki, Greece Centre for Research and Technology Hellas; Thessaloniki, GreeceView Profile , Kalina Bontcheva University of Sheffield, Sheffield, UK University of Sheffield, Sheffield, UKView Profile , Eva Jaho Athens Technology Center, Athens, Greece Athens Technology Center, Athens, GreeceView Profile , Mihai Lupu Vienna University of Technology, Vienna, Austria Vienna University of Technology, Vienna, AustriaView Profile , Carlos Castillo Sapienza University of Rome, Rome, Italy Sapienza University of Rome, Rome, ItalyView Profile Authors Info & Claims ACM Transactions on Information SystemsVolume 34Issue 3May 2016 Article No.: 14pp 1–5https://doi.org/10.1145/2870630Published:11 April 2016Publication History 20citation1,578DownloadsMetricsTotal Citations20Total Downloads1,578Last 12 Months57Last 6 weeks12 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Online Social Networks (OSN) allow their users to organize their friends into groups, also known as social circles. These social circles can be used to better manage who has access to users' posted content and also to control the content posted from other users that they view. Unfortunately, these social circles are generated manually and this can be a laborious process for users with more than a few friends. In this paper, we propose an approach for automatically generating social circles that takes into account both the profile information of the friends to be grouped and the social network connectivity between them, while it allows multiple membership of friends in social circles. The approach is based on an adaptation of the widely used Latent Dirichlet Allocation model and, despite the fact that it does not explicitly model social network connectivity, as other state of the art methods do, it manages to achieve results that are competitive and even better than those obtained from such methods, at a considerably lower computational cost.

Due to the increasing popularity of microblogging platforms, the amount of messages (posts) related to public events, especially posts encompassing multimedia content, is steadily increasing. The inclusion of images can convey much more information about the event, compared to their text, which is typically very short (e.g., tweets). Although such messages can be quite informative regarding different aspects of the event, there is a lot of spam and redundancy making it challenging to extract pertinent insights. In this work, we describe a summarization framework that, given a set of social media messages about an event, aims to select a subset of images derived from them, that, at the same time, maximizes the relevance of the selected images and minimizes their redundancy. To this end, we propose a topic modeling technique to capture the relevance of messages to event topics and a graph-based algorithm to produce a diverse ranking of the selected high-relevance images. A user-centered evaluation on a large Twitter dataset around several real-world events demonstrates that the proposed method considerably outperforms a number of state-of-the-art summarization algorithms in terms of result relevance, while at the same time it is also highly competitive in terms of diversity. Namely, we get an improvement of 25% in terms of precision compared to the second best result, and 7% in terms of diversity.

Results returned by commercial image search engines should include relevant and diversified depictions of queries in order to ensure good coverage of users' information needs. While relevance has drastically improved in recent years, diversity is still an open problem. In this paper we propose a reranking method that could be implemented on top of such engines in order to provide a better balance between relevance and diversity. Our method formulates the reranking problem as an optimization of a utility function that jointly considers relevance and diversity. Our main contribution is the replacement of the unsupervised definition of relevance that is commonly used in this formulation with a supervised classification model that strives to capture a query and application-specific notion of relevance. This model provides more accurate relevance scores that lead to significantly improved diversification performance. Furthermore, we propose a stacking-type ensemble learning approach that allows combining multiple features in a principled way when computing the relevance of an image. An empirical evaluation carried out on the datasets of the MediaEval 2013 and 2014 "Retrieving Diverse Social Images" (RDSI) benchmarks confirms the superior performance of the proposed method compared to other participating systems as well as a state-of-the-art, unsupervised reranking method.

We present a technical demonstration of a novel smartphone application that enables efficient browsing and tagging of landmark and event photos. The application employs a hierarchical mode of exploration enabling zooming from the level of a city, through the level of an area/neighbourhood, down to the level of a specific spot. This navigation mechanism combined with photo clustering, and a dual map-list viewing mechanism, enables efficient browsing of hundreds of thousands of publicly available geotagged photos on a smartphone. Due to the large number of markers on the map, an adaptive clustering strategy is employed to reduce clutter on the screen. In addition to the advanced browsing capabilities, the application enables users to import their personal photos from Flickr and to easily annotate them by propagating to them metadata (location, tags) from the currently viewed object (area, landmark). The application currently supports more than 30 cities worldwide.

A large variety of features can be extracted from raw multimedia items. Moreover, in many contexts, like in the case of multimedia uploaded by users of social media platforms, items may be linked to metadata that can be very useful for a variety of analysis tasks. Nevertheless, such features are typically heterogeneous and are difficult to combine in a unified representation that would be suitable for analysis. In this paper, we discuss the problem of clustering collections of multimedia items with the purpose of detecting social events. In order to achieve this, a novel multimodal clustering algorithm is proposed. The proposed method uses a known clustering in the currently examined domain, in order to supervise the multimodal fusion and clustering procedure. It is tested on the MediaEval social event detection challenge data and is compared to a multimodal spectral clustering approach that uses early fusion. By taking advantage of the explicit supervisory signal, it achieves superior clustering accuracy and additionally requires the specification of a much smaller number of parameters. Moreover, the proposed approach has wider scope; it is not only applicable to the task of social event detection, but to other multimodal clustering problems as well.

We present a technical demonstration of an online city exploration application that helps users identify interesting spots in a city by use of spatio-temporal analysis and clustering of user contributed photos. Our framework analyzes the spatial distribution of large city-centered collections of user contributed photos at different time scales in order to index the most popular spots of a city in a time-aware manner. Subsequently, the photo sets belonging to the same spatiotemporal context are clustered in order to extract representative photos for each spot. The resulting application enables users to obtain flexible summaries of the most important spots in a city given a temporal slice (time of the day, month, season). The demonstration will be based on a photo dataset covering major European cities.