International audience

International audience

Tropes — recurring narrative elements like the “smoking gun” or the “veil of secrecy” — are often used in movies to convey familiar patterns. However, they also play a significant role in online communication about societal issues, where they can oversimplify complex matters and deteriorate public discourse. Recognizing these tropes can offer insights into the emotional manipulation and potential bias present in online discussions. This paper addresses the challenge of automatically detecting tropes in social media posts. We define the task, distinguish it from previous work, and create a ground-truth dataset of social media posts related to vaccines and immigration, manually labeled with tropes. Using this dataset, we develop a supervised machine learning technique for multi-label classification, fine-tune a model, and demonstrate its effectiveness experimentally. Our results show that tropes are common across domains and that fine-tuned models can detect them with high accuracy.

In recent years, continuous integration and deployment (CI/CD) has become increasingly popular in both the opensource community and industry. Evaluating CI/CD performance is a critical aspect of software development, as it not only helps minimize execution costs but also ensures faster feedback for developers. Despite its importance, there is limited fine-grained knowledge about the performance of CI/CD processes, while this knowledge is essential for identifying bottlenecks and optimization opportunities. Moreover, the availability of large-scale, publicly accessible datasets of CI/CD logs remains scarce. The few datasets that do exist are often outdated and lack comprehensive coverage. To address this gap, we introduce GHALogs, a new dataset comprising 116k CI/CD workflows executed using GitHub Actions (GHA) across 25k public code projects spanning 20 different programming languages. This dataset includes 513k workflow runs encompassing 2.3 million individual steps. For each workflow run, we provide detailed metadata along with complete run logs. To the best of our knowledge, this is the largest dataset of CI/CD runs that includes full log data. The inclusion of these logs enables more in-depth analysis of CI/CD pipelines, offering insights that cannot be gleaned solely from code repositories. We postulate that this dataset will facilitate future CI/CD pipeline behavior research through log-based analysis. Potential applications include performance evaluation (e.g., measuring task execution times) and root cause analysis (e.g., identifying reasons for pipeline failures).

While internet communications have been originally all in the clear, the past decade has seen secure protocols like TLS becoming pervasive, significantly improving internet security for individuals and enterprises. However, encrypted traffic raises new challenges for intrusion detection and network monitoring. Existing interception solutions such as Man-In-The-Middle are undesirable in many settings: they tend to lower overall security or are challenging to use at scale. We present X-Ray-TLS, a new target-agnostic TLS decryption method that supports TLS 1.2, TLS 1.3, and QUIC. Our method relies only on existing kernel facilities and does not require a hypervisor or modification of the target programs, making it easily applicable at scale. X-Ray-TLS works on major TLS libraries by extracting TLS secrets from process memory using a memory changes reconstruction algorithm. It works with TLS hardening, such as certificate pinning and perfect forward secrecy. We benchmark X-Ray-TLS on major TLS libraries, CLI tools, and a web browser. We show that X-Ray-TLS significantly reduces the manual effort required to decrypt TLS traffic of programs running locally, thus simplifying security analysis or reverse engineering. We identified several use cases for X-Ray-TLS, such as large-scale TLS decryption for CI/CD pipelines to support the detection of software supply chain attacks.

Misinformation has a pervasive thread running through society, causing confusion, mistrust, and uncertainty. The detection, tracking, and countering of misinformation is a very active research area with an intense need for data about circulating claims and their attributes, fact-checks, and verification outcomes. Although various relevant datasets exist, they tend to be of limited scope in terms of time coverage, topics, country, language, and quantity. In this paper, we introduce CimpleKG as an open and continuously updated semantic resource. CimpleKG links daily updated data from 77 fact-checking organisations with over 217k documents from static misinformation datasets. The knowledge graph is also augmented with relevant textual features and entities extracted from the textual data integrated into the graph. At the time of writing, the knowledge graph contains more than 15m triples, including 263k+ distinct entities and 1m textual features with over 203k fact-checked claims, spanning 26 languages and 36 countries. CimpleKG is publicly available and has been used in various research studies and web applications. Resource Type: Knowledge Graph. License: CC BY-NC-SA 4.0. SPARQL Endpoint: https://purl.org/net/cimplekg/sparql . KG Releases: https://purl.org/net/cimplekg/knowledge-graph . KG Explorer: https://purl.org/net/cimplekg/explorer .

The complexity of Information and Communications Technology (ICT) systems, such as enterprise or Internet access provider networks, entails uncertainty in causal reasoning for efficient incident management. In this work, we propose to use knowledge graphs and explicit representation of incident context to enable support teams to provide a quick and effective response to complex incident situations. Formal analysis and expert opinions are used to analyze challenges in providing knowledge about relationships between events and incidents in network operations. We make use of an RDF knowledge graph generated from a real industrial settings and representing the network topology in terms of equipments and applications, past incidents and their resolutions. We then demonstrate the effectiveness of using a graph embeddings-based classifier to categorize incident tickets based on context and link anomaly models with their logical representation.

Traditional topic modeling approaches generally rely on document-term co-occurrence statistics to find latent topics in a collection of documents. However, relying only on such statistics can yield incoherent or hard to interpret results for the end-users in many applications where the interest lies in interpreting the resulting topics (e.g. labeling documents, comparing corpora, guiding content exploration, etc.). In this work, we propose to leverage external common sense knowledge, i.e. information from the real world beyond word co-occurrence, to find topics that are more coherent and more easily interpretable by humans. We introduce the Common Sense Topic Model (CSTM), a novel and efficient approach that augments clustering with knowledge extracted from the ConceptNet knowledge graph. We evaluate this approach on several datasets alongside commonly used models using both automatic and human evaluation, and we show how it shows superior affinity to human judgement. The code for the experiments as well as the training data and human evaluation are available at https://github.com/D2KLab/CSTM.

Pre-trained word embeddings constitute an essential building block for many NLP systems and applications, notably when labeled data is scarce. However, since they compress word meanings into a fixed-dimensional representation, their use usually lack interpretability beyond a measure of similarity and linear analogies that do not always reflect real-world word relatedness, which can be important for many NLP applications. In this paper, we propose a model which extracts topics from text documents based on the common-sense knowledge available in ConceptNet [Speer et al., 2017] - a semantic concept graph that explicitly encodes real-world relations between words - and without any human supervision. When combining both ConceptNet’s knowledge graph and graph embeddings, our approach outperforms other baselines in the zero-shot setting, while generating a human-understandable explanation for its predictions through the knowledge graph. We study the importance of some modeling choices and criteria for designing the model, and we demonstrate that it can be used to label data for a supervised classifier to achieve an even better performance without relying on any humanly-annotated training data. We publish the code of our approach at https://github.com/D2KLab/ZeSTE and we provide a user friendly demo at https://zeste.tools.eurecom.fr/.

SKG4J targeted contributions at the interface between Artificial Intelligence, Data Management and its implications for journalistic practice. The first version of the workshop accepted three submissions with topics emphasising the complementary requirements for delivering realistic journalistic knowledge extraction/management platforms.

Technological developments in comprehensive video understanding - detecting and identifying visual elements of a scene, combined with audio understanding (music, speech), as well as aligned with textual information such as captions, subtitles, etc. and background knowledge - have been undergoing a significant revolution during recent years. The workshop brings together experts from academia and industry in order to discuss the latest progress in artificial intelligence research in topics related to multimodal information analysis, and in particular, semantic analysis of video, audio, and textual information for smart digital TV content production, access and delivery.

More than 2 millions of new books are published every year and choosing a good book among the huge amount of available options can be a challenging endeavor. Recommender systems help in choosing books by providing personalized suggestions based on the user reading history. However, most book recommender systems are based on collaborative filtering, involving a long onboarding process that requires to rate many books before providing good recommendations. Tinderbook provides book recommendations, given a single book that the user likes, through a card-based playful user interface that does not require an account creation. Tinderbook is strongly rooted in semantic technologies, using the DBpedia knowledge graph to enrich book descriptions and extending a hybrid state-of-the-art knowledge graph embeddings algorithm to derive an item relatedness measure for cold start recommendations. Tinderbook is publicly available ( http://www.tinderbook.it ) and has already generated interest in the public, involving passionate readers, students, librarians, and researchers. The online evaluation shows that Tinderbook achieves almost 50% of precision of the recommendations.

Modern vehicles produce big data with a wide variety of formats due to missing open standards. Thus, abstractions of such data in the form of descriptive labels are desired to facilitate the development of applications in the automotive domain. We propose an approach to reduce vehicle sensor data into semantic outcomes of dangerous driving events based on aggressive maneuvers. The supervised time-series classification is implemented with Random Forest and Recurrent Neural Network separately. Our approach works with signals of a real vehicle obtained through a back-end solution, with the challenge of low and variable sampling rates. We introduce the idea of having a dangerous driving classifier as the first discriminant of relevant instances for further enrichment (e.g., type of maneuver). Additionally, we suggest a method to increment the number of driving samples for training machine learning models by weighting the window instances based on the portion of the labeled event they include. We show that a dangerous driving classifier can be used as a first discriminant to enable data integration and that transitions in driving events are relevant to consider when the dataset is limited, and sensor data has a low and unreliable frequency.

No abstract available.

Technological developments in comprehensive video understanding - detecting and identifying visual elements of a scene, combined with audio understanding (music, speech), as well as aligned with textual information such as captions, subtitles, etc. and background knowledge - have been undergoing a significant revolution during recent years. The workshop brings together experts from academia and industry in order to discuss the latest progress in artificial intelligence research in topics related to multimodal information analysis, and in particular, semantic analysis of video, audio, and textual information for smart digital TV content production, access and delivery.

Three major French cultural institutions—the French National Library (BnF), Radio France and the Philharmonie de Paris—have come together in order to develop shared methods to describe semantically their catalogs of music works and events. This process comprises the construction of knowledge graphs representing the data contained in these catalogs following a novel agreed upon ontology that extends CIDOC-CRM and FRBRoo, the linking of these graphs and their open publication on the web. A number of specialized tools that allow for the reproduction of this process are developed, as well as web applications for easy access and navigation through the data. The paper presents one of the main outcomes of this project—the DOREMUS knowledge graph, consisting of three linked datasets describing classical music works and their associated events (e.g., performances in concerts). This resource fills an important gap between library content description and music metadata. We present the DOREMUS pipeline for lifting and linking the data, the tools developed for these purposes, as well as a search application allowing to explore the data.

Second screen applications are becoming key for broadcasters exploiting the convergence of TV and Internet. Authoring such applications however remains costly. In this paper, we present a second screen authoring application that leverages multimedia content analytics and social media monitoring. A back-office is dedicated to easy and fast content ingestion, segmentation, description and enrichment with links to entities and related content. From the back-end, broadcasters can push enriched content to front-end applications providing customers with highlights, entity and content links, overviews of social network, etc. The demonstration operates on political debates ingested during the 2017 French presidential election, enabling insights on the debates.

Knowledge Graphs have proven to be extremely valuable to recommender systems, as they enable hybrid graph-based recommendation models encompassing both collaborative and content information. Leveraging this wealth of heterogeneous information for top-N item recommendation is a challenging task, as it requires the ability of effectively encoding a diversity of semantic relations and connectivity patterns. In this work, we propose entity2rec, a novel approach to learning user-item relatedness from knowledge graphs for top-N item recommendation. We start from a knowledge graph modeling user-item and item-item relations and we learn property-specific vector representations of users and items applying neural language models on the network. These representations are used to create property-specific user-item relatedness features, which are in turn fed into learning to rank algorithms to learn a global relatedness model that optimizes top-N item recommendations. We evaluate the proposed approach in terms of ranking quality on the MovieLens 1M dataset, outperforming a number of state-of-the-art recommender systems, and we assess the importance of property-specific relatedness scores on the overall ranking quality.

Ensuring data quality in Linked Open Data is a complex process as it consists of structured information supported by models, ontologies and vocabularies and contains queryable endpoints and links. In this paper, the authors first propose an objective assessment framework for Linked Data quality. The authors build upon previous efforts that have identified potential quality issues but focus only on objective quality indicators that can measured regardless on the underlying use case. Secondly, the authors present an extensible quality measurement tool that helps on one hand data owners to rate the quality of their datasets, and on the other hand data consumers to choose their data sources from a ranked set. The authors evaluate this tool by measuring the quality of the LOD cloud. The results demonstrate that the general state of the datasets needs attention as they mostly have low completeness, provenance, licensing and comprehensibility quality scores.

In this paper, we present the 3cixty Knowledge Base, which collects and harmonizes descriptions of events, places, transportation facilities and user-generated data such as reviews of the city and Expo site of Milan. This knowledge base is used by a set of web and mobile applications to guide Expo Milano 2015 visitors in the city and in the exhibit, allowing them to find places, satellite events and transportation facilities around Milan. As of July 24th, 2015 the knowledge base contains 18665 unique events, 225821 unique places, 94789 reviews, and 9343 transportation facilities, collected from several static, near- and real time local and global data providers, including Expo Milano 2015 official services and numerous social media platforms. The ontologies used as a backbone for structuring the knowledge base follow a rigorous development method where the design principle has generally been to re-use existing ontologies when they exist. We think that the lessons learned from this development will be useful for similar endeavors in other cities or large events around the world with a similar ecosystem of data provisioning services.

The Web enables to have access to silo-ed information describing news articles, often offering a multitude of viewpoints that, once combined, can provide a broader picture of the story being reported on the news. In this paper, we propose an approach that automatically extracts representative features of a news item, namely named entities, from textual content attached to a video item (subtitles) and from a set of documents from the Web collected using entity expansion techniques. Approaches relying on entity expansion generally try to collect and process the important facts behinds a particular news item, but they are often too dependent on frequency-based functions and information retrieval techniques thus neglecting the multi-dimensional relationships that are established among the entities. We propose a concentric-based approach that enables to represent the context of a news item, by harmonizing into a single model the representative entities, which can be extracted using information retrieval and natural language processing techniques (Core), and other entities that get prominent according to different dimensions such as informativeness, semantic connectivity, or popularity (Crust). We compare our approach with a baseline by analyzing the compactness of the generated summary on an existing gold standard available on the Web. Results of the experiments show that our approach converges faster to the ideal compact news snapshot with an improvement of 36.9% over the baseline.

We live in a constantly evolving world where news stories and relevant facts are happening every moment. For each of those stories, numerous news articles, posts, and social media reactions are created, offering a multitude of viewpoints about what is happening around us. Many applications have tried to deal with this complexity from very different angles, targeting particular needs, reconstructing certain parts of the story, and exploiting certain visualization paradigms. In this paper, we identify those challenges and study how an adequate news story representation can effectively support the different phases of the news consumption process. We propose an innovative model called News Semantic Snapshot (NSS) that is designed to capture the entire context of a news item. This model can feed very different applications assisting the users before, during, and after the news story consumption. It formalizes a duality in the news annotations that distinguishes between representative entities and relevant entities, and considers different relevancy dimensions that are incorporated into the model in the form of concentric layers. Finally, we analyze the impact of this NSS on existing prototypes and how it can support future ones.

We present GERBIL, an evaluation framework for semantic entity annotation. The rationale behind our framework is to provide developers, end users and researchers with easy-to-use interfaces that allow for the agile, fine-grained and uniform evaluation of annotation tools on multiple datasets. By these means, we aim to ensure that both tool developers and end users can derive meaningful insights pertaining to the extension, integration and use of annotation applications. In particular, GERBIL provides comparable results to tool developers so as to allow them to easily discover the strengths and weaknesses of their implementations with respect to the state of the art. With the permanent experiment URIs provided by our framework, we ensure the reproducibility and archiving of evaluation results. Moreover, the framework generates data in machine-processable format, allowing for the efficient querying and post-processing of evaluation results. Finally, the tool diagnostics provided by GERBIL allows deriving insights pertaining to the areas in which tools should be further refined, thus allowing developers to create an informed agenda for extensions and end users to detect the right tools for their purposes. GERBIL aims to become a focal point for the state of the art, driving the research agenda of the community by presenting comparable objective evaluation results.

This paper introduces a framework for establishing links between related media fragments within a collection of videos. A set of analysis techniques is applied for extracting information from different types of data. Visual-based shot and scene segmentation is performed for defining media fragments at different granularity levels, while visual cues are detected from keyframes of the video via concept detection and optical character recognition (OCR). Keyword extraction is applied on textual data such as the output of OCR, subtitles and metadata. This set of results is used for the automatic identification and linking of related media fragments. The proposed framework exhibited competitive performance in the Video Hyperlinking sub-task of MediaEval 2013, indicating that video scene segmentation can provide more meaningful segments, compared to other decomposition methods, for hyperlinking purposes.

This paper presents the 2012 Social Event Detection dataset (SED2012). The dataset constitutes a challenging benchmark for methods that detect social events in large collections of multimedia items. More specifically, the dataset comprises more than 160 thousands of Flickr photos and their accompanying metadata, as well as a list of 149 manually selected and annotated target events, each of which is defined as a set of relevant photos. This paper discusses the challenges defined as part of SED 2012, the data collection process, the dataset and its basic statistics, the ground truth creation and the suggested evaluation methodology.

An ever increasing number of social services offer thousands of diverse events per day. Users tend to be overwhelmed by the massive amount of information available, especially with limited browsing options perceived in many event web services. To alleviate this information overload, a recommender system becomes a vital component for assisting users selecting relevant events. However, such system faces a number of challenges owed to the the inherent complex nature of an event. In this paper, we propose a novel hybrid approach built on top of Semantic Web. On the one hand, we use a content-based system enriched with Linked Data to overcome the data sparsity, a problem induced by the transiency of events. On the other hand, we incorporate a collaborative filtering to involve the social aspect, an influential feature in decision making. This hybrid system is enhanced by the integration of a user diversity model designed to detect user propensity towards specific topics. We show how the hybridization of CB+CF systems and the integration of interest diversity features are important to improve predictions. Experimental results demonstrate the effectiveness of our approach using precision and recall measures.

This paper presents a theoretical framework to analyze the relative merits of the two most general, dominant approaches to speaker diarization involving bottom-up and top-down hierarchical clustering. We present an original qualitative comparison which argues how the two approaches are likely to exhibit different behavior in speaker inventory optimization and model training: bottom-up approaches will capture comparatively purer models and will thus be more sensitive to nuisance variation such as that related to the speech content; top-down approaches, in contrast, will produce less discriminative speaker models but, importantly, models which are potentially better normalized against nuisance variation. We report experiments conducted on two standard, single-channel NIST RT evaluation datasets which validate our hypotheses. Results show that competitive performance can be achieved with both bottom-up and top-down approaches (average DERs of 21% and 22%), and that neither approach is superior. Speaker purification, which aims to improve speaker discrimination, gives more consistent improvements with the top-down system than with the bottom-up system (average DERs of 19% and 25%), thereby confirming that the top-down system is less discriminative and that the bottom-up system is less stable. Finally, we report a new combination strategy that exploits the merits of the two approaches. Combination delivers an average DER of 17% and confirms the intrinsic complementary of the two approaches.

Spoken term detection (STD) is a key technology for spoken information retrieval. As compared to the conventional speech transcription and keyword spotting, STD is an open-vocabulary task and has to address out-of-vocabulary (OOV) terms. Approaches based on subword units, for example phones, are widely used to solve the OOV issue; however, performance on OOV terms is still substantially inferior to that of in-vocabulary (INV) terms. The performance degradation on OOV terms can be attributed to a multitude of factors. One particular factor we address in this article is the unreliable confidence estimation caused by weak acoustic and language modeling due to the absence of OOV terms in the training corpora. We propose a direct posterior confidence derived from a discriminative model, such as multilayer perceptron (MLP). The new confidence considers a wide-range acoustic context which is usually important for speech recognition and retrieval; moreover, it localizes on detected speech segments and therefore avoids the impact of long-span word context which is usually unreliable for OOV term detection. In this article, we first develop an extensive discussion about the modeling weakness problem associated with OOV terms, and then propose our approach to address this problem based on direct poster confidence. Our experiments carried out on spontaneous and conversational multiparty meeting speech, demonstrate that the proposed technique provides a significant improvement in STD performance as compared to conventional lattice-based confidence, in particular for OOV terms. Furthermore, the new confidence estimation approach is fused with other advanced techniques for OOV treatment, such as stochastic pronunciation modeling and discriminative confidence normalization. This leads to an integrated solution for OOV term detection that results in a large performance improvement.

While bottom-up approaches have emerged as the standard, default approach to clustering for speaker diarization we have always found the top-down approach gives equivalent or superior performance. Our recent work shows that significant gains in performance can be obtained when cluster purification is applied to the output of top down systems but that it can degrade performance when applied to the output of bottom-up systems. This paper demonstrates that these observations can be accounted for by factors unrelated to the speaker and that they can impact more strongly on the performance of bottom-up clustering strategies than top-down strategies. Experimental results confirm that clusters produced through top-down clustering are better normalized against phone variation than those produced through bottom-up clustering and that this accounts for the observed inconsistencies in purification performance. The work highlights the need for marginalization strategies which should encourage convergence toward different speakers rather than toward nuisance factors such as that those related to the linguistic content.

Spoken term detection (STD) systems usually arrive at many overlapping detections which are often addressed with some pragmatic approaches, e.g. choosing the best detection to represent all the overlaps. In this paper we present a theoretical study based on a concept of acceptance space. In particular, we present two confidence estimation approaches based on Bayesian and evidence perspectives respectively. Analysis shows that both approaches possess respective ad vantages and shortcomings, and that their combination has the potential to provide an improved confidence estimation. Experiments conducted on meeting data confirm our analysis and show considerable performance improvement with the combined approach, in particular for out-of-vocabulary spoken term detection with stochastic pronunciation modeling.

We present a method combining semantic inferencing and visual analysis for finding automatically media (photos and videos) illustrating events. We report on experiments validating our heuristic for mining media sharing platforms and large event directories in order to mutually enrich the descriptions of the content they host. Our overall goal is to design a web-based environment that allows users to explore and select events, to inspect associated media, and to discover meaningful, surprising or entertaining connections between events, media and people participating in events. We present a large dataset composed of semantic descriptions of events, photos and videos interlinked with the larger Linked Open Data cloud and we show the benefits of using semantic web technologies for integrating multimedia metadata.

International audience

Out-of-vocabulary (OOV) terms present a significant challenge to spoken term detection (STD). This challenge, to a large ex-tent, lies in the high degree of uncertainty in pronunciations of OOV terms. In previous work, we presented a stochastic pro-nunciation modeling (SPM) approach to compensate for this uncertainty. A shortcoming of our original work, however, is that the SPM was based on a joint-multigram model (JMM), which is suboptimal. In this paper, we propose to use con-ditional random fields (CRFs) for letter-to-sound conversion, which significantly improves quality of the predicted pronun-ciations. When applied to OOV STD, we achieve consider-able performance improvement with both a 1-best system and an SPM-based system. Index Terms: speech recognition, spoken term detection, con-ditional random field, joint multigram model

In this paper, we describe two examples of implementations of the Media Fragments URI specification which is currently being developed by the W3C Media Fragments Working Group. The group's mission is to create standard addressing schemes for media fragments on the Web using Uniform Resource Identifiers (URIs). We describe two scenarios to illustrate the implementations. More specifically, we show how User Agents (UA) will either be able to resolve media fragment URIs without help from the server, or will make use of a media fragments-aware server. Finally, we present some ongoing discussions and issues regarding the implementation of the Media Fragments specification.

We present an innovative approach for storing and processing extracted low-level image features based on current semantic Web technologies. We propose to use the COMM multimedia ontology as a "semantic" alternative to the MPEG-7 standard, which is at the same time largely compliant with it. We describe how COMM can be used directly or through its associated Java API.

Enabling an intelligent access to multimedia data requires a powerful description language. In this paper we demonstrate why the MPEG-7 standard fails to fulfill this task. We introduce then our proposition: an audio-visual specific description language modular reduced but designed to be extensible. This language is centered on the notions of descriptor and structure with a well-defined semantics. A descriptor can be a low-level feature automatically extracted from the signal or a higher semantic concept that will be used to annotate the video documents. The descriptors can be combined into structures according to defined models that provide description patterns.