Open information extraction (OIE) methods extract plenty of OIE triples$< $ $>from unstructured text, which compose large open knowledge bases (OKBs). Noun phrases and relation phrases in such OKBs are not canonicalized, which leads to scattered and redundant facts. It is found that two views of knowledge (i.e., a fact view based on the fact triple and a context view based on the fact triple's source context) provide complementary information that is vital to the task of OKB canonicalization, which clusters synonymous noun phrases and relation phrases into the same group and assigns them unique identifiers. In order to leverage these two views of knowledge jointly, we propose CMVC+, a novel unsupervised framework for canonicalizing OKBs without the need for manually annotated labels. Specifically, we propose a multi-view CHF K-Means clustering algorithm to mutually reinforce the clustering of view-specific embeddings learned from each view by considering the clustering quality in a fine-grained manner. Furthermore, we propose a novel contrastive learning module to refine the learned view-specific embeddings and further enhance the canonicalization performance. We demonstrate the superiority of our framework through extensive experiments on multiple real-world OKB data sets against state-of-the-art methods.

The enormous growth of sensing devices on the Internet of Multimedia Things (IoMT) has emphasized the need for event processing technologies that deal with the complex nature of unstructured multimodal content. The paper highlights the emerging challenges for event processing applications as we shift to creating more real-time multimodal streams from IoMT. A key challenge is that fewer assumptions hold on the nature of the multimodal events produced, including the existence of a predefined schema and the semantics and meaning of unstructured content. This problem poses an opportunity for a new Open World Event Processing (OWEP) paradigm to natively support unstructured multimodal contents and unknown event types. OWEP offers unique challenges, which, when addressed, improve the practicality of event processing for modern multimodal event streams. We detail a high-level approach for an OWEP engine and highlight this new paradigm’s research opportunities.

Two major trends significantly changed the global Artificial Intelligence (AI) and Data landscape. Recent AI and Machine Learning developments are driving a paradigm shift to creating large task-agnostic foundation models pre-trained using web-scale data. Foundation models are then adapted to different downstream tasks via techniques such as fine-tuning. At the same time, we see a movement to the creation of large-scale data-sharing infrastructures. Data Spaces are an emerging approach to data management and sharing at the core of the European Data Strategy to provide access to high-quality data for AI. This paper brings together work on foundation models and data spaces into a holistic vision for Foundation Data Space Models. The paper highlights the data management requirements challenges for data spaces and details a high-level approach for foundation data space models together with a unified lifecycle for data spaces and foundation models. Finally, it sets out a research agenda.

Multimedia Event Processing (MEP) systems play a critical role in various Internet of Things (IoT) applications, including Smart Cities and Health and Safety, by processing large amounts of multimedia data streams. These systems often leverage state-of-the-art Deep Neural Network (DNN) models to enhance their capabilities. However, the growth of Cloud and Edge Big Data applications has imposed a significant environ-mental burden, further intensified by the substantial energy consumption associated with certain DNN model operations.This study addresses the environmental impact of growing Big Data stream applications and focuses on optimising MEP systems to mitigate this issue. We tackle uncertainties arising from user-defined Quality of Service (QoS) interpretations and service worker measurement imprecisions by using uncertainty-aware solutions for the service selection problem in order to improve the QoS within MEP systems.Our results reveal substantial advantages in employing uncertainty-aware strategies. These approaches consistently enhance QoS metrics, outperforming their uncertainty-oblivious counterparts. Specifically, we report improvements in more than 67%, 69%, and 20% of the scenarios, on average, for energy consumption, latency, and accuracy, respectively. These enhancements become evident within just three hours of processing, resulting in energy savings of up to 1.2 kilowatt-hours and latency reductions of 213 seconds, with a 0.29% average loss in query accuracy. These strategies improve system efficiency and ecological sustainability while incurring a small accuracy trade-off. When extrapolated over a year, the environmental benefits become even more noticeable, surpassing the energy requirements for a 1000 Km electric vehicle round-trip from Amsterdam to Paris and back.

When the global pandemic struck in 2020, most countries established task forces to meet a challenge that impacted governmental resources. It became apparent that data, intelligence gathering, and both modelling and predictive capabilities were required. While artificial intelligence (AI) based solutions had already begun to emerge within the public sector, the Covid-19 pandemic accelerated this process. In particular, modelling of case numbers with the development of predictive algorithms. The development of AI solutions for public sector organizations is inherently multidisciplinary. This is crucial to understanding how solutions can be developed, outputs understood, and the benefits and risks measured. Furthermore, the development of AI solutions often requires data which may not be accessible from a single location. In the case of Covid-19 modelling, data must be extracted from multiple locations to construct data assets. In this research, a collaborative approach to developing machine learning expertise for the public sector is presented. Using Covid-19 as a case study, the role of different government sectors when building data assets is examined along with the use of standard data models, and how this type of cooperation led to the development of a pipeline for data assets to underpin AI solutions for the public sector.

Amidst prevailing healthcare challenges, a dynamic solution emerges, fusing knowledge graph technology, clinical trials optimization, dataspace integration, and AI innovation. This unified approach tackles issues like limited patient insights, suboptimal trial designs, and imprecise treatments. By interlinking diverse data through knowledge graphs, this method illuminates disease trends, therapeutic efficacies, and patient prognoses. AI techniques, especially machine learning, contribute predictive power by unveiling hidden patterns for accurate diagnostics, prognostics, and personalized treatments. This multidisciplinary fusion transforms clinical trials, enhancing comprehensiveness and precision through real-world data analysis and subgroup identification. In reshaping healthcare, this proposition aims to accelerate treatment personalization, elevate therapeutic efficacy, and empower informed medical decisions, encompassing the essence of ’Advancing Healthcare through Innovation: Knowledge Graphs, Clinical Trials, Dataspace, and AI’.

Foundation Models are pivotal in advancing artificial intelligence, driving notable progress across diverse areas. When merged with dataspace, these models enhance our capability to develop algorithms that are powerful, predictive, and honor data sovereignty and quality. This paper highlights the potential benefits of a comprehensive repository of Foundation Models, contextualized within dataspace. Such an archive can streamline research, development, and education by offering a comparative analysis of various models and their applications. While serving as a consistent reference point for model assessment and fostering collaborative learning, the repository does face challenges like unbiased evaluations, data privacy, and comprehensive information delivery. The paper also notes the importance of the repository being globally applicable, ethically constructed, and user-friendly. We delve into the nuances of integrating Foundation Models within dataspace, balancing the repository’s strengths against its limitations.

Scene graph generation aims to capture the semantic elements in images by modelling objects and their relationships in a structured manner, which are essential for visual understanding and reasoning tasks including image captioning, visual question answering, multimedia event processing, visual storytelling and image retrieval. The existing scene graph generation approaches provide limited performance and expressiveness for higher-level visual understanding and reasoning. This challenge can be mitigated by leveraging commonsense knowledge, such as related facts and background knowledge, about the semantic elements in scene graphs. In this paper, we propose the infusion of diverse commonsense knowledge about the semantic elements in scene graphs to generate rich and expressive scene graphs using a heterogeneous knowledge source that contains commonsense knowledge consolidated from seven different knowledge bases. The graph embeddings of the object nodes are used to leverage their structural patterns in the knowledge source to compute similarity metrics for graph refinement and enrichment. We performed experimental and comparative analysis on the benchmark Visual Genome dataset, in which the proposed method achieved a higher recall rate ( $$R@K = 29.89, 35.4, 39.12$$ for $$K = 20, 50, 100$$ ) as compared to the existing state-of-the-art technique ( $$R@K = 25.8, 33.3, 37.8$$ for $$K = 20, 50, 100$$ ). The qualitative results of the proposed method in a downstream task of image generation showed that more realistic images are generated using the commonsense knowledge-based scene graphs. These results depict the effectiveness of commonsense knowledge infusion in improving the performance and expressiveness of scene graph generation for visual understanding and reasoning tasks.

Advances in Deep Neural Network (DNN) techniques have revolutionized video analytics and unlocked the potential for querying and mining video event patterns. This paper details GNOSIS, an event processing platform to perform near-real-time video event detection in a distributed setting. GNOSIS follows a serverless approach where its component acts as independent microservices and can be deployed at multiple nodes. GNOSIS uses a declarative query-driven approach where users can write customize queries for spatiotemporal video event reasoning. The system converts the incoming video streams into a continuous evolving graph stream using machine learning (ML) and DNN models pipeline and applies graph matching for video event pattern detection. GNOSIS can perform both stateful and stateless video event matching. To improve Quality of Service (QoS), recent work in GNOSIS incorporates optimization techniques like adaptive scheduling, energy efficiency, and content-driven windows. This paper demonstrates the Occupational Health and Safety query use cases to show the GNOSIS efficacy.

The Internet of Multimedia Things (IoMT) is an emerging concept due to the large amount of multimedia data produced by sensing devices. Existing event-based systems mainly focus on scalar data, and multimedia event-based solutions are domain-specific. Multiple applications may require handling of numerous known/unknown concepts which may belong to the same/different domains with an unbounded vocabulary. Although deep neural network-based techniques are effective for image recognition, the limitation of having to train classifiers for unseen concepts will lead to an increase in the overall response-time for users. Since it is not practical to have all trained classifiers available, it is necessary to address the problem of training of classifiers on demand for unbounded vocabulary. By exploiting transfer learning based techniques, evaluations showed that the proposed framework can answer within ~0.01 min to ~30 min of response-time with accuracy ranges from 95.14% to 98.53%, even when all subscriptions are new/unknown.

The heterogeneity of energy ontologies hinders the interoperability between ontology-based energy management applications to perform a large-scale energy management. Thus, there is the need for a global ontology that provides common vocabularies to represent the energy subdomains. A global energy ontology must provide a balance of reusability–usability to moderate the effort required to reuse it in different applications. This paper presents DABGEO: a reusable and usable global ontology for the energy domain that provides a common representation of energy domains represented by existing energy ontologies. DABGEO can be reused by ontology engineers to develop ontologies for specific energy management applications. In contrast to previous global energy ontologies, it follows a layered structure to provide a balance of reusability–usability. In this work, we provide an overview of the structure of DABGEO and we explain how to reuse it in a particular application case. In addition, the paper includes an evaluation of DABGEO to demonstrate that it provides a balance of reusability–usability.

Video data is highly expressive and has traditionally been very difficult for a machine to interpret. Querying event patterns from video streams is challenging due to its unstructured representation. Middleware systems such as Complex Event Processing (CEP) mine patterns from data streams and send notifications to users in a timely fashion. Current CEP systems have inherent limitations to query video streams due to their unstructured data model and lack of expressive query language. In this work, we focus on a CEP framework where users can define high-level expressive queries over videos to detect a range of spatiotemporal event patterns. In this context, we propose- i) VidCEP, an in-memory, on the fly, near real-time complex event matching framework for video streams. The system uses a graph-based event representation for video streams which enables the detection of high-level semantic concepts from video using cascades of Deep Neural Network models, ii) a Video Event Query language (VEQL) to express high-level user queries for video streams in CEP, iii) a complex event matcher to detect spatiotemporal video event patterns by matching expressive user queries over video data. The proposed approach detects spatiotemporal video event patterns with an F-score ranging from 0.66 to 0. S9. VidCEP maintains near real-time performance with an average throughput of 70 frames per second for 5 parallel videos with sub-second matching latency.

The proliferation of sensor devices and services along with the advances in event processing brings many new opportunities as well as challenges. It is now possible to provide, analyze and react upon real-time, complex events in urban environments. When existing event services do not provide such complex events directly, an event service composition maybe required. However, it is difficult to determine which event service candidates (or service compositions) best suit users' and applications' quality-of-service requirements. A sub-optimal service composition may lead to inaccurate event detection, lack of system robustness etc. In this paper, the authors address these issues by first providing a quality-of-service aggregation schema for complex event service compositions and then developing a genetic algorithm to efficiently create near-optimal event service compositions. The authors evaluate their approach with both real sensor data collected via Internet-of-Things services as well as synthesised datasets.

This paper introduces the flag-verify-fix pattern that employs spatial crowdsourcing for city maintenance. The patterns motivates the need for appropriate assignment of dynamically arriving spatial tasks to a pool for workers on the ground. The assignment is aimed at maximizing the coverage of tasks spread over spatial locations; however, the coverage depends of willingness of workers to perform tasks assigned to them. We introduce the maximum coverage assignment problem that formulates two design issues of dynamic assignment. The quantity issue determines the number of worker required for a task and selection issue determines the set of workers. We propose an adaptive algorithm that uses location diversity based on a location-based social network to address the quantity issue and employs Thompson sampling for selecting the workers by learning their willingness. We evaluate the performance of the proposed algorithm in terms of coverage and number of assignments using real world datasets. The results show that our proposed algorithm achieves 30%--50% more coverage than the baseline algorithms, while requiring less workers per task.

The Internet of Things (IoT) envisions smart objects collecting and sharing data at a global scale via the Internet. One challenging issue is how to disseminate data to relevant consumers efficiently. This paper leverages semantic technologies, such as Linked Data, which can facilitate machine-to-machine (M2M) communications to build an efficient information dissemination system for semantic IoT. The system integrates Linked Data streams generated from various data collectors and disseminates matched data to relevant data consumers based on conjunctive triple pattern queries registered in the system by the consumers. We also design a new data structure, CTP-automata, to meet the high performance needs of Linked Data dissemination. We evaluate our system using a real-world dataset generated from a Smart Building Project. With CTP-automata, the proposed system can disseminate Linked Data at a speed of an order of magnitude faster than the existing approach with thousands of registered conjunctive queries.

The Internet of Things (IoT) envisions smart objects collecting and sharing data at a global scale via the Internet. One challenging issue is how to disseminate data to relevant data consumers efficiently. In this paper, we leverage semantic technologies which can facilitate machine-to-machine communications, such as Linked Data, to build an efficient information dissemination system for semantic IoT. The system integrates Linked Data streams generated from various data collectors and disseminates matched data to relevant data consumers based on Basic Graph Patterns (BGPs) registered in the system by those consumers. To efficiently match BGPs against Linked Data streams, we introduce two types of matching, namely semantic matching and pattern matching, by considering whether the matching process supports semantic relatedness computation. Two new data structures, namely MVR-tree and TP-automata, are introduced to suit these types of matching respectively. Experiments show that an MVR-tree designed for semantic matching can achieve a twofold increase in throughput compared with the naive R-tree based method. TP-automata, as the first approach designed for pattern matching over Linked Data streams, also provides two to three orders of magnitude improvements on throughput compared with semantic matching approaches.

The dynamicity of sensor data sources and publishing real-time sensor data over a generalised infrastructure like the Web pose a new set of integration challenges. Semantic Sensor Networks demand excessive expressivity for efficient formal analysis of sensor data. This article specifically addresses the problem of adapting data model specific or context-specific properties in automatic generation of multidimensional data cubes. The idea is to generate data cubes on-the-fly from syntactic sensor data to sustain decision making, event processing and to publish this data as Linked Open Data.

This paper demonstrates Treo, a natural language query mechanism for Linked Data graphs. The approach uses a distributional semantic vector space model to semantically match user query terms with data, supporting vocabulary-independent (or schema-agnostic) queries over structured data.