Multimodal AI models are increasingly used in fields like healthcare, finance, and autonomous driving, where information is drawn from multiple sources or modalities such as images, texts, audios, videos. However, effectively managing uncertainty—arising from noise, insufficient evidence, or conflicts between modalities—is crucial for reliable decision-making. Current uncertainty-aware machine learning methods leveraging, for example, evidence averaging, or evidence accumulation underestimate uncertainties in high-conflict scenarios. Moreover, the state-of-the-art evidence averaging strategy is not order invariant and fails to scale to multiple modalities. To address these challenges, we propose a novel multimodal learning method with order-invariant evidence fusion and introduce a conflict-based discounting mechanism that reallocates uncertain mass when unreliable modalities are detected. We provide both theoretical analysis and experimental validation, demonstrating that unlike the previous work, the proposed approach effectively distinguishes between conflicting and non-conflicting samples based on the provided uncertainty estimates, and outperforms the previous models in uncertainty-based conflict detection.

Multimodal Deep Learning enhances decision-making by integrating diverse information sources, such as texts, images, audio, and videos. To develop trustworthy multimodal approaches, it is essential to understand how uncertainty impacts these models. We propose LUMA, a unique multimodal dataset, featuring audio, image, and textual data from 50 classes, specifically designed for learning from uncertain data. It extends the well-known CIFAR 10/100 dataset with audio samples extracted from three audio corpora, and text data generated using the Gemma-7B Large Language Model (LLM). The LUMA dataset enables the controlled injection of varying types and degrees of uncertainty to achieve and tailor specific experiments and benchmarking initiatives. LUMA is also available as a Python package including the functions for generating multiple variants of the dataset with controlling the diversity of the data, the amount of noise for each modality, and adding out-of-distribution samples. A baseline pre-trained model is also provided alongside three uncertainty quantification methods: Monte-Carlo Dropout, Deep Ensemble, and Reliable Conflictive Multi-View Learning. This comprehensive dataset and its tools are intended to promote and support the development, evaluation, and benchmarking of trustworthy and robust multimodal deep learning approaches. We anticipate that the LUMA dataset will help the research community to design more trustworthy and robust machine learning approaches for safety critical applications. The code and instructions for downloading and processing the dataset can be found at: https://github.com/bezirganyan/LUMA.

Choosing a suitable deep learning architecture for multimodal data fusion is a challenging task, as it requires the effective integration and processing of diverse data types, each with distinct structures and characteristics. In this paper, we introduce MixMAS, a novel framework for sampling-based mixer architecture search tailored to multimodal learning. Our approach automatically selects the optimal MLP-based architecture for a given multimodal machine learning (MML) task. Specifically, MixMAS utilizes a sampling-based micro-benchmarking strategy to explore various combinations of modality-specific encoders, fusion functions, and fusion networks, systematically identifying the architecture that best meets the task’s performance metrics.

In this paper, we propose M2-Mixer, an MLP-Mixer based architecture with multi-head loss for multimodal classification. It achieves better performances than the convolutional, recurrent, or neural architecture search based baseline models with the main advantage of conceptual and computational simplicity. The proposed multi-head loss function addresses the problem of modality predominance (i.e., when one of the modalities is favored over the others by the training algorithm). Our experiments demonstrate that our multimodal mixer architecture, combined with the multi-head loss function, outperforms the baseline models on two benchmark multimodal datasets: AVMNIST and MIMIC-III with respectively, on average, + 0.43% in accuracy and 6. 4 times reduction in training time and + 0.33% in accuracy and 13. 3 times reduction in training time, compared with previous best performing models.

Abstract In this article, we describe an imbalanced regression method for making predictions over imbalanced data streams. We present MORSTS (Multiple Output Regression for Streaming Time Series), an online ensemble regressors devoted to non‐stationary and imbalanced data streams. MORSTS relies on several multiple output regressor submodels, adopts a cost sensitive weighting technique for dealing with imbalanced datasets, and handles overfitting by means of the K‐fold cross validation. For assessment purposes, experiments have been conducted on known real datasets and compared with known base regression techniques.

The Web of Things (WoT) describes a set of standards by the World Wide Web Consortium (W3C) for the interoperability of different Internet of Things (IoT) platforms, and different application domains. Thus, it guarantees not only device-to-device interactions, but also, application-to-application communications, despite their platform heterogeneity. To identify and use provided services (also called resources) that are exposed by either the devices or the applications connected to a Web environment, describing them using an open, shared and dynamic knowledge representation is required, allowing them to interoperate on both syntactic and semantic levels. In this paper, we propose WoR, a Web of Resources ontology that provides a modular and a common vocabulary to describe Web resources. WoR can: (1) ease the discovery, the selection, and the composition of different kind of resources (exposed by connected Web devices or Web applications), (2) provide reasoning means to discover new information, and (3) allow future extensibility and adaptation to new domains needs. Experiments were made to evaluate our proposed WoR ontology, showing promising results on the effectiveness and the performance levels.

Semantic web service discovery has attracted a lot of attention in the last decade. Research conducted in this area can be (mainly) summarized as follows: (1) "monolith" matchmaking algorithms (and systems), and (2) schema matching-based techniques. In this paper we describe a flexible approach that takes leverage of existing schema matchers, leading to a multiple choice strategy for semantic service discovery. The approach has been implemented and validated in using the data collection provided by the S3 (Semantic Service Selection) community, and led to promising preliminary results.

Actually, the evaluation of matching tools is an entire, complex and complicated research subject which we are interested in. Complex because matching systems can regroup several matching techniques and complicated considering their multiple users. Considering quality as an important element to define, use and evolve particular systems (as information and manufacturing systems), we extend traditional approaches and we propose an evaluation approach based on software product quality principles. In this paper, we offer an evaluation method based on a quality model (characteristics, sub-characteristics, measures...) adapted to the specificities of scalable matching tools. To illustrate our approach, we provide some evaluation results over two scalable matching tools COMA++ and PLASMA.

Automated matching of service descriptions is the key to service discovery and composition. In this paper, we propose an approach for web services discovery and composition. The approach relies on (1) SAWSDL, a simple and generic annotation language, (2) an XML representation of a web service that carries both syntactic (e.g., WSDL) and semantic (e.g., SAWSDL) information, and (3) the reuse of available schema matchers. The approach departs from exiting ones because it does not advocate a specific matchmaking algorithm, and it promotes the combination of different schema matchers, allowing multiple discovery and composition strategies.