Despite achieving excellent performance on benchmarks, deep neural networks often underperform in real-world scenarios due to their sensitivity to minor shifts in input data, referred to as distributional shifts. These shifts are common in practical scenarios but are rarely accounted for during evaluations, leading to inflated performance metrics. To address this gap, we propose a novel methodology for the verification, evaluation, and risk assessment of deep learning systems. Our approach explicitly models the incidence of distributional shifts at run time by estimating their probability from the outputs of out-of-distribution detectors. We combine these estimates with conditional probabilities of network correctness, structuring them in a binary tree. By traversing this tree, we can compute reliable and precise estimates of network accuracy. We assess our approach on five datasets, simulating deployment conditions characterized by different frequencies of distributional shift. Our approach consistently outperforms conventional evaluations, with accuracy estimation errors typically ranging between 0.01 and 0.10. We further showcase the potential of our approach on a medical segmentation benchmark, wherein we apply our methods to risk assessment by associating costs with tree nodes, informing cost–benefit analyses and decision-making. Overall, our approach offers a robust framework for improving the reliability and trustworthiness of deep learning systems, particularly in safety-critical applications, by providing more accurate evaluations and actionable risk assessments.

Information disorder is one of the most typical challenges in the current era of science and technology. The amount of information on the internet is increasing, but its correctness and authenticity are not always guaranteed, leading to false information, fake news, etc. The mentioned problem negatively affects users' reception and use of information. Unlike deepfake, cheapfake is created using simple techniques and does not rely on AI to produce fake multimedia. Cheapfake is becoming increasingly popular due to its ease of creation. Thus, there is a growing need to develop techniques that can detect cheapfake content. Following previous events, the Grand Challenge on Detecting Cheapfakes at ACM ICMR 2024 continues to seek contributions from researchers on cheapfake detection with the goal of improving effectiveness and creativity in approach, and understanding the limitations of the current dataset. This challenge has accepted 6 new proposed methods from participants with the highest private test accuracies achieved at 72.2% for Task 1 and 54.84% for Task 2. The highest public test accuracies for the two tasks are 95.6% and 93% respectively. These new methods focus on incorporating new AI models such as Stable Diffusion, LLM. These new findings represent the latest advancements in cheapfake detection research and introduce new potential approaches for future research.

Being able to automatically link social media information and data to remote-sensed data holds large possibilities for society and research. In this paper, we present a system called JORD that is able to autonomously collect social media data about technological and environmental disasters, and link it automatically to remote-sensed data. In addition, we demonstrate that queries in local languages that are relevant to the exact position of natural disasters retrieve more accurate information about a disaster event. To show the capabilities of the system, we present some examples of disaster events detected by the system. To evaluate the quality of the provided information and usefulness of JORD from the potential users point of view we include a crowdsourced user study.

In this paper, we present LireSolr, an open source image retrieval server, build on top of the LIRE library and the Apache Solr search server. With LireSolr, visual information retrieval can be run on a server, which allows better distribution of workloads and simplifies applications in several areas including mobile and web. Furthermore, we showcase several example scenarios how LireSolr can be used to point out the broad range of possibilities and applications. The system is easy to install and setup, and the large number of retrieval tools either provided by LIRE or by other Apache Solr is made easily available on the search server. Moreover, our tool demonstrates how predictions from CNNs can easily be used to extend the visual information retrieval functionality.

Exploring and annotating collections of images without meta-data is a complex task which requires convenient ways of presenting datasets to a user. Visual analytics and information visualization can help users by providing interfaces, and in this paper, we present an open source application that allows users from any domain to use feature-based clustering of large image collections to perform explorative browsing and annotation. For this, we use various image feature extraction mechanisms, different unsupervised clustering algorithms and hierarchical image collection visualization. The performance of the presented open source software allows users to process and display thousands of images at the same time by utilizing heterogeneous resources such as GPUs and different optimization techniques.

Users search for multimedia content with different underlying motivations or intentions. Study of user search intentions is an emerging topic in information retrieval since understanding why a user is searching for a content is crucial for satisfying the user's need. In this paper, we aimed at automatically recognizing a user's intent for image search in the early stage of a search session. We designed seven different search scenarios under the intent conditions of finding items, re-finding items and entertainment. We collected facial expressions, physiological responses, eye gaze and implicit user interactions from 51 participants who performed seven different search tasks on a custom-built image retrieval platform. We analyzed the users' spontaneous and explicit reactions under different intent conditions. Finally, we trained machine learning models to predict users' search intentions from the visual content of the visited images, the user interactions and the spontaneous responses. After fusing the visual and user interaction features, our system achieved the F-1 score of 0.722 for classifying three classes in a user-independent cross-validation. We found that eye gaze and implicit user interactions, including mouse movements and keystrokes are the most informative features. Given that the most promising results are obtained by modalities that can be captured unobtrusively and online, the results demonstrate the feasibility of deploying such methods for improving multimedia retrieval platforms.