This paper presents an updated iteration of the VERGE interactive video retrieval system. It offers various search options like free text and concept-based text search, color similarity, people and face detection, and visual and semantic similarity search. The system is designed to handle large amounts of data efficiently using advanced indexing techniques and state-of-the-art AI technology for visual content analysis. This paper describes enhancements made to improve usability for non-expert users, particularly through changes to the search and browsing interface.

With the evolution of 3D tools, there is now plenty of 3D data for digital applications. This includes 3D retrieval, which seeks to access such data across varied representations such as point clouds, meshes, and multi-view images. However, comprehensive analysis of how to efficiently utilize these representations, or modalities, for retrieval has been missing. This paper evaluates different encodings of each modality in uni-modal retrieval and explores optimal combinations for multimodal retrieval, with state-of-the-art methods from the 3D and image retrieval domains. Results indicate, e.g., that the MuseHash method performs best on mean average precision (MAP), while the CMCL method excels in recall.

Increased traffic on social media platforms, such as X (formerly known as Twitter), is associated with disaster events and consequently fire incidents. Although detecting fires through social media has garnered research interest in recent years, managing the overwhelming volume of daily posts remains challenging. Efficient collection and filtering of fire-related posts are crucial for detecting fires through X. The FireXPosts dataset presented in this article is a collection of posts from Canada and Greece, binarily annotated to aid emergency responders effectively. We train and evaluate uni-modal and bi-modal models for filtering fire-relevant posts and establishing performance baselines on the FireXPosts dataset. Experimental results indicate relatively similar performance between uni-modalities and the best-performing bi-modal models, suggesting that late fusion do not positively influence fire detection on X posts.

Escalating wildfire incidents necessitate improved post-disaster management practices for more effective response and recovery. This study advances the integration of Earth Observation technologies into the wildfire damage assessment phase, contributing a novel approach to augment disaster recovery efforts. Multi-temporal satellite imaging is crucial for monitoring wildfire-affected areas, and the widespread availability of multispectral images with high revisit frequencies substantially improves the comprehensive study of these changes. This paper presents an examination of deep learning techniques for change detection, employing a Siamese convolutional neural network enhanced with an Atrous Spatial Pyramid Pooling block for efficient image data processing. The model is trained and validated on the “Sentinel-2 Wildfire Change Detection Dataset” (S2-WCD), a custom-made dataset aimed at change detection methodologies. By introducing this specialized dataset and applying advanced neural network techniques, the study fills crucial research gaps, offering improvements in wildfire disaster management, particularly in the critical recovery phase following wildfire events.

Social media have evolved into a major source of communication and information sharing, and gradually become impactful in monitoring natural disasters such as wildfires, complementing traditional wildfire monitoring technologies. This paper proposes a comprehensive social media-sensing framework for early wildfire detection, encompassing functionalities like social media crawling, visual analytics, and geolocation, for the analysis of social media posts from the X (former Twitter) platform. Upon analysis, a fire event detection module clusters collected posts into fire events, generating relevant alerts. The framework synergizes with a computer vision algorithm, based on a YOLOv8 architecture, performing object detection on UAV imagery for the detection of individuals in danger in affected areas. The collaborative utilization of social media data and UAV imagery improves situational awareness, by providing information for both the fire incidents and the affected subjects in the area, allowing for a more informed decision making.

Geohazards related to water quality have become critical especially due to the harmful impacts of climate change and human activities. This constitutes the timely, efficient and accurate enough water quality monitoring as a significant component in the emergency management cycle, which can be realised by satellite remote sensing. In this paper, the first image-based benchmark dataset dedicated to mapping muddy waters is presented, named MUDDAT. The dataset is based on Sentinel-2 10m resolution images depicting various muddy water incidents across nine European countries and can essentially be used for semantic segmentation. The annotation procedure is based on an ensemble of three independent methods which reduces biases, namely, NDTI-MNDWI, SID and k-means clustering. Finally, a UNet deep learning model which is considered state-of-the-art in semantic segmentation tasks is trained and fine-tuned. Both qualitative and quantitative results in unseen regions reach high values in classification metrics, constituting MUDDAT a considerable option in water quality monitoring for operational use.

We present the 3D Media Search Engine (3DMSE), which is designed to facilitate the exploration and retrieval of 3D models and images. 3DMSE incorporates unimodal, cross-modal and multimodal retrieval, using any combinations of mesh, point-cloud and multi-image representations. The 3DMSE system is built on the recently proposed MuseHash approach for multimodal representation, and offers a user-friendly web interface that enables formulating queries, presenting search results, and visualising 3D information in an accessible manner.

This paper presents a novel method for supporting multiple modalities in the field of image retrieval, called Multimodal Bayesian Supervised Hashing (MuseHash). The method takes into consideration the semantic information of the training data through the use of Bayesian regression to estimate the semantic probabilities and statistical properties in the retrieval process. MuseHash is an extension of the previously proposed Bayesian ridge-based Semantic Preserving Hashing (BiasHash) method. Experimentation on various domain-specific and benchmark datasets demonstrates that MuseHash outperforms seven existing state-of-the-art methods in image retrieval performance, regardless of the feature extractor type, code length, and visual or textual descriptors used. This highlights the robustness and adaptability of MuseHash, making it a promising solution for multimodal image retrieval.

The availability of satellite images has increased due to the fast development of remote sensing technology. As a result several deep learning change detection methods have been developed to capture spatial changes from multi temporal satellite images that are of great importance in remote sensing, monitoring environmental changes and land use. Recently, a supervised deep learning network called FresUNet has been proposed, which performs a pixel-level change detection from image pairs. In this paper, we extend this method by inserting a Bayesian framework that uses Monte Carlo Dropout, motivated by a recent work in image segmentation. The proposed Bayesian FresUNet (BiasUNet) approach is shown to outperform four state-of-the-art deep learning networks on Sentinel-2 ONERA Satellite Change Detection (OSCD) benchmark dataset, both in terms of precision and quality.

Accurate and timely flood mapping is important in emergency management which can be greatly served by Synthetic Aperture Radar (SAR). Research on SAR flood detection is mostly based on thresholding that has low time complexity and seems ideal for emergency response, although human intervention is needed. Machine learning methods have fewer errors and minimize human intervention but their computational complexity is higher. This work aims to provide a lightweight convolutional neural network baseline for pixel-wise time series flood classification in open land on SAR satellite data. Quantitative and qualitative evaluation of results indicate that the approach is promising.

Social media data have been widely used in disaster management and particularly for the early detection of disaster emergencies. However, apart from sudden crises, there are also creeping crises, which are less evident but can be equally threatening to human lives, such as water pollution. The question raised is whether social media data can be used for discovering issues of water quality. In this work we attempt to answer this question by collecting posts from Twitter during the period of one year, which contain keywords about water quality, and applying three well-known techniques for event detection, i.e. Z-score, STA/LTA, and DBSCAN. A detailed presentation of the detected events, both relevant and not relevant, is given to provide more insight and proves that it is indeed feasible to identify water quality events with social media data. In addition, a quantitative evaluation of the three methods, in terms of precision, shows the superiority of Z-score for this particular topic.

The increase of social media use in recent years has shown potential also for the identification of specific trends in the data that could be used to locate earthquakes. In this work, we implemented a pipeline that uses Twitter data to identify locations of earthquakes and use the information to trigger EO data analysis. We tested the pipeline for almost a year over Japan, an area where earthquake events are frequent, as well as the use of social media in the population. Here, we show the results and discuss the potential development of such procedures. In the future, considering the rapid development and the increase of satellite constellations aimed at global coverage with short revisit times, algorithms of this kind could be used to prioritize satellite acquisitions for the detection of the areas most affected by earthquake damages.

Recent developments in remote sensing have shown that snow depth can be estimated accurately on a global scale using satellite images through cross-polarization and copolarization backscatter measurements. This method does, however, have some limitations in low-land areas with dense forest coverage and shallow snow, which are often found nearby urban areas. In these areas, citizen observations can be fused with satellite-based estimations to deliver more accurate solutions. To that end, we use snow-related tweets that have been annotated by artificial intelligence (AI) methods and are introduced in a novel neural network model, aiming to increase the estimation accuracy of the state-of-the-art remote sensing method. The proposed model combines the estimated snow depth from Sentinel 1 images with the number of Twitter posts and Twitter images that are semantically relevant to snow. The use of instant social media data for purposes of snow depth estimation is investigated, validated, and tested in Finland. Our results show that this approach does improve the snow depth estimation, highlighting its potential for use in civil protection agencies in managing snow conditions.

Artificial Intelligence (AI) is already part of our lives and is extensively entering the space sector to offer value-added Earth Observation (EO) products and services. The Copernicus programme provides data on a free, full and open basis, while the recently launched Data and Information Access Service (DIAS) providers index, store and exchange tremendous amounts of data and cloud infrastructure computational resources. Copernicus data and other georeferenced data sources are often highly heterogeneous, distributed and semantically fragmented. One example is the massively generated social media data from citizen observations, including visual, textual and spatiotemporal information. Social media information offers reliable, timely and very prescriptive information about a crisis event. In this work we present the multimodal fusion aspects for combining satellite images and social media for emergency response, such as flood monitoring and extreme weather conditions in polar regions.

Effective multimedia retrieval requires the combination of the heterogeneous media contained within multimedia objects and the features that can be extracted from them. To this end, we extend a unifying framework that integrates all well-known weighted, graph-based, and diffusion-based fusion techniques that combine two modalities (textual and visual similarities) to model the fusion of multiple modalities. We also provide a theoretical formula for the optimal number of documents that need to be initially selected, so that the memory cost in the case of multiple modalities remains the same as in the case of two modalities. Experiments using two test collections and three modalities (similarities based on visual descriptors, visual concepts, and textual concepts) indicate improvements in the effectiveness over bimodal fusion under the same memory complexity.