This paper presents an updated iteration of the VERGE interactive video retrieval system. It offers various search options like free text and concept-based text search, color similarity, people and face detection, and visual and semantic similarity search. The system is designed to handle large amounts of data efficiently using advanced indexing techniques and state-of-the-art AI technology for visual content analysis. This paper describes enhancements made to improve usability for non-expert users, particularly through changes to the search and browsing interface.

With the evolution of 3D tools, there is now plenty of 3D data for digital applications. This includes 3D retrieval, which seeks to access such data across varied representations such as point clouds, meshes, and multi-view images. However, comprehensive analysis of how to efficiently utilize these representations, or modalities, for retrieval has been missing. This paper evaluates different encodings of each modality in uni-modal retrieval and explores optimal combinations for multimodal retrieval, with state-of-the-art methods from the 3D and image retrieval domains. Results indicate, e.g., that the MuseHash method performs best on mean average precision (MAP), while the CMCL method excels in recall.

Increased traffic on social media platforms, such as X (formerly known as Twitter), is associated with disaster events and consequently fire incidents. Although detecting fires through social media has garnered research interest in recent years, managing the overwhelming volume of daily posts remains challenging. Efficient collection and filtering of fire-related posts are crucial for detecting fires through X. The FireXPosts dataset presented in this article is a collection of posts from Canada and Greece, binarily annotated to aid emergency responders effectively. We train and evaluate uni-modal and bi-modal models for filtering fire-relevant posts and establishing performance baselines on the FireXPosts dataset. Experimental results indicate relatively similar performance between uni-modalities and the best-performing bi-modal models, suggesting that late fusion do not positively influence fire detection on X posts.

Escalating wildfire incidents necessitate improved post-disaster management practices for more effective response and recovery. This study advances the integration of Earth Observation technologies into the wildfire damage assessment phase, contributing a novel approach to augment disaster recovery efforts. Multi-temporal satellite imaging is crucial for monitoring wildfire-affected areas, and the widespread availability of multispectral images with high revisit frequencies substantially improves the comprehensive study of these changes. This paper presents an examination of deep learning techniques for change detection, employing a Siamese convolutional neural network enhanced with an Atrous Spatial Pyramid Pooling block for efficient image data processing. The model is trained and validated on the “Sentinel-2 Wildfire Change Detection Dataset” (S2-WCD), a custom-made dataset aimed at change detection methodologies. By introducing this specialized dataset and applying advanced neural network techniques, the study fills crucial research gaps, offering improvements in wildfire disaster management, particularly in the critical recovery phase following wildfire events.

Social media have evolved into a major source of communication and information sharing, and gradually become impactful in monitoring natural disasters such as wildfires, complementing traditional wildfire monitoring technologies. This paper proposes a comprehensive social media-sensing framework for early wildfire detection, encompassing functionalities like social media crawling, visual analytics, and geolocation, for the analysis of social media posts from the X (former Twitter) platform. Upon analysis, a fire event detection module clusters collected posts into fire events, generating relevant alerts. The framework synergizes with a computer vision algorithm, based on a YOLOv8 architecture, performing object detection on UAV imagery for the detection of individuals in danger in affected areas. The collaborative utilization of social media data and UAV imagery improves situational awareness, by providing information for both the fire incidents and the affected subjects in the area, allowing for a more informed decision making.

Geohazards related to water quality have become critical especially due to the harmful impacts of climate change and human activities. This constitutes the timely, efficient and accurate enough water quality monitoring as a significant component in the emergency management cycle, which can be realised by satellite remote sensing. In this paper, the first image-based benchmark dataset dedicated to mapping muddy waters is presented, named MUDDAT. The dataset is based on Sentinel-2 10m resolution images depicting various muddy water incidents across nine European countries and can essentially be used for semantic segmentation. The annotation procedure is based on an ensemble of three independent methods which reduces biases, namely, NDTI-MNDWI, SID and k-means clustering. Finally, a UNet deep learning model which is considered state-of-the-art in semantic segmentation tasks is trained and fine-tuned. Both qualitative and quantitative results in unseen regions reach high values in classification metrics, constituting MUDDAT a considerable option in water quality monitoring for operational use.

We present the 3D Media Search Engine (3DMSE), which is designed to facilitate the exploration and retrieval of 3D models and images. 3DMSE incorporates unimodal, cross-modal and multimodal retrieval, using any combinations of mesh, point-cloud and multi-image representations. The 3DMSE system is built on the recently proposed MuseHash approach for multimodal representation, and offers a user-friendly web interface that enables formulating queries, presenting search results, and visualising 3D information in an accessible manner.

This paper presents a novel method for supporting multiple modalities in the field of image retrieval, called Multimodal Bayesian Supervised Hashing (MuseHash). The method takes into consideration the semantic information of the training data through the use of Bayesian regression to estimate the semantic probabilities and statistical properties in the retrieval process. MuseHash is an extension of the previously proposed Bayesian ridge-based Semantic Preserving Hashing (BiasHash) method. Experimentation on various domain-specific and benchmark datasets demonstrates that MuseHash outperforms seven existing state-of-the-art methods in image retrieval performance, regardless of the feature extractor type, code length, and visual or textual descriptors used. This highlights the robustness and adaptability of MuseHash, making it a promising solution for multimodal image retrieval.

The availability of satellite images has increased due to the fast development of remote sensing technology. As a result several deep learning change detection methods have been developed to capture spatial changes from multi temporal satellite images that are of great importance in remote sensing, monitoring environmental changes and land use. Recently, a supervised deep learning network called FresUNet has been proposed, which performs a pixel-level change detection from image pairs. In this paper, we extend this method by inserting a Bayesian framework that uses Monte Carlo Dropout, motivated by a recent work in image segmentation. The proposed Bayesian FresUNet (BiasUNet) approach is shown to outperform four state-of-the-art deep learning networks on Sentinel-2 ONERA Satellite Change Detection (OSCD) benchmark dataset, both in terms of precision and quality.

Accurate and timely flood mapping is important in emergency management which can be greatly served by Synthetic Aperture Radar (SAR). Research on SAR flood detection is mostly based on thresholding that has low time complexity and seems ideal for emergency response, although human intervention is needed. Machine learning methods have fewer errors and minimize human intervention but their computational complexity is higher. This work aims to provide a lightweight convolutional neural network baseline for pixel-wise time series flood classification in open land on SAR satellite data. Quantitative and qualitative evaluation of results indicate that the approach is promising.

Recent developments in remote sensing have shown that snow depth can be estimated accurately on a global scale using satellite images through cross-polarization and copolarization backscatter measurements. This method does, however, have some limitations in low-land areas with dense forest coverage and shallow snow, which are often found nearby urban areas. In these areas, citizen observations can be fused with satellite-based estimations to deliver more accurate solutions. To that end, we use snow-related tweets that have been annotated by artificial intelligence (AI) methods and are introduced in a novel neural network model, aiming to increase the estimation accuracy of the state-of-the-art remote sensing method. The proposed model combines the estimated snow depth from Sentinel 1 images with the number of Twitter posts and Twitter images that are semantically relevant to snow. The use of instant social media data for purposes of snow depth estimation is investigated, validated, and tested in Finland. Our results show that this approach does improve the snow depth estimation, highlighting its potential for use in civil protection agencies in managing snow conditions.

The paper presents a method of splitting patent drawings into subimages.For the image based patent retrieval and automatic document understanding it is required to use the individual subimages that are referenced in the text of a patent document.Our method utilizes the fact that subimages have their individual captions inscribed into the compound image.To find the approximate positions of subimages, first the specific captions are localized.Then subimages are found using the empirical rules concerning the relative positions of connected components to the subimage captions.These rules are based on the common sense observation that distances between connected components belonging to the same subimage are smaller than distances between connected components belonging to various subimages and that captions are located close to the corresponding subimages.Alternatively, the image segmentation can be defined as a specific optimization problem, that is aimed on maximizing the gaps between hypothetical subimages while preserving their relations to corresponding captions.The proposed segmentation method can be treated as the approximate solution of this problem.

Effective multimedia retrieval requires the combination of the heterogeneous media contained within multimedia objects and the features that can be extracted from them. To this end, we extend a unifying framework that integrates all well-known weighted, graph-based, and diffusion-based fusion techniques that combine two modalities (textual and visual similarities) to model the fusion of multiple modalities. We also provide a theoretical formula for the optimal number of documents that need to be initially selected, so that the memory cost in the case of multiple modalities remains the same as in the case of two modalities. Experiments using two test collections and three modalities (similarities based on visual descriptors, visual concepts, and textual concepts) indicate improvements in the effectiveness over bimodal fusion under the same memory complexity.

Abstract Environmental and meteorological conditions are of utmost importance for the population, as they are strongly related to the quality of life. Citizens are increasingly aware of this importance. This awareness results in an increasing demand for environmental information tailored to their specific needs and background. We present an environmental information platform that supports submission of user queries related to environmental conditions and orchestrates results from complementary services to generate personalized suggestions. The system discovers and processes reliable data in the Web in order to convert them into knowledge. At runtime, this information is transferred into an ontology‐structured knowledge base, from which then information relevant to the specific user is deduced and communicated in the language of their preference. The platform is demonstrated with real world use cases in the south area of Finland, showing the impact it can have on the quality of everyday life.

Environmental data are considered of utmost importance for human life, since weather conditions, air quality and pollen are strongly related to health issues and affect everyday activities. This paper addresses the problem of discovery of air quality and pollen forecast Web resources, which are usually presented in the form of heatmaps (i.e. graphical representation of matrix data with colors). Towards the solution of this problem, we propose a discovery methodology, which builds upon a general purpose search engine and a novel post processing heatmap recognition layer. The first step involves generation of domain-specific queries, which are submitted to the search engine, while the second involves an image classification step based on visual low level features to identify Web sites including heatmaps. Experimental results comparing various visual features combinations show that relevant environmental sites can be efficiently recognized and retrieved.