Low-light imaging has become a popular topic in image processing, with the quality enhancement of low light images being as a significant challenge, due to the difficulty in retaining colors, patterns, texture and style when generating a normal light image. Our objectives are mainly to firstly better preserve texture regions in image enhancement, while, secondly, preserving colors via color histogram blocks and, finally, to enhance the quality of image through dense denoising blocks. Our proposed novel framework, namely HDD-Unet, is a double Unet based on photorealistic style transfer for low-light image enhancement. The proposed low-light image enhancement method combines color histogram-based fusion, Haar wavelet pooling, dense-denoising blocks and U-net as a backbone architecture to enhance the contrast, reduce noise, and improve the visibility of low light images. Experimental results demonstrate that our proposed method outperforms existing methods in terms of PSNR and SSIM quantitative evaluation metrics, reaching or outperforming state-of-the-art accuracy, but with less resources. We also conduct an ablation study to investigate the impact of our approach on overexposed images, and systematic analysis on the late fusion weighting parameters. Multiple experiments were conducted with artificial noise inserted to accomplish more efficient comparison. The results show that the proposed framework enhances accurately images with various gamma corrections. The proposed method represents a significant advance in the field of low light image enhancement and has the potential to address several challenges associated with low light imaging. • Low-light image enhancement based on a combination of two U-Net branches with wavelet transformations, Adaptive Instance Normalization, color histogram blocks and denoising dense blocks. • Analytic framework creation for the HDD-Unet architecture. • Evaluation of the proposed method in a qualitative and a quantitative way.

Object localization is a fundamental task in computer vision that traditionally requires labeled datasets for accurate results. Recent progress in self-supervised learning has enabled unsupervised object localization, reducing reliance on manual annotations. Unlike supervised encoders, which depend on annotated training data, self-supervised encoders learn semantic representations directly from large collections of unlabeled images. This makes them the natural foundation for unsupervised object localization, as they capture object-relevant features while eliminating the need for costly manual labels. These encoders produce semantically coherent patch embeddings. Grouping these embeddings reveals sets of patches that correspond to objects in an image. These patch sets can be converted into object masks or bounding boxes, enabling tasks such as single-object discovery, multi-object detection, and instance segmentation. By applying off-line mask clustering or using pre-trained vision-language models, unsupervised localization methods can assign semantic labels to discovered objects. This transforms initially class-agnostic objects (objects without class labels) into class-aware ones (objects with class labels), aligning these tasks with their supervised counterparts. This paper provides a structured review of unsupervised object localization methods in both class-agnostic and class-aware settings. In contrast, previous surveys have focused only on class-agnostic localization. We discuss state-of-the-art object discovery strategies based on self-supervised features and provide a detailed comparison of experimental results across a wide range of tasks, datasets, and evaluation metrics.

Abstract Infrastructure as Code (IaC) uses versionable software code to define, deploy, and configure physical computational resources, software execution platforms, and applications. As a result, IaC enables the scalable management of complex computing environments while preventing environment drift. IaC frameworks typically offer specific languages such as the industrial Terraform, Ansible, Chef, or TOSCA—standing for Topology and Orchestration Specification for Cloud Applications—the OASIS (Organization for the Advancement of Structured Information Standards) open standard approach to IaC. Developing high-quality IaC for deploying and managing applications demands expertise and knowledge in specific IaC languages, infrastructure resources, resource providers, quality issues in IaC scripts, and so on. While several model-driven engineering (MDE) approaches have been proposed to simplify IaC development, they cannot capture and use expert knowledge to assist with modeling tasks and MDE processes by providing interactive recommendations. This paper presents a knowledge-based framework for guiding the model-driven development of IaC. We use TOSCA as the target IaC language as it is an open standard. We enable IaC and resource experts to share their IaC and resource-related knowledge with application operational experts to help simplify the development of application deployment models. We use an ontology to record the relevant deployment knowledge and ontology reasoning to implement modeling guidance capabilities such as TOSCA model auto-completion, code smell and error detection, and model element matchmaking. We show the flexibility of our methodology by applying it to three industrial applications, covering cloud, edge, and HPC (High-Performance Computing) domains. Moreover, we also assess the use acceptance of our approach and framework by conducting controlled experiments with expert and non-expert IaC users. The results indicate that our method can simplify IaC development by providing appropriate recommendations.

ABSTRACT Environmental acoustics, particularly urban soundscape monitoring, has gained increasing consideration since the United Nations established the Sustainable Development Goals in 2015, and to an even greater extent with the rise of privacy concerns following the introduction of global regulations such as GDPR. As a result, privacy‐compliant devices have become essential for soundscape monitoring in urban environments. In this paper, we present the design and implementation of a portable AI‐driven, privacy‐compliant urban sound recording device that locally captures and processes acoustic data on the edge. In more detail, this device operates as a sensor that captures soundscapes and processes them through a pipeline, which employs a pre‐trained open‐source AI model to anonymize human voices, ensuring privacy without compromising the integrity of the acoustic environment. The anonymization process alters human speech in a way that protects identity while maintaining environmental audio quality. The device can function as a standalone sensor or as part of a synchronized network of distributed sensors. Privacy‐focused evaluation of the device's recordings indicates that, while the anonymization process impacts speech intelligibility, it preserves the overall soundscape with a recall rate of 96%. The system was deployed in a real‐world setting with four temporally synchronized sensors. While network synchronization was achieved, a 1 to 2‐s deviation was occasionally observed in the first duty cycle interval, reflecting timing variability inherent to Cron‐based script triggering. This limitation has been identified for future refinement. This work demonstrates the feasibility of deploying privacy‐compliant, edge‐based soundscape sensors in urban environments, contributing to privacy preservation, and enhanced public safety.

Existing approaches for disfluency detection typically require the existence of large annotated datasets. However, current datasets for this task are limited, suffer from class imbalance, and lack some types of disfluencies that are encountered in real-world scenarios. At the same time, augmentation techniques for disfluency detection are not able to model complex types of disfluencies. This limits such approaches to only performing pre-training since the generated data are not indicative of disfluencies that occur in real scenarios and, as a result, cannot be directly used for training disfluency detection models, as we experimentally demonstrate. This imposes significant constraints on the usefulness of such approaches in practice since real disfluencies still have to be collected in order to train the models. In this work, we propose Large-scale ARtificial Disfluency Generation (LARD), a method for automatically generating artificial disfluencies, and more specifically repairs, from fluent text. Unlike existing augmentation techniques, LARD can simulate all the different and complex types of disfluencies. In addition, it incorporates contextual embeddings into the disfluency generation to produce realistic, context-aware artificial disfluencies. LARD can be used effectively for training disfluency detection models, bypassing the requirement of annotated disfluent data. Our empirical evaluation shows that LARD outperforms existing rule-based augmentation methods and increases the accuracy of existing disfluency detectors. In addition, experiments demonstrate that the proposed method can be effectively used in a low-resource setup.

Abstract The aim of this work is to provide a systematic literature review of techniques for taxonomy generation across the cybersecurity domain. Cybersecurity taxonomies can be classified into manual and dynamic, each one of which focuses on different characteristics and tails different goals. Under this premise, we investigate the current state of the art in both categories with respect to their characteristics, applications and methods. To this end, we perform a systematic literature review in accordance with an extensive analysis of the tremendous need for dynamic taxonomies in the cybersecurity landscape. This analysis provides key insights into the advantages and limitations of both techniques, and it discusses the datasets which are most commonly used to generate cybersecurity taxonomies.

Maintaining runway infrastructure is vital for air transport safety, with defects like cracks and tire marks posing significant risks to take-off and landing. Researchers have proposed various methods for automatic surface defect detection using computer vision and machine learning. However, they often require explicitly annotated datasets that demand significant workload and field expertise. Additionally, detection outcomes usually follow the low-level training labels scheme to describe the detected defects, requiring post-processing for high-level semantic information extraction such as damage severity. In this work, we present a novel method for defect detection and damage severity estimation on runway surfaces, leveraging the Contrastive Language-Image Pre-training (CLIP) architecture for imagetext pairing. Our model processes runway images and attaches text descriptions mentioning detected defects and severity level, identifying three defect types (crack, joint, and tire mark) and categorizing damage severity into three levels (low, medium, and high). Utilizing natural language annotations simplifies the labeling process, eliminating the need for labor-intensive lowlevel image-based annotations. The model exploits the natural language labels for direct estimation of damage severity and delivers high-level semantic information to the end-user as text, providing a comprehensive runway condition assessment tool. The proposed method demonstrates high performance across various test sets, posing a valuable human-centric approach for efficient defect detection and damage estimation on runway surfaces.

This paper presents an updated iteration of the VERGE interactive video retrieval system. It offers various search options like free text and concept-based text search, color similarity, people and face detection, and visual and semantic similarity search. The system is designed to handle large amounts of data efficiently using advanced indexing techniques and state-of-the-art AI technology for visual content analysis. This paper describes enhancements made to improve usability for non-expert users, particularly through changes to the search and browsing interface.

With the evolution of 3D tools, there is now plenty of 3D data for digital applications. This includes 3D retrieval, which seeks to access such data across varied representations such as point clouds, meshes, and multi-view images. However, comprehensive analysis of how to efficiently utilize these representations, or modalities, for retrieval has been missing. This paper evaluates different encodings of each modality in uni-modal retrieval and explores optimal combinations for multimodal retrieval, with state-of-the-art methods from the 3D and image retrieval domains. Results indicate, e.g., that the MuseHash method performs best on mean average precision (MAP), while the CMCL method excels in recall.

Increased traffic on social media platforms, such as X (formerly known as Twitter), is associated with disaster events and consequently fire incidents. Although detecting fires through social media has garnered research interest in recent years, managing the overwhelming volume of daily posts remains challenging. Efficient collection and filtering of fire-related posts are crucial for detecting fires through X. The FireXPosts dataset presented in this article is a collection of posts from Canada and Greece, binarily annotated to aid emergency responders effectively. We train and evaluate uni-modal and bi-modal models for filtering fire-relevant posts and establishing performance baselines on the FireXPosts dataset. Experimental results indicate relatively similar performance between uni-modalities and the best-performing bi-modal models, suggesting that late fusion do not positively influence fire detection on X posts.

Escalating wildfire incidents necessitate improved post-disaster management practices for more effective response and recovery. This study advances the integration of Earth Observation technologies into the wildfire damage assessment phase, contributing a novel approach to augment disaster recovery efforts. Multi-temporal satellite imaging is crucial for monitoring wildfire-affected areas, and the widespread availability of multispectral images with high revisit frequencies substantially improves the comprehensive study of these changes. This paper presents an examination of deep learning techniques for change detection, employing a Siamese convolutional neural network enhanced with an Atrous Spatial Pyramid Pooling block for efficient image data processing. The model is trained and validated on the “Sentinel-2 Wildfire Change Detection Dataset” (S2-WCD), a custom-made dataset aimed at change detection methodologies. By introducing this specialized dataset and applying advanced neural network techniques, the study fills crucial research gaps, offering improvements in wildfire disaster management, particularly in the critical recovery phase following wildfire events.

Vision-based monitoring methods have been actively studied in the construction industry as they can automatically generate information related to progress, productivity, and safety. 3D reconstruction is key in such monitoring techniques, allowing the inference of job-site context, the creation of digital counterparts of physical spaces, and the comparisons between asdesigned and as-built conditions. However, 3D applications in construction currently produce large volumes of unstructured data and unusable point clouds, which are time-consuming to convert into an interactive environment for Building Information Modelling (BIM) or Digital Twins. While radiance field rendering methods are increasingly gaining traction, the adoption of generated Neural Radiance Fields or Gaussian Splatting models by digital construction technology is still tentative. This study introduces a framework that uses Neural Radiance Fields (NeRFs) to improve 3D inspection and maintenance on construction sites. It merges NeRF's high-resolution, real-time 3D modelling with an interactive platform, facilitating detailed remote site analysis and defect detection. The framework incorporates a custom version of TurboNeRF, tailored specifically for construction site inspections. Through this paper, we aim to highlight the potential of combining 3D imaging technology, the use of drone imagery and ontological models to improve construction site management practices.

Social media have evolved into a major source of communication and information sharing, and gradually become impactful in monitoring natural disasters such as wildfires, complementing traditional wildfire monitoring technologies. This paper proposes a comprehensive social media-sensing framework for early wildfire detection, encompassing functionalities like social media crawling, visual analytics, and geolocation, for the analysis of social media posts from the X (former Twitter) platform. Upon analysis, a fire event detection module clusters collected posts into fire events, generating relevant alerts. The framework synergizes with a computer vision algorithm, based on a YOLOv8 architecture, performing object detection on UAV imagery for the detection of individuals in danger in affected areas. The collaborative utilization of social media data and UAV imagery improves situational awareness, by providing information for both the fire incidents and the affected subjects in the area, allowing for a more informed decision making.

Geohazards related to water quality have become critical especially due to the harmful impacts of climate change and human activities. This constitutes the timely, efficient and accurate enough water quality monitoring as a significant component in the emergency management cycle, which can be realised by satellite remote sensing. In this paper, the first image-based benchmark dataset dedicated to mapping muddy waters is presented, named MUDDAT. The dataset is based on Sentinel-2 10m resolution images depicting various muddy water incidents across nine European countries and can essentially be used for semantic segmentation. The annotation procedure is based on an ensemble of three independent methods which reduces biases, namely, NDTI-MNDWI, SID and k-means clustering. Finally, a UNet deep learning model which is considered state-of-the-art in semantic segmentation tasks is trained and fine-tuned. Both qualitative and quantitative results in unseen regions reach high values in classification metrics, constituting MUDDAT a considerable option in water quality monitoring for operational use.

This tutorial provides an overview of multimedia retrieval in the context of eXtended Reality (XR), including using virtual and augmented/mixed reality as a user interface for multimedia retrieval, as well as multimedia search tasks addressing content needs for the creation of XR experiences.It will discuss the opportunities and limitations of XR-based search, the evaluation of XR-based multimedia retrieval systems, the demonstration of selected research systems, and open research challenges.

We present the 3D Media Search Engine (3DMSE), which is designed to facilitate the exploration and retrieval of 3D models and images. 3DMSE incorporates unimodal, cross-modal and multimodal retrieval, using any combinations of mesh, point-cloud and multi-image representations. The 3DMSE system is built on the recently proposed MuseHash approach for multimodal representation, and offers a user-friendly web interface that enables formulating queries, presenting search results, and visualising 3D information in an accessible manner.

We face up to the challenge of explainability in Multimodal Artificial Intelligence (MMAI). At the nexus of neuroscience-inspired and quantum computing, interpretable and transparent spin-geometrical neural architectures for early fusion of large-scale, heterogeneous, graph-structured data are envisioned, harnessing recent evidence for relativistic quantum neural coding of (co-)behavioral states in the self-organizing brain, under competitive, multidimensional dynamics. The designs draw on a self-dual classical description – via special Clifford-Lipschitz operations – of spinorial quantum states within registers of at most 16 qubits for efficient encoding of exponentially large neural structures. Formally ‘trained’, Lorentz neural architectures with precisely one lateral layer of exclusively inhibitory interneurons accounting for anti-modalities, as well as their co-architectures with intra-layer connections are highlighted. The approach accommodates the fusion of up to 16 time-invariant interconnected (anti-)modalities and the crystallization of latent multidimensional patterns. Comprehensive insights are expected to be gained through applications to Multimodal Big Data, under diverse real-world scenarios.

Object detection appears to be omnipresent nowadays with detectors being available for every problem available, covering solutions from extra-light to ultra resource demanding models. Yet, the vast majority of these approaches are based on large datasets to provide the required feature diversity. This work focuses on object detection solutions which do not rely heavily on abundant training datasets but rather on medium-sized data collections. It uses Efficientdet object detector as base for the application of novel modifications which achieve better performance both in efficiency as well in effectiveness. The focus on medium-sized datasets aim at representing more commonplace datasets which can be accumulated and compiled with relative ease.

This paper presents a novel method for supporting multiple modalities in the field of image retrieval, called Multimodal Bayesian Supervised Hashing (MuseHash). The method takes into consideration the semantic information of the training data through the use of Bayesian regression to estimate the semantic probabilities and statistical properties in the retrieval process. MuseHash is an extension of the previously proposed Bayesian ridge-based Semantic Preserving Hashing (BiasHash) method. Experimentation on various domain-specific and benchmark datasets demonstrates that MuseHash outperforms seven existing state-of-the-art methods in image retrieval performance, regardless of the feature extractor type, code length, and visual or textual descriptors used. This highlights the robustness and adaptability of MuseHash, making it a promising solution for multimodal image retrieval.

Web bots are vital for the web as they can be used to automate several actions, some of which would have otherwise been impossible or very time consuming. These actions can be benign, such as website testing and web indexing, or malicious, such as unauthorised content scraping, scalping, vulnerability scanning, and more. To detect malicious web bots, recent approaches examine the visitors’ fingerprint and behaviour. For the latter, several values (i.e., features) are usually extracted from visitors’ web logs and used as input to train machine learning models. In this research we show that web bots can use recent advances in machine learning, and, more specifically, Reinforcement Learning (RL), to effectively evade behaviour-based detection techniques. To evaluate these evasive bots, we examine (i) how well they can evade a pre-trained bot detection framework, (ii) how well they can still evade detection after the detection framework is re-trained on new behaviours generated from the evasive web bots, and (iii) how bots perform if re-trained again on the re-trained detection framework. We show that web bots can repeatedly evade detection and adapt to the re-trained detection framework to showcase the importance of considering such types of bots when designing web bot detection frameworks.

The identification of key actors in complex networks has gathered significant interest by virtue of their importance in modern applications. Several of the existing methods employ standard centrality measures to achieve their goal and as a result, one of the main challenges is identifying key actor nodes with high relevance across all such measures. In this work, we propose a model based on the use of graph convolutional networks (GeNs) that retrieves the key actors in a network based on a centrality measure ranking aggregation scheme. We experimentally demonstrate the effectiveness of our solution compared to baseline and state-of-the-art approaches in terms of: i) accuracy, ii) performance compared to standard machine learning approaches, and iii) influence propagation capabilities.

This paper focuses on the determination of the evoked sentiments to people by observing outdoor and indoor spaces, aiming to create a tool for designers and architects that can be utilized for sophisticated designs. Since sentiment is subjective, the design process can be facilitated by an ancillary automated tool for sentiment extraction. Simultaneously, a dataset containing both real and virtual images of vacant architectural spaces is introduced, while the SUN attributes are also extracted from the images in order to be included throughout training. The dataset is annotated towards both valence and arousal, while five established and two custom architectures, one which has never been used before in classifying abstract concepts, are evaluated on the collected data.

In the construction domain, Digital twins are mostly used for facilities management of buildings, but their applications are still very limited. The virtualization of buildings and bridges in the last 15 years in the form of Building or Bridge Information Models is clearly identified as the starting point for the DTs. The industry has erected a frame with semantically rich 3D reference models that are now heavily enriched with visual sensor data captured on construction sites. This article provides an overview of the research and current practices of computer vision methods in the construction industry and presents typical examples of their applications for 3D reconstruction, safety management and structural monitoring for quality assurance. It then highlights the dominant achievements presented in the literature and concludes with the challenges and research directions applicable to digital twins that need to be addressed and exploited in the future.

The availability of satellite images has increased due to the fast development of remote sensing technology. As a result several deep learning change detection methods have been developed to capture spatial changes from multi temporal satellite images that are of great importance in remote sensing, monitoring environmental changes and land use. Recently, a supervised deep learning network called FresUNet has been proposed, which performs a pixel-level change detection from image pairs. In this paper, we extend this method by inserting a Bayesian framework that uses Monte Carlo Dropout, motivated by a recent work in image segmentation. The proposed Bayesian FresUNet (BiasUNet) approach is shown to outperform four state-of-the-art deep learning networks on Sentinel-2 ONERA Satellite Change Detection (OSCD) benchmark dataset, both in terms of precision and quality.

Accurate and timely flood mapping is important in emergency management which can be greatly served by Synthetic Aperture Radar (SAR). Research on SAR flood detection is mostly based on thresholding that has low time complexity and seems ideal for emergency response, although human intervention is needed. Machine learning methods have fewer errors and minimize human intervention but their computational complexity is higher. This work aims to provide a lightweight convolutional neural network baseline for pixel-wise time series flood classification in open land on SAR satellite data. Quantitative and qualitative evaluation of results indicate that the approach is promising.

Color comprises one of the most significant and dominant cues for various applications. As one of the most noticeable and stable attributes of vehicles, color can constitute a valuable key component in several practices of intelligent surveillance systems. In this paper, we propose a deep-learning-based framework that combines semantic segmentation masks with pixels clustering for automatic vehicle color recognition. Different from conventional methods, which usually consider only the features of the vehicle's front side, the proposed algorithm is able for view-independent color identification, which is more effective for the surveillance tasks. To the best of our knowledge, this is the first work that employs semantic segmentation masks along with color clustering for the extraction of the vehicle's color representative parts and the recognition of the dominant color, respectively. To evaluate the performance of the proposed method, we introduce a challenging multi-view dataset of 500 car-related RGB images extending the publicly available DSMLR Car Parts dataset for vehicle parts segmentation. The experiments demonstrate that the proposed approach achieves excellent performance and accurate results reaching an accuracy of 93.06% in the multi-view scenario. To facilitate further research, the evaluation dataset and the pre-trained models will be released at https://github.com/klearchos-stav/vehicle_color_recognition.

Social media data have been widely used in disaster management and particularly for the early detection of disaster emergencies. However, apart from sudden crises, there are also creeping crises, which are less evident but can be equally threatening to human lives, such as water pollution. The question raised is whether social media data can be used for discovering issues of water quality. In this work we attempt to answer this question by collecting posts from Twitter during the period of one year, which contain keywords about water quality, and applying three well-known techniques for event detection, i.e. Z-score, STA/LTA, and DBSCAN. A detailed presentation of the detected events, both relevant and not relevant, is given to provide more insight and proves that it is indeed feasible to identify water quality events with social media data. In addition, a quantitative evaluation of the three methods, in terms of precision, shows the superiority of Z-score for this particular topic.

Social media and web sites provide an access to pub-lic opinions on certain aspects and therefore play an important role in getting insights on targeted audiences. Designers have been investigating how to use them for grasping social feelings and needs associated with the arrangement of spaces that surround people in everyday life to find inspiration and come up with ideas for adaptive designs. Following this, we propose a novel design-oriented tool-set that retrieves and analyses online public information from Twitter and focused content from relevant web sites. We present the data collection pipeline and multilingual analysis algorithms like concept extraction and sentiment analysis on interior and urban design. Finally, we showcase an application of the proposed tool-set within two case studies.

Disfluency detection is a critical task in real-time dialogue systems. However, despite its importance, it remains a relatively unexplored field, mainly due to the lack of appropriate datasets. At the same time, existing datasets suffer from various issues, including class imbalance issues, which can significantly affect the performance of the model on rare classes, as it is demonstrated in this paper. To this end, we propose LARD, a method for generating complex and realistic artificial disfluencies with little effort. The proposed method can handle three of the most common types of disfluencies: repetitions, replacements, and restarts. In addition, we release a new large-scale dataset with disfluencies that can be used on four different tasks: disfluency detection, classification, extraction, and correction. Experimental results on the LARD dataset demonstrate that the data produced by the proposed method can be effectively used for detecting and removing disfluencies, while also addressing limitations of existing datasets.

Unexploded Ordnance (UXO) classification is a challenging task which is currently tackled using electromagnetic induction devices that are expensive and may require physical presence in potentially hazardous environments. The limited availability of open UXO data has, until now, impeded the progress of image-based UXO classification, which may offer a safe alternative at a reduced cost. In addition, the existing sporadic efforts focus mainly on small scale experiments using only a subset of common UXO categories. Our work aims to stimulate research interest in image-based UXO classification, with the curation of a novel dataset that consists of over 10000 annotated images from eight major UXO categories. Through extensive experimentation with supervised deep learning we uncover key insights into the challenging aspects of this task. Finally, we set the baseline on our novel benchmark by training state-of-the-art Convolutional Neural Networks and a Vision Transformer that are able to discriminate between highly overlapping UXO categories with 84.33% accuracy.

The increase of social media use in recent years has shown potential also for the identification of specific trends in the data that could be used to locate earthquakes. In this work, we implemented a pipeline that uses Twitter data to identify locations of earthquakes and use the information to trigger EO data analysis. We tested the pipeline for almost a year over Japan, an area where earthquake events are frequent, as well as the use of social media in the population. Here, we show the results and discuss the potential development of such procedures. In the future, considering the rapid development and the increase of satellite constellations aimed at global coverage with short revisit times, algorithms of this kind could be used to prioritize satellite acquisitions for the detection of the areas most affected by earthquake damages.

Recent developments in remote sensing have shown that snow depth can be estimated accurately on a global scale using satellite images through cross-polarization and copolarization backscatter measurements. This method does, however, have some limitations in low-land areas with dense forest coverage and shallow snow, which are often found nearby urban areas. In these areas, citizen observations can be fused with satellite-based estimations to deliver more accurate solutions. To that end, we use snow-related tweets that have been annotated by artificial intelligence (AI) methods and are introduced in a novel neural network model, aiming to increase the estimation accuracy of the state-of-the-art remote sensing method. The proposed model combines the estimated snow depth from Sentinel 1 images with the number of Twitter posts and Twitter images that are semantically relevant to snow. The use of instant social media data for purposes of snow depth estimation is investigated, validated, and tested in Finland. Our results show that this approach does improve the snow depth estimation, highlighting its potential for use in civil protection agencies in managing snow conditions.

Surveillance systems currently deploy a variety of devices that can capture visual content (such as CCTV, body-worn cameras, and smartphone cameras), thus rendering the monitoring of video footage obtained from multiple such devices a complex task. This becomes especially challenging when monitoring social events that involve large crowds, particularly when there is a risk of crowd violence. This paper presents and demonstrates a crowd violence detection system that can process, analyze, and alert potential stakeholders, when violence-related content is identified in crowd-based video footage. Based on deep neural networks, the proposed end-to-end framework utilizes a 3D Convolutional Neural Network (CNN) to deal with the (near) real-time analysis of video streams and video files for crowd violence detection. The framework is trained, evaluated, and demonstrated using the Violent Flows dataset, a dataset related to crowd violence that is widely used for research. The presented framework is provided as a standalone application for desktop environments and can analyze both video streams and video files.

Artificial Intelligence (AI) is already part of our lives and is extensively entering the space sector to offer value-added Earth Observation (EO) products and services. The Copernicus programme provides data on a free, full and open basis, while the recently launched Data and Information Access Service (DIAS) providers index, store and exchange tremendous amounts of data and cloud infrastructure computational resources. Copernicus data and other georeferenced data sources are often highly heterogeneous, distributed and semantically fragmented. One example is the massively generated social media data from citizen observations, including visual, textual and spatiotemporal information. Social media information offers reliable, timely and very prescriptive information about a crisis event. In this work we present the multimodal fusion aspects for combining satellite images and social media for emergency response, such as flood monitoring and extreme weather conditions in polar regions.

This work presents a spatio-temporal activity detection and recognition framework for untrimmed surveillance videos consisting of a three-step pipeline: object detection, tracking, and activity recognition. The framework relies on the YOLO v4 architecture for object detection, Euclidean distance for tracking, while the activity recognizer uses a 3D Convolutional Deep learning architecture employing spatio-temporal boundaries and addressing it as multi-label classification. The evaluation experiments on the VIRAT dataset achieve accurate detections of the temporal boundaries and recognitions of activities in untrimmed videos, with better performance for the multi-label compared to the multi-class activity recognition.

Despite the fact that automatic content analysis has made remarkable progress over the last decade - mainly due to significant advances in machine learning - interactive video retrieval is still a very challenging problem, with an increasing relevance in practical applications. The Video Browser Showdown (VBS) is an annual evaluation competition that pushes the limits of interactive video retrieval with state-of-the-art tools, tasks, data, and evaluation metrics. In this paper, we analyse the results and outcome of the 8th iteration of the VBS in detail. We first give an overview of the novel and considerably larger V3C1 dataset and the tasks that were performed during VBS 2019. We then go on to describe the search systems of the six international teams in terms of features and performance. And finally, we perform an in-depth analysis of the per-team success ratio and relate this to the search strategies that were applied, the most popular features, and problems that were experienced. A large part of this analysis was conducted based on logs that were collected during the competition itself. This analysis gives further insights into the typical search behavior and differences between expert and novice users. Our evaluation shows that textual search and content browsing are the most important aspects in terms of logged user interactions. Furthermore, we observe a trend towards deep learning based features, especially in the form of labels generated by artificial neural networks. But nevertheless, for some tasks, very specific content-based search features are still being used. We expect these findings to contribute to future improvements of interactive video search systems.

Comprehensive and fair performance evaluation of information retrieval systems represents an essential task for the current information age. Whereas Cranfield-based evaluations with benchmark datasets support development of retrieval models, significant evaluation efforts are required also for user-oriented systems that try to boost performance with an interactive search approach. This article presents findings from the 9th Video Browser Showdown, a competition that focuses on a legitimate comparison of interactive search systems designed for challenging known-item search tasks over a large video collection. During previous installments of the competition, the interactive nature of participating systems was a key feature to satisfy known-item search needs, and this article continues to support this hypothesis. Despite the fact that top-performing systems integrate the most recent deep learning models into their retrieval process, interactive searching remains a necessary component of successful strategies for known-item search tasks. Alongside the description of competition settings, evaluated tasks, participating teams, and overall results, this article presents a detailed analysis of query logs collected by the top three performing systems, SOMHunter, VIRET, and vitrivr. The analysis provides a quantitative insight to the observed performance of the systems and constitutes a new baseline methodology for future events. The results reveal that the top two systems mostly relied on temporal queries before a correct frame was identified. An interaction log analysis complements the result log findings and points to the importance of result set and video browsing approaches. Finally, various outlooks are discussed in order to improve the Video Browser Showdown challenge in the future.

The main goal of this document is to describe the implementation of each component of the colMOOC platform and the integration stage of the operational platform as a whole. Additionally, a first simple operational scenario that has been implemented is also being described, in order to demonstrate the interoperability of the platform. Finally, this paper presents a positive technical assessment of the 1st version of colMOOC platform through performance benchmarking experiments performed on system's components in order to verify its correct operation and interoperability of the different integrated modules to meet the requirements of the use cases for the end users.

Recently, research endeavors have shown the potentiality of Cycle-Consistent Adversarial Networks (CycleGAN) in style transfer. In Cycle-Consistent Adversarial Networks, the consistency loss is introduced to measure the difference between the original images and the reconstructed in both directions, forward and backward. In this work, the combination of Cycle-Consistent Adversarial Networks with Fast and Adaptive Bidimensional Empirical Mode Decomposition (FABEMD) is proposed to perform style transfer on images. In the proposed approach the cycle-consistency loss is modified to include the differences between the extracted Intrinsic Mode Functions (BIMFs) images. Instead of an estimation of pixel-to-pixel difference between the produced and input images, the FABEMD is applied and the extracted BIMFs are involved in the computation of the total cycle loss. This method enriches the computation of the total loss in a content-to-content and style-to-style comparison by connecting the spatial information to the frequency components. The experimental results reveal that the proposed method is efficient and produces qualitative results comparable to state-of-the-art methods.

This works focuses on examining the performance of the Single Shot Detector (SSD) model in resource restricted systems where maintaining the power of the full model comprises a significant prerequisite. The proposed SSD variations examine the behavior of lighter versions of SSD while propose measures to limit the unavoidable performance shortage. The outcomes of the conducted research demonstrate a remarkable trade-off between performance losses, speed improvement and the required resource reservation. Thus, the experimental results evidence the efficiency of the presented SSD alterations towards accomplishing higher frame rates and retaining the performance of the original model.

This work examines violence detection in video scenes of crowds and proposes a crowd violence detection framework based on a 3D convolutional deep learning architecture, the 3D-ResNet model with 50 layers. The proposed framework is evaluated on the Violent Flows dataset against several state-of-the-art approaches and achieves higher accuracy values in almost all cases, while also performing the violence detection activities in (near) real-time.

Abstract Dialogue‐based systems often consist of several components, such as communication analysis, dialogue management, domain reasoning, and language generation. In this paper, we present Converness, an ontology‐driven, rule‐based framework to facilitate domain reasoning for conversational awareness in multimodal dialogue‐based agents. Converness uses Web Ontology Language 2 (OWL 2) ontologies to capture and combine the conversational modalities of the domain, for example, deictic gestures and spoken utterances, fuelling conversational topic understanding, and interpretation using description logics and rules. At the same time, defeasible rules are used to couple domain and user‐centred knowledge to further assist the interaction with end users, facilitating advanced conflict resolution and personalised context disambiguation. We illustrate the capabilities of the framework through its integration into a multimodal dialogue‐based agent that serves as an intelligent interface between users (elderly, caregivers, and health experts) and an ambient assistive living platform in real home settings.

Automated programs (bots) are responsible for a large percentage of website traffic. These bots can either be used for benign purposes, such as Web indexing, Website monitoring (validation of hyperlinks and HTML code), feed fetching Web content and data extraction for commercial use or for malicious ones, including, but not limited to, content scraping, vulnerability scanning, account takeover, distributed denial of service attacks, marketing fraud, carding and spam. To ensure their security, Web servers try to identify bot sessions and apply special rules to them, such as throttling their requests or delivering different content. The methods currently used for the identification of bots are based either purely on rule-based bot detection techniques or a combination of rule-based and machine learning techniques. While current research has developed highly adequate methods for Web bot detection, these methods' adequacy when faced with Web bots that try to remain undetected hasn't been studied. For this reason, we created and evaluated a Web bot detection framework on its ability to detect conspicuous bots separately from its ability to detect advanced Web bots. We assessed the proposed framework performance using real HTTP traffic from a public Web server. Our experimental results show that the proposed framework has significant ability to detect Web bots that do not try to hide their bot identity using HTTP Web logs (balanced accuracy in a false-positive intolerant server > 95%). However, detecting advanced Web bots that present a browser fingerprint and may present a humanlike behaviour as well is considerably more difficult.

In this paper, we present work in progress on the development of an intelligent interconnected infrastructure for public security and protection domain. In contrast to other Internet of Things (IoT) frameworks, the proposed system aims to effectively combine device and human awareness to achieve situational awareness, so as to provide a protection and security environment for citizens. The emphasis is placed on tourists, by creating the appropriate infrastructure to address a set of urgent situations, such as health-related problems and missing children in overcrowded environments, supporting smart links between humans and entities on the basis of goals, and adapting device operation to comply with human objectives, profiles and privacy. The framework effectively combines state-of-the-art technologies on IoT data collection and analytics, knowledge representation and interoperability, crowdsourcing, data fusion and decision-making.

This work summarizes the findings of the 7th iteration of the Video Browser Showdown (VBS) competition organized as a workshop at the 24th International Conference on Multimedia Modeling in Bangkok. The competition focuses on video retrieval scenarios in which the searched scenes were either previously observed or described by another person (i.e., an example shot is not available). During the event, nine teams competed with their video retrieval tools in providing access to a shared video collection with 600 hours of video content. Evaluation objectives, rules, scoring, tasks, and all participating tools are described in the article. In addition, we provide some insights into how the different teams interacted with their video browsers, which was made possible by a novel interaction logging mechanism introduced for this iteration of the VBS. The results collected at the VBS evaluation server confirm that searching for one particular scene in the collection when given a limited time is still a challenging task for many of the approaches that were showcased during the event. Given only a short textual description, finding the correct scene is even harder. In ad hoc search with multiple relevant scenes, the tools were mostly able to find at least one scene, whereas recall was the issue for many teams. The logs also reveal that even though recent exciting advances in machine learning narrow the classical semantic gap problem, user-centric interfaces are still required to mediate access to specific content. Finally, open challenges and lessons learned are presented for future VBS events.

A novel work for Ambient Assisted Living applications is presented here. More specifically, this paper focuses on activity recognition from recordings of daily living captured by wearable cameras. It constructs a discriminant object centric motion descriptor for representing the micro-actions within the viewpoint of the action maker so as to later define the activity that he/she performs. The accumulation of these activities build patterns over time that can be used to study the behavior of the end-users, which is very useful for health application and monitoring of patients inside their own dwellings or their behavior inside a controlled environment.

Oil spills pose a major threat of the oceanic and coastal environments, hence, an automatic detection and a continuous monitoring system comprises an appealing option for minimizing the response time of relevant operations. Numerous efforts have been conducted towards such solutions by exploiting a variety of sensing systems such as satellite Synthetic Aperture Radar (SAR) which can identify oil spills over sea surfaces in any environmental conditions and operational time. Such approaches include the use of artificial neural networks which effectively identify the polluted areas. Considering their remarkable abilities in many applications, deep Convolutional Neural Networks (DCNN) could surpass limitations and performances of previously proposed methods. This paper describes the application of an approach that combines the merits of a DCNN with SAR imagery in order to provide a fully automated oil spill identification system. The model semantically segments the input SAR images into multiple areas of interest. The deployed DCNN was trained using multiple SAR images acquired from the Sentinel-1 satellite provided by ESA and based on EMSA records for maritime pollution events. Experiments on such challenging benchmark datasets for such an abstract problem demonstrate that the algorithm can accurately identify oil spills leading to an effective detection solution.

The deliberate misuse of technical infrastructure (including the Web and social media) for cyber deviant and cybercriminal behaviour, ranging from the spreading of extremist and terrorism-related material to online fraud and cyber security attacks, is on the rise. This workshop aims to better understand such phenomena and develop methods for tackling them in an effective and efficient manner. The workshop brings together interdisciplinary researchers and experts in Web search, security informatics, social media analysis, machine learning, and digital forensics, with particular interests in cyber security. The workshop programme includes refereed papers, invited talks and a panel discussion for better understanding the current landscape, as well as the future of data mining for detecting cyber deviance.

Focused crawlers enable the automatic discovery of Web resources about a given topic by automatically navigating through the Web link structure and selecting the hyperlinks to follow by estimating their relevance to the topic of interest. This work proposes a generic focused crawling framework for discovering resources on any given topic that reside on the Surface or the Dark Web. The proposed crawler is able to seamlessly navigate through the Surface Web and several darknets present in the Dark Web (i.e., Tor, I2P, and Freenet) during a single crawl by automatically adapting its crawling behavior and its classifier-guided hyperlink selection strategy based on the destination network type and the strength of the local evidence present in the vicinity of a hyperlink. It investigates 11 hyperlink selection methods, among which a novel strategy proposed based on the dynamic linear combination of a link-based and a parent Web page classifier. This hybrid focused crawler is demonstrated for the discovery of Web resources containing recipes for producing homemade explosives. The evaluation experiments indicate the effectiveness of the proposed focused crawler both for the Surface and the Dark Web.

Interactive video retrieval tools developed over the past few years are emerging as powerful alternatives to automatic retrieval approaches by giving the user more control as well as more responsibilities. Current research tries to identify the best combinations of image, audio and text features that combined with innovative UI design maximize the tools performance. We present the last installment of the Video Browser Showdown 2015 which was held in conjunction with the International Conference on MultiMedia Modeling 2015 (MMM 2015) and has the stated aim of pushing for a better integration of the user into the search process. The setup of the competition including the used dataset and the presented tasks as well as the participating tools will be introduced . The performance of those tools will be thoroughly presented and analyzed. Interesting highlights will be marked and some predictions regarding the research focus within the field for the near future will be made.

This work proposes a generic focused crawling framework for discovering resources on any given topic that reside on the Surface or the Dark Web. The proposed crawler is able to seamlessly traverse the Surface Web and several darknets present in the Dark Web (i.e. Tor, I2P and Freenet) during a single crawl by automatically adapting its crawling behavior and its classifier-guided hyperlink selection strategy based on the network type. This hybrid focused crawler is demonstrated for the discovery of Web resources containing recipes for producing homemade explosives. The evaluation experiments indicate the effectiveness of the proposed ap-proach both for the Surface and the Dark Web.

Effective multimedia retrieval requires the combination of the heterogeneous media contained within multimedia objects and the features that can be extracted from them. To this end, we extend a unifying framework that integrates all well-known weighted, graph-based, and diffusion-based fusion techniques that combine two modalities (textual and visual similarities) to model the fusion of multiple modalities. We also provide a theoretical formula for the optimal number of documents that need to be initially selected, so that the memory cost in the case of multiple modalities remains the same as in the case of two modalities. Experiments using two test collections and three modalities (similarities based on visual descriptors, visual concepts, and textual concepts) indicate improvements in the effectiveness over bimodal fusion under the same memory complexity.

This work investigates the effectiveness of a state-of-the-art concept detection framework for the automatic classification of multimedia content, namely images and videos, embedded in publicly available Web resources containing recipes for the synthesis of Home Made Explosives (HMEs), to a set of predefined semantic concepts relevant to the HME domain. The concept detection framework employs advanced methods for video (shot) segmentation, visual feature extraction (using SIFT, SURF, and their variations), and classification based on machine learning techniques (logistic regression). The evaluation experiments are performed using an annotated collection of multimedia HME content discovered on the Web, and a set of concepts, which emerged both from an empirical study, and were also provided by domain experts and interested stakeholders, including Law Enforcement Agencies personnel. The experiments demonstrate the satisfactory performance of our framework, which in turn indicates the significant potential of the adopted approaches on the HME domain.

This work proposes a novel framework that integrates diverse state-of-the-art technologies for the discovery, analysis, retrieval, and recommendation of heterogeneous Web resources containing multimedia information about homemade explosives (HMEs), with particular focus on HME recipe information. The framework corresponds to a knowledge management platform that enables the interaction with HME information, and consists of three major components: (i) a discovery component that allows for the identification of HME resources on the Web, (ii) a content-based multimedia analysis component that detects HME-related concepts in multimedia content, and (iii) an indexing, retrieval, and recommendation component that processes the available HME information to enable its (semantic) search and provision of similar information. The proposed framework is being developed in a user-driven manner, based on the requirements of law enforcement and security agencies personnel, as well as HME domain experts. In addition, its development is guided by the characteristics of HME Web resources, as these have been observed in an empirical study conducted by HME domain experts. Overall, this framework is envisaged to increase the operational effectiveness and efficiency of law enforcement and security agencies in their quest to keep the citizen safe.

Ontology alignment is the process where two different ontologies that usually describe similar domains are ’aligned’, i.e. a set of correspondences between their entities, regarding semantic equivalence, is determined. In order to identify these correspondences several methods and metrics that measure semantic equivalence have been proposed in literature. The most common features that these metrics employ are string-, lexical-, structure- and semantic-based similarities for which several approaches have been developed. However, what hasn’t been investigated is the usage of visual-based features for determining entity similarity in cases where images are associated with concepts. Nowadays the existence of several resources (e.g. ImageNet) that map lexical concepts onto images allows for exploiting visual similarities for this purpose. In this paper, a novel approach for ontology matching based on visual similarity is presented. Each ontological entity is associated with sets of images, retrieved through ImageNet or web-based search, and state of the art visual feature extraction, clustering and indexing for computing the similarity between entities is employed. An adaptation of a popular Wordnet-based matching algorithm to exploit the visual similarity is also proposed. Our method is compared with traditional metrics against a standard ontology alignment benchmark dataset and demonstrates promising results.

Abstract Environmental and meteorological conditions are of utmost importance for the population, as they are strongly related to the quality of life. Citizens are increasingly aware of this importance. This awareness results in an increasing demand for environmental information tailored to their specific needs and background. We present an environmental information platform that supports submission of user queries related to environmental conditions and orchestrates results from complementary services to generate personalized suggestions. The system discovers and processes reliable data in the Web in order to convert them into knowledge. At runtime, this information is transferred into an ontology‐structured knowledge base, from which then information relevant to the specific user is deduced and communicated in the language of their preference. The platform is demonstrated with real world use cases in the south area of Finland, showing the impact it can have on the quality of everyday life.

Environmental data are considered of utmost importance for human life, since weather conditions, air quality and pollen are strongly related to health issues and affect everyday activities. This paper addresses the problem of discovery of air quality and pollen forecast Web resources, which are usually presented in the form of heatmaps (i.e. graphical representation of matrix data with colors). Towards the solution of this problem, we propose a discovery methodology, which builds upon a general purpose search engine and a novel post processing heatmap recognition layer. The first step involves generation of domain-specific queries, which are submitted to the search engine, while the second involves an image classification step based on visual low level features to identify Web sites including heatmaps. Experimental results comparing various visual features combinations show that relevant environmental sites can be efficiently recognized and retrieved.

This paper investigates the role of gaze movements as implicit user feedback during interactive video retrieval tasks. In this context, we use a content-based video search engine to perform an interactive video retrieval experiment, during which, we record the user gaze movements with the aid of an eye-tracking device and generate features for each video shot based on aggregated past user eye fixation and pupil dilation data. Then, we employ support vector machines, in order to train a classifier that could identify shots marked as relevant to a new query topic submitted by new users. The positive results provided by the classifier are used as recommendations for future users, who search for similar topics. The evaluation shows that important information can be extracted from aggregated gaze movements during video retrieval tasks, while the involvement of pupil dilation data improves the performance of the system and facilitates interactive video search.