Low-light imaging has become a popular topic in image processing, with the quality enhancement of low light images being as a significant challenge, due to the difficulty in retaining colors, patterns, texture and style when generating a normal light image. Our objectives are mainly to firstly better preserve texture regions in image enhancement, while, secondly, preserving colors via color histogram blocks and, finally, to enhance the quality of image through dense denoising blocks. Our proposed novel framework, namely HDD-Unet, is a double Unet based on photorealistic style transfer for low-light image enhancement. The proposed low-light image enhancement method combines color histogram-based fusion, Haar wavelet pooling, dense-denoising blocks and U-net as a backbone architecture to enhance the contrast, reduce noise, and improve the visibility of low light images. Experimental results demonstrate that our proposed method outperforms existing methods in terms of PSNR and SSIM quantitative evaluation metrics, reaching or outperforming state-of-the-art accuracy, but with less resources. We also conduct an ablation study to investigate the impact of our approach on overexposed images, and systematic analysis on the late fusion weighting parameters. Multiple experiments were conducted with artificial noise inserted to accomplish more efficient comparison. The results show that the proposed framework enhances accurately images with various gamma corrections. The proposed method represents a significant advance in the field of low light image enhancement and has the potential to address several challenges associated with low light imaging. • Low-light image enhancement based on a combination of two U-Net branches with wavelet transformations, Adaptive Instance Normalization, color histogram blocks and denoising dense blocks. • Analytic framework creation for the HDD-Unet architecture. • Evaluation of the proposed method in a qualitative and a quantitative way.

Object localization is a fundamental task in computer vision that traditionally requires labeled datasets for accurate results. Recent progress in self-supervised learning has enabled unsupervised object localization, reducing reliance on manual annotations. Unlike supervised encoders, which depend on annotated training data, self-supervised encoders learn semantic representations directly from large collections of unlabeled images. This makes them the natural foundation for unsupervised object localization, as they capture object-relevant features while eliminating the need for costly manual labels. These encoders produce semantically coherent patch embeddings. Grouping these embeddings reveals sets of patches that correspond to objects in an image. These patch sets can be converted into object masks or bounding boxes, enabling tasks such as single-object discovery, multi-object detection, and instance segmentation. By applying off-line mask clustering or using pre-trained vision-language models, unsupervised localization methods can assign semantic labels to discovered objects. This transforms initially class-agnostic objects (objects without class labels) into class-aware ones (objects with class labels), aligning these tasks with their supervised counterparts. This paper provides a structured review of unsupervised object localization methods in both class-agnostic and class-aware settings. In contrast, previous surveys have focused only on class-agnostic localization. We discuss state-of-the-art object discovery strategies based on self-supervised features and provide a detailed comparison of experimental results across a wide range of tasks, datasets, and evaluation metrics.

Recent works have established that AI models introduce spectral artifacts into generated images and propose approaches for learning to capture them using labeled data. However, the significant differences in such artifacts among different generative models hinder these approaches from generalizing to generators not seen during training. In this work, we build upon the key idea that the spectral distribution of real images constitutes both an invariant and highly discriminative pattern for AI-generated image detection. To model this under a self-supervised setup, we employ masked spectral learning using the pretext task of frequency reconstruction. Since generated images constitute out-of-distribution samples for this model, we propose spectral reconstruction similarity to capture this divergence. Moreover, we introduce spectral context attention, which enables our approach to efficiently capture subtle spectral inconsistencies in images of any resolution. Our spectral AI-generated image detection approach (SPAI) achieves a 5.5% absolute improvement in AUC over the previous state-of-the-art across 13 recent generative approaches, while exhibiting robustness against common online perturbations. Code is available on https://mever-team.github.io/spai.

Synthetic aperture radar automatic target recognition (SAR-ATR) systems have rapidly evolved to tackle incremental recognition challenges in operational settings. Data scarcity remains a major hurdle that conventional SAR-ATR techniques struggle to address. To cope with this challenge, we propose a few-shot class-incremental learning (FSCIL) framework based on a dual-branch architecture that focuses on local feature extraction and leverages the discrete Fourier transform and global filters to capture long-term spatial dependencies. This incorporates a lightweight cross-attention mechanism that fuses domain-specific features with global dependencies to ensure robust feature interaction, while maintaining computational efficiency by introducing minimal scale-shift parameters. The framework combines focal loss for class distinction under imbalance and center loss for compact intra-class distributions to enhance class separation boundaries. Experimental results on the MSTAR benchmark dataset demonstrate that the proposed framework consistently outperforms state-of-the-art methods in FSCIL SAR-ATR, attesting to its effectiveness in real-world scenarios.

In this paper, we introduce InDistill, a method that serves as a warmup stage for enhancing Knowledge Distillation (KD) effectiveness. InDistill focuses on transferring criti-cal information flow paths from a heavyweight teacher to a lightweight student. This is achieved via a training scheme based on curriculum learning that considers the distillation difficulty of each layer and the critical learning peri-ods when the information flow paths are established. This procedure can lead to a student model that is better pre-pared to learn from the teacher. To ensure the applicability of InDstill across a wide range of teacher-student pairs, we also incorporate a pruning operation when there is a discrepancy in the width of the teacher and student layers. This pruning operation reduces the width of the teacher's intermediate layers to match those of the student, allowing direct distillation without the need for an encoding stage. The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10, CIFAR-100, and ImageNet datasets demonstrating that preserving the information flow paths consistently increases the per-formance of the baseline KD approaches on both classi-fication and retrieval settings. The code is available at https://github.com/gsarridis/InDistill.

Abstract The aim of this work is to provide a systematic literature review of techniques for taxonomy generation across the cybersecurity domain. Cybersecurity taxonomies can be classified into manual and dynamic, each one of which focuses on different characteristics and tails different goals. Under this premise, we investigate the current state of the art in both categories with respect to their characteristics, applications and methods. To this end, we perform a systematic literature review in accordance with an extensive analysis of the tremendous need for dynamic taxonomies in the cybersecurity landscape. This analysis provides key insights into the advantages and limitations of both techniques, and it discusses the datasets which are most commonly used to generate cybersecurity taxonomies.

Stress has a severe impact on individuals irrespective of age, sex, work, or background. The reliable development of stress detection techniques enhances the social, educational, physical, economic, and professional quality of life, preventing chronic stress and proposing alleviation strategies. Research studies examine psychological, cognitive, behavioral, and physiological reactions to identify stress adequately. Deep Learning (DL) has received significant attention in recent years as it deals with high-dimensional, heterogeneous data and automatically learns representative features. This paper presents a survey on stress detection with recent DL approaches, leveraging data from all possible sources (physiological, speech, facial expressions, gestures, and social media content). The methodological outlines, the best results, and the main contributions of each study are discussed. We also describe publicly available datasets used by several of the presented works. Finally, we emphasize various open issues within the field of research and highlight key directions for future work.

This paper presents an updated iteration of the VERGE interactive video retrieval system. It offers various search options like free text and concept-based text search, color similarity, people and face detection, and visual and semantic similarity search. The system is designed to handle large amounts of data efficiently using advanced indexing techniques and state-of-the-art AI technology for visual content analysis. This paper describes enhancements made to improve usability for non-expert users, particularly through changes to the search and browsing interface.

With the evolution of 3D tools, there is now plenty of 3D data for digital applications. This includes 3D retrieval, which seeks to access such data across varied representations such as point clouds, meshes, and multi-view images. However, comprehensive analysis of how to efficiently utilize these representations, or modalities, for retrieval has been missing. This paper evaluates different encodings of each modality in uni-modal retrieval and explores optimal combinations for multimodal retrieval, with state-of-the-art methods from the 3D and image retrieval domains. Results indicate, e.g., that the MuseHash method performs best on mean average precision (MAP), while the CMCL method excels in recall.

Increased traffic on social media platforms, such as X (formerly known as Twitter), is associated with disaster events and consequently fire incidents. Although detecting fires through social media has garnered research interest in recent years, managing the overwhelming volume of daily posts remains challenging. Efficient collection and filtering of fire-related posts are crucial for detecting fires through X. The FireXPosts dataset presented in this article is a collection of posts from Canada and Greece, binarily annotated to aid emergency responders effectively. We train and evaluate uni-modal and bi-modal models for filtering fire-relevant posts and establishing performance baselines on the FireXPosts dataset. Experimental results indicate relatively similar performance between uni-modalities and the best-performing bi-modal models, suggesting that late fusion do not positively influence fire detection on X posts.

Escalating wildfire incidents necessitate improved post-disaster management practices for more effective response and recovery. This study advances the integration of Earth Observation technologies into the wildfire damage assessment phase, contributing a novel approach to augment disaster recovery efforts. Multi-temporal satellite imaging is crucial for monitoring wildfire-affected areas, and the widespread availability of multispectral images with high revisit frequencies substantially improves the comprehensive study of these changes. This paper presents an examination of deep learning techniques for change detection, employing a Siamese convolutional neural network enhanced with an Atrous Spatial Pyramid Pooling block for efficient image data processing. The model is trained and validated on the “Sentinel-2 Wildfire Change Detection Dataset” (S2-WCD), a custom-made dataset aimed at change detection methodologies. By introducing this specialized dataset and applying advanced neural network techniques, the study fills crucial research gaps, offering improvements in wildfire disaster management, particularly in the critical recovery phase following wildfire events.

Biometric Verification (BV) systems often exhibit accuracy disparities across different demographic groups, leading to biases in BV applications. Assessing and quantifying these biases is essential for ensuring the fairness of BV systems. However, existing bias evaluation metrics in BV have limitations, such as focusing exclusively on match or non-match error rates, overlooking bias on demographic groups with performance levels falling between the best and worst performance levels, and neglecting the magnitude of the bias present. This paper presents an in-depth analysis of the limitations of current bias evaluation metrics in BV and, through experimental analysis, demonstrates their contextual suitability, merits, and limitations. Additionally, it introduces a novel general-purpose bias evaluation measure for BV, the “Sum of Group Error Differences$(\mathbf{SED}_{G})$”. Our experimental results on controlled synthetic datasets demonstrate the effectiveness of demographic bias quantification when using existing metrics and our own proposed measure. We discuss the applicability of the bias evaluation metrics in a set of simulated demographic bias scenarios and provide scenario-based metric recommendations. Our code is publicly available under https://github.com/alaaobeid/SEDG.

Social media have evolved into a major source of communication and information sharing, and gradually become impactful in monitoring natural disasters such as wildfires, complementing traditional wildfire monitoring technologies. This paper proposes a comprehensive social media-sensing framework for early wildfire detection, encompassing functionalities like social media crawling, visual analytics, and geolocation, for the analysis of social media posts from the X (former Twitter) platform. Upon analysis, a fire event detection module clusters collected posts into fire events, generating relevant alerts. The framework synergizes with a computer vision algorithm, based on a YOLOv8 architecture, performing object detection on UAV imagery for the detection of individuals in danger in affected areas. The collaborative utilization of social media data and UAV imagery improves situational awareness, by providing information for both the fire incidents and the affected subjects in the area, allowing for a more informed decision making.

Geohazards related to water quality have become critical especially due to the harmful impacts of climate change and human activities. This constitutes the timely, efficient and accurate enough water quality monitoring as a significant component in the emergency management cycle, which can be realised by satellite remote sensing. In this paper, the first image-based benchmark dataset dedicated to mapping muddy waters is presented, named MUDDAT. The dataset is based on Sentinel-2 10m resolution images depicting various muddy water incidents across nine European countries and can essentially be used for semantic segmentation. The annotation procedure is based on an ensemble of three independent methods which reduces biases, namely, NDTI-MNDWI, SID and k-means clustering. Finally, a UNet deep learning model which is considered state-of-the-art in semantic segmentation tasks is trained and fine-tuned. Both qualitative and quantitative results in unseen regions reach high values in classification metrics, constituting MUDDAT a considerable option in water quality monitoring for operational use.

Memes are an increasingly prevalent element of online discourse in social networks, especially among young audiences. They carry ideas and messages that range from humorous to hateful, and are widely consumed. Their potentially high impact requires adequate means of control to moderate their use in large scale. In this work, we propose SimCLIP a deep learning-based architecture for cross-modal understanding of memes, leveraging a pre-trained CLIP encoder to produce context-aware embeddings and a Siamese fusion technique to capture the interactions between text and image. We perform an extensive experimentation on seven meme classification tasks across six datasets. We establish a new state of the art in Memotion7k with a 7.25% relative F1-score improvement, and achieve super-human performance on Harm-P with 13.73% F1-Score improvement. Our approach demonstrates the potential for compact meme classification models, enabling accurate and efficient meme monitoring. We share our code at https://github.com/jahuerta92/meme-classification-simclip.

We present the 3D Media Search Engine (3DMSE), which is designed to facilitate the exploration and retrieval of 3D models and images. 3DMSE incorporates unimodal, cross-modal and multimodal retrieval, using any combinations of mesh, point-cloud and multi-image representations. The 3DMSE system is built on the recently proposed MuseHash approach for multimodal representation, and offers a user-friendly web interface that enables formulating queries, presenting search results, and visualising 3D information in an accessible manner.

We face up to the challenge of explainability in Multimodal Artificial Intelligence (MMAI). At the nexus of neuroscience-inspired and quantum computing, interpretable and transparent spin-geometrical neural architectures for early fusion of large-scale, heterogeneous, graph-structured data are envisioned, harnessing recent evidence for relativistic quantum neural coding of (co-)behavioral states in the self-organizing brain, under competitive, multidimensional dynamics. The designs draw on a self-dual classical description – via special Clifford-Lipschitz operations – of spinorial quantum states within registers of at most 16 qubits for efficient encoding of exponentially large neural structures. Formally ‘trained’, Lorentz neural architectures with precisely one lateral layer of exclusively inhibitory interneurons accounting for anti-modalities, as well as their co-architectures with intra-layer connections are highlighted. The approach accommodates the fusion of up to 16 time-invariant interconnected (anti-)modalities and the crystallization of latent multidimensional patterns. Comprehensive insights are expected to be gained through applications to Multimodal Big Data, under diverse real-world scenarios.

This Signal Processing Grand Challenge (SPGC) targets a difficult automatic prediction problem of societal and medical relevance, namely, the detection of Alzheimer’s Dementia (AD). Participants were invited to employ signal processing and machine learning methods to create predictive models based on spontaneous speech data. The Challenge has been designed to assess the extent to which predictive models built based on speech in one language (English) generalise to another language (Greek). To the best of our knowledge no work has investigated acoustic features of the speech signal in multilingual AD detection. Our baseline system used conventional machine learning algorithms with Active Data Representation of acoustic features, achieving accuracy of 73.91% on AD detection, and 4.95 root mean squared error on cognitive score prediction.

Object detection appears to be omnipresent nowadays with detectors being available for every problem available, covering solutions from extra-light to ultra resource demanding models. Yet, the vast majority of these approaches are based on large datasets to provide the required feature diversity. This work focuses on object detection solutions which do not rely heavily on abundant training datasets but rather on medium-sized data collections. It uses Efficientdet object detector as base for the application of novel modifications which achieve better performance both in efficiency as well in effectiveness. The focus on medium-sized datasets aim at representing more commonplace datasets which can be accumulated and compiled with relative ease.

This paper presents a novel method for supporting multiple modalities in the field of image retrieval, called Multimodal Bayesian Supervised Hashing (MuseHash). The method takes into consideration the semantic information of the training data through the use of Bayesian regression to estimate the semantic probabilities and statistical properties in the retrieval process. MuseHash is an extension of the previously proposed Bayesian ridge-based Semantic Preserving Hashing (BiasHash) method. Experimentation on various domain-specific and benchmark datasets demonstrates that MuseHash outperforms seven existing state-of-the-art methods in image retrieval performance, regardless of the feature extractor type, code length, and visual or textual descriptors used. This highlights the robustness and adaptability of MuseHash, making it a promising solution for multimodal image retrieval.

Web bots are vital for the web as they can be used to automate several actions, some of which would have otherwise been impossible or very time consuming. These actions can be benign, such as website testing and web indexing, or malicious, such as unauthorised content scraping, scalping, vulnerability scanning, and more. To detect malicious web bots, recent approaches examine the visitors’ fingerprint and behaviour. For the latter, several values (i.e., features) are usually extracted from visitors’ web logs and used as input to train machine learning models. In this research we show that web bots can use recent advances in machine learning, and, more specifically, Reinforcement Learning (RL), to effectively evade behaviour-based detection techniques. To evaluate these evasive bots, we examine (i) how well they can evade a pre-trained bot detection framework, (ii) how well they can still evade detection after the detection framework is re-trained on new behaviours generated from the evasive web bots, and (iii) how bots perform if re-trained again on the re-trained detection framework. We show that web bots can repeatedly evade detection and adapt to the re-trained detection framework to showcase the importance of considering such types of bots when designing web bot detection frameworks.

The identification of key actors in complex networks has gathered significant interest by virtue of their importance in modern applications. Several of the existing methods employ standard centrality measures to achieve their goal and as a result, one of the main challenges is identifying key actor nodes with high relevance across all such measures. In this work, we propose a model based on the use of graph convolutional networks (GeNs) that retrieves the key actors in a network based on a centrality measure ranking aggregation scheme. We experimentally demonstrate the effectiveness of our solution compared to baseline and state-of-the-art approaches in terms of: i) accuracy, ii) performance compared to standard machine learning approaches, and iii) influence propagation capabilities.

This paper focuses on the determination of the evoked sentiments to people by observing outdoor and indoor spaces, aiming to create a tool for designers and architects that can be utilized for sophisticated designs. Since sentiment is subjective, the design process can be facilitated by an ancillary automated tool for sentiment extraction. Simultaneously, a dataset containing both real and virtual images of vacant architectural spaces is introduced, while the SUN attributes are also extracted from the images in order to be included throughout training. The dataset is annotated towards both valence and arousal, while five established and two custom architectures, one which has never been used before in classifying abstract concepts, are evaluated on the collected data.

In the construction domain, Digital twins are mostly used for facilities management of buildings, but their applications are still very limited. The virtualization of buildings and bridges in the last 15 years in the form of Building or Bridge Information Models is clearly identified as the starting point for the DTs. The industry has erected a frame with semantically rich 3D reference models that are now heavily enriched with visual sensor data captured on construction sites. This article provides an overview of the research and current practices of computer vision methods in the construction industry and presents typical examples of their applications for 3D reconstruction, safety management and structural monitoring for quality assurance. It then highlights the dominant achievements presented in the literature and concludes with the challenges and research directions applicable to digital twins that need to be addressed and exploited in the future.

The availability of satellite images has increased due to the fast development of remote sensing technology. As a result several deep learning change detection methods have been developed to capture spatial changes from multi temporal satellite images that are of great importance in remote sensing, monitoring environmental changes and land use. Recently, a supervised deep learning network called FresUNet has been proposed, which performs a pixel-level change detection from image pairs. In this paper, we extend this method by inserting a Bayesian framework that uses Monte Carlo Dropout, motivated by a recent work in image segmentation. The proposed Bayesian FresUNet (BiasUNet) approach is shown to outperform four state-of-the-art deep learning networks on Sentinel-2 ONERA Satellite Change Detection (OSCD) benchmark dataset, both in terms of precision and quality.

Accurate and timely flood mapping is important in emergency management which can be greatly served by Synthetic Aperture Radar (SAR). Research on SAR flood detection is mostly based on thresholding that has low time complexity and seems ideal for emergency response, although human intervention is needed. Machine learning methods have fewer errors and minimize human intervention but their computational complexity is higher. This work aims to provide a lightweight convolutional neural network baseline for pixel-wise time series flood classification in open land on SAR satellite data. Quantitative and qualitative evaluation of results indicate that the approach is promising.

Color comprises one of the most significant and dominant cues for various applications. As one of the most noticeable and stable attributes of vehicles, color can constitute a valuable key component in several practices of intelligent surveillance systems. In this paper, we propose a deep-learning-based framework that combines semantic segmentation masks with pixels clustering for automatic vehicle color recognition. Different from conventional methods, which usually consider only the features of the vehicle's front side, the proposed algorithm is able for view-independent color identification, which is more effective for the surveillance tasks. To the best of our knowledge, this is the first work that employs semantic segmentation masks along with color clustering for the extraction of the vehicle's color representative parts and the recognition of the dominant color, respectively. To evaluate the performance of the proposed method, we introduce a challenging multi-view dataset of 500 car-related RGB images extending the publicly available DSMLR Car Parts dataset for vehicle parts segmentation. The experiments demonstrate that the proposed approach achieves excellent performance and accurate results reaching an accuracy of 93.06% in the multi-view scenario. To facilitate further research, the evaluation dataset and the pre-trained models will be released at https://github.com/klearchos-stav/vehicle_color_recognition.

Social media data have been widely used in disaster management and particularly for the early detection of disaster emergencies. However, apart from sudden crises, there are also creeping crises, which are less evident but can be equally threatening to human lives, such as water pollution. The question raised is whether social media data can be used for discovering issues of water quality. In this work we attempt to answer this question by collecting posts from Twitter during the period of one year, which contain keywords about water quality, and applying three well-known techniques for event detection, i.e. Z-score, STA/LTA, and DBSCAN. A detailed presentation of the detected events, both relevant and not relevant, is given to provide more insight and proves that it is indeed feasible to identify water quality events with social media data. In addition, a quantitative evaluation of the three methods, in terms of precision, shows the superiority of Z-score for this particular topic.

Social media and web sites provide an access to pub-lic opinions on certain aspects and therefore play an important role in getting insights on targeted audiences. Designers have been investigating how to use them for grasping social feelings and needs associated with the arrangement of spaces that surround people in everyday life to find inspiration and come up with ideas for adaptive designs. Following this, we propose a novel design-oriented tool-set that retrieves and analyses online public information from Twitter and focused content from relevant web sites. We present the data collection pipeline and multilingual analysis algorithms like concept extraction and sentiment analysis on interior and urban design. Finally, we showcase an application of the proposed tool-set within two case studies.

Unexploded Ordnance (UXO) classification is a challenging task which is currently tackled using electromagnetic induction devices that are expensive and may require physical presence in potentially hazardous environments. The limited availability of open UXO data has, until now, impeded the progress of image-based UXO classification, which may offer a safe alternative at a reduced cost. In addition, the existing sporadic efforts focus mainly on small scale experiments using only a subset of common UXO categories. Our work aims to stimulate research interest in image-based UXO classification, with the curation of a novel dataset that consists of over 10000 annotated images from eight major UXO categories. Through extensive experimentation with supervised deep learning we uncover key insights into the challenging aspects of this task. Finally, we set the baseline on our novel benchmark by training state-of-the-art Convolutional Neural Networks and a Vision Transformer that are able to discriminate between highly overlapping UXO categories with 84.33% accuracy.

Computer Vision (CV) has achieved remarkable results, outperforming humans in several tasks. Nonetheless, it may result in significant discrimination if not handled properly. Indeed, CV systems highly depend on training datasets and can learn and amplify biases that such datasets may carry. Thus, the problem of understanding and discovering bias in visual datasets is of utmost importance; yet, it has not been studied in a systematic way to date. Hence, this work aims to: (i) describe the different kinds of bias that may manifest in visual datasets; (ii) review the literature on methods for bias discovery and quantification in visual datasets; (iii) discuss existing attempts to collect visual datasets in a bias-aware manner. A key conclusion of our study is that the problem of bias discovery and quantification in visual datasets is still open, and there is room for improvement in terms of both methods and the range of biases that can be addressed. Moreover, there is no such thing as a bias-free dataset, so scientists and practitioners must become aware of the biases in their datasets and make them explicit. To this end, we propose a checklist to spot different types of bias during visual dataset collection.

Abstract In this paper, we address the problem of high performance and computationally efficient content-based video retrieval in large-scale datasets. Current methods typically propose either: (i) fine-grained approaches employing spatio-temporal representations and similarity calculations, achieving high performance at a high computational cost or (ii) coarse-grained approaches representing/indexing videos as global vectors, where the spatio-temporal structure is lost, providing low performance but also having low computational cost. In this work, we propose a Knowledge Distillation framework, called Distill-and-Select (DnS), that starting from a well-performing fine-grained Teacher Network learns: (a) Student Networks at different retrieval performance and computational efficiency trade-offs and (b) a Selector Network that at test time rapidly directs samples to the appropriate student to maintain both high retrieval performance and high computational efficiency. We train several students with different architectures and arrive at different trade-offs of performance and efficiency, i.e., speed and storage requirements, including fine-grained students that store/index videos using binary representations. Importantly, the proposed scheme allows Knowledge Distillation in large, unlabelled datasets—this leads to good students. We evaluate DnS on five public datasets on three different video retrieval tasks and demonstrate (a) that our students achieve state-of-the-art performance in several cases and (b) that the DnS framework provides an excellent trade-off between retrieval performance, computational speed, and storage space. In specific configurations, the proposed method achieves similar mAP with the teacher but is 20 times faster and requires 240 times less storage space. The collected dataset and implementation are publicly available: https://github.com/mever-team/distill-and-select .

The increase of social media use in recent years has shown potential also for the identification of specific trends in the data that could be used to locate earthquakes. In this work, we implemented a pipeline that uses Twitter data to identify locations of earthquakes and use the information to trigger EO data analysis. We tested the pipeline for almost a year over Japan, an area where earthquake events are frequent, as well as the use of social media in the population. Here, we show the results and discuss the potential development of such procedures. In the future, considering the rapid development and the increase of satellite constellations aimed at global coverage with short revisit times, algorithms of this kind could be used to prioritize satellite acquisitions for the detection of the areas most affected by earthquake damages.

Recent developments in remote sensing have shown that snow depth can be estimated accurately on a global scale using satellite images through cross-polarization and copolarization backscatter measurements. This method does, however, have some limitations in low-land areas with dense forest coverage and shallow snow, which are often found nearby urban areas. In these areas, citizen observations can be fused with satellite-based estimations to deliver more accurate solutions. To that end, we use snow-related tweets that have been annotated by artificial intelligence (AI) methods and are introduced in a novel neural network model, aiming to increase the estimation accuracy of the state-of-the-art remote sensing method. The proposed model combines the estimated snow depth from Sentinel 1 images with the number of Twitter posts and Twitter images that are semantically relevant to snow. The use of instant social media data for purposes of snow depth estimation is investigated, validated, and tested in Finland. Our results show that this approach does improve the snow depth estimation, highlighting its potential for use in civil protection agencies in managing snow conditions.

The recognition of Steady State Visual Evoked Potentials (SSVEP) constitutes a challenging problem in Brain Computer Interfaces (BCI), especially when the number of EEG sensors is limited. In this work, we propose a new sparse representation classification scheme that extends current schemes by exploiting the graph properties of relevant features. Based on this scheme each test signal is represented as a linear combination of train signals. Our expectation is that this constrained linear combination, exploiting the graph's structure of the training data, will lead to representations that are more robust. Moreover, in order to avoid overfitting and provide a model with good generalization abilities we adopt the bayesian framework and, in particular, the Variational Bayesian Framework since we use a specific prior distribution to exploit the graph structure of the data. The proposed algorithm has been evaluated on two SSVEP datasets achieving state-of- the-art performance against well known classification methods in SSVEP literature.

In this work, we present a deep learning-based approach for image tampering localization fusion. This approach is designed to combine the outcomes of multiple image forensics algorithms and provides a fused tampering localization map, which requires no expert knowledge and is easier to interpret by end users. Our fusion framework includes a set of five individual tampering localization methods for splicing localization on JPEG images. The proposed deep learning fusion model is an adapted architecture, initially proposed for the image restoration task, that performs multiple operations in parallel, weighted by an attention mechanism to enable the selection of proper operations depending on the input signals. This weighting process can be very beneficial for cases where the input signal is very diverse, as in our case where the output signals of multiple image forensics algorithms are combined. Evaluation in three publicly available forensics datasets demonstrates that the performance of the proposed approach is competitive, outperforming the individual forensics techniques as well as another recently proposed fusion framework in the majority of cases.

This paper explores the development of a framework for content-aware user profiling, studying how image producers and consumers can be better understood and consequently better served through services such as matchmaking and friend recommendations. User interests and similarities are extracted and analyzed on the edge employing state of the art CNN models over user images for the tasks of classification, as well as, building latent user representations from personal media content. A private-by-design approach is employed through the development and deployment of models on-device, avoiding the need for communicating personal data to a central service. Experimental results show that user profiling can provide accurate ranking of the users' interests and meaningful user associations through profile similarity.

Surveillance systems currently deploy a variety of devices that can capture visual content (such as CCTV, body-worn cameras, and smartphone cameras), thus rendering the monitoring of video footage obtained from multiple such devices a complex task. This becomes especially challenging when monitoring social events that involve large crowds, particularly when there is a risk of crowd violence. This paper presents and demonstrates a crowd violence detection system that can process, analyze, and alert potential stakeholders, when violence-related content is identified in crowd-based video footage. Based on deep neural networks, the proposed end-to-end framework utilizes a 3D Convolutional Neural Network (CNN) to deal with the (near) real-time analysis of video streams and video files for crowd violence detection. The framework is trained, evaluated, and demonstrated using the Violent Flows dataset, a dataset related to crowd violence that is widely used for research. The presented framework is provided as a standalone application for desktop environments and can analyze both video streams and video files.

Artificial Intelligence (AI) is already part of our lives and is extensively entering the space sector to offer value-added Earth Observation (EO) products and services. The Copernicus programme provides data on a free, full and open basis, while the recently launched Data and Information Access Service (DIAS) providers index, store and exchange tremendous amounts of data and cloud infrastructure computational resources. Copernicus data and other georeferenced data sources are often highly heterogeneous, distributed and semantically fragmented. One example is the massively generated social media data from citizen observations, including visual, textual and spatiotemporal information. Social media information offers reliable, timely and very prescriptive information about a crisis event. In this work we present the multimodal fusion aspects for combining satellite images and social media for emergency response, such as flood monitoring and extreme weather conditions in polar regions.

This work presents a spatio-temporal activity detection and recognition framework for untrimmed surveillance videos consisting of a three-step pipeline: object detection, tracking, and activity recognition. The framework relies on the YOLO v4 architecture for object detection, Euclidean distance for tracking, while the activity recognizer uses a 3D Convolutional Deep learning architecture employing spatio-temporal boundaries and addressing it as multi-label classification. The evaluation experiments on the VIRAT dataset achieve accurate detections of the temporal boundaries and recognitions of activities in untrimmed videos, with better performance for the multi-label compared to the multi-class activity recognition.

In this paper, we address the problem of global-scale image geolocation, proposing a mixed classification-retrieval scheme. Unlike other methods that strictly tackle the problem as a classification or retrieval task, we combine the two practices in a unified solution leveraging the advantages of each approach with two different modules. The first leverages the EfficientNet architecture to assign images to a specific geographic cell in a robust way. The second introduces a new residual architecture that is trained with contrastive learning to map input images to an embedding space that minimizes the pairwise geodesic distance of same-location images. For the final location estimation, the two modules are combined with a search-within-cell scheme, where the locations of most similar images from the predicted geographic cell are aggregated based on a spatial clustering scheme. Our approach demonstrates very competitive performance on four public datasets, achieving new state-of-the-art performance in fine granularity scales, i.e., 15.0% at 1km range on Im2GPS3k.

Self-similarity is a prominent characteristic of natural images that can play a major role when it comes to their denoising, restoration or compression. In this paper, we propose a novel probabilistic model that is based on the concept of image patch similarity and applied to the problem of Single Image Super Resolution. Based on this model, we derive a Variational Bayes algorithm, which super resolves low-resolution images, where the assumed distribution for the quantified similarity between two image patches is heavy-tailed. Moreover, we prove mathematically that the proposed algorithm is both an extended and superior version of the probabilistic Non-Local Means (NLM). Its prime advantage remains though, which is that it requires no training. A comparison of the proposed approach with state-of-the-art methods, using various quantitative metrics shows that it is almost on par, for images depicting rural themes and in terms of the Structural Similarity Index (SSIM) with the best performing methods that rely on trained deep learning models. On the other hand, it is clearly inferior to them, for urban themed images and in terms of all metrics, especially for the Mean-Squared-Error (MSE). In addition, qualitative evaluation of the proposed approach is performed using the Perceptual Index metric, which has been introduced to better mimic the human perception of the image quality. This evaluation favors our approach when compared to the best performing method that requires no training, even if they perform equally in qualitative terms, reinforcing the argument that MSE is not always an accurate metric for image quality.

Personalized PageRank (PPR) is a popular scheme for scoring the relevance of network nodes to a set of seed ones through a random walk with restart process. Calculating the scores of all network nodes often involves the power method, which iterates the PPR formula until convergence to an empirically selected numerical tolerance. However, finding a tolerance that is not so lax as to impact pairwise node comparisons but not so strict as to require a high number of iterations to converge requires time-consuming empirical investigation. In this work we aim to avoid this investigation by stopping power method iterations when node score order is robust against subsequent changes. To do this, we analyse the expected fraction of random walks considered at a given iteration and identify a potential stopping point that depends on a (fixed) confidence level of future iterations preserving node order. Experiments on four real-world networks show that a confidence level of 98% runs in a fraction of the time and yields more than 0.999 Spearman correlation with the node order of 10-20numerical tolerance. Furthermore, that stopping point is comparable to empirically selecting a numerical tolerance that yields robust node order.

The recognition of Steady State Visual Evoked Responses (SSVEP) is a problem that arises frequently into the Brain Computer Interface (BCI) concept. In this paper, we present a novel algorithm for their recognition by formulating the pattern recognition problem in terms of linear regression. The inverse problem is solved using the bayesian framework and the decision is ruled in favor of the class with the minimum distance. The experimental results demonstrate that the proposed algorithm achieves better performance, in terms of classification accuracy, than the competing approaches.

The main goal of this document is to describe the implementation of each component of the colMOOC platform and the integration stage of the operational platform as a whole. Additionally, a first simple operational scenario that has been implemented is also being described, in order to demonstrate the interoperability of the platform. Finally, this paper presents a positive technical assessment of the 1st version of colMOOC platform through performance benchmarking experiments performed on system's components in order to verify its correct operation and interoperability of the different integrated modules to meet the requirements of the use cases for the end users.

In this work, we address the problem of audio-based near-duplicate video retrieval. We propose the Audio Similarity Learning (AuSiL) approach that effectively captures temporal patterns of audio similarity between video pairs. For the robust similarity calculation between two videos, we first extract representative audio-based video descriptors by leveraging transfer learning based on a Convolutional Neural Network (CNN) trained on a large scale dataset of audio events, and then we calculate the similarity matrix derived from the pairwise similarity of these descriptors. The similarity matrix is subsequently fed to a CNN network that captures the temporal structures existing within its content. We train our network following a triplet generation process and optimizing the triplet loss function. To evaluate the effectiveness of the proposed approach, we have manually annotated two publicly available video datasets based on the audio duplicity between their videos. The proposed approach achieves very competitive results compared to three state-of-the-art methods. Also, unlike the competing methods, it is very robust to the retrieval of audio duplicates generated with speed transformations.

Recently, research endeavors have shown the potentiality of Cycle-Consistent Adversarial Networks (CycleGAN) in style transfer. In Cycle-Consistent Adversarial Networks, the consistency loss is introduced to measure the difference between the original images and the reconstructed in both directions, forward and backward. In this work, the combination of Cycle-Consistent Adversarial Networks with Fast and Adaptive Bidimensional Empirical Mode Decomposition (FABEMD) is proposed to perform style transfer on images. In the proposed approach the cycle-consistency loss is modified to include the differences between the extracted Intrinsic Mode Functions (BIMFs) images. Instead of an estimation of pixel-to-pixel difference between the produced and input images, the FABEMD is applied and the extracted BIMFs are involved in the computation of the total cycle loss. This method enriches the computation of the total loss in a content-to-content and style-to-style comparison by connecting the spatial information to the frequency components. The experimental results reveal that the proposed method is efficient and produces qualitative results comparable to state-of-the-art methods.

This works focuses on examining the performance of the Single Shot Detector (SSD) model in resource restricted systems where maintaining the power of the full model comprises a significant prerequisite. The proposed SSD variations examine the behavior of lighter versions of SSD while propose measures to limit the unavoidable performance shortage. The outcomes of the conducted research demonstrate a remarkable trade-off between performance losses, speed improvement and the required resource reservation. Thus, the experimental results evidence the efficiency of the presented SSD alterations towards accomplishing higher frame rates and retaining the performance of the original model.

This work examines violence detection in video scenes of crowds and proposes a crowd violence detection framework based on a 3D convolutional deep learning architecture, the 3D-ResNet model with 50 layers. The proposed framework is evaluated on the Violent Flows dataset against several state-of-the-art approaches and achieves higher accuracy values in almost all cases, while also performing the violence detection activities in (near) real-time.

The first Multimodal Sentiment Analysis in Real-life Media (MuSe) 2020 was a Challenge-based Workshop held in conjunction with ACM Multimedia'20. It addresses three distinct 'in-the-wild` Sub-challenges: sentiment/ emotion recognition (MuSe-Wild), emotion-target engagement (MuSe-Target) and trustworthiness detection (MuSe-Trust). A large multimedia dataset MuSe-CaR was used, which was specifically designed with the intention of improving machine understanding approaches of how sentiment (e.g. emotion) is linked to a topic in emotional, user-generated reviews. In this summary, we describe the motivation, first of its kind 'in-the-wild` database, challenge conditions, participation, as well as giving an overview of utilised state-of-the-art techniques.

Abstract Dialogue‐based systems often consist of several components, such as communication analysis, dialogue management, domain reasoning, and language generation. In this paper, we present Converness, an ontology‐driven, rule‐based framework to facilitate domain reasoning for conversational awareness in multimodal dialogue‐based agents. Converness uses Web Ontology Language 2 (OWL 2) ontologies to capture and combine the conversational modalities of the domain, for example, deictic gestures and spoken utterances, fuelling conversational topic understanding, and interpretation using description logics and rules. At the same time, defeasible rules are used to couple domain and user‐centred knowledge to further assist the interaction with end users, facilitating advanced conflict resolution and personalised context disambiguation. We illustrate the capabilities of the framework through its integration into a multimodal dialogue‐based agent that serves as an intelligent interface between users (elderly, caregivers, and health experts) and an ambient assistive living platform in real home settings.

Local community detection is an emerging topic in network analysis that aims to detect well-connected communities encompassing sets of priorly known seed nodes. In this work, we explore the similar problem of ranking network nodes based on their relevance to the communities characterized by seed nodes. However, seed nodes may not be central enough or sufficiently many to produce high quality ranks. To solve this problem, we introduce a methodology we call seed oversampling, which first runs a node ranking algorithm to discover more nodes that belong to the community and then reruns the same ranking algorithm for the new seed nodes. We formally discuss why this process improves the quality of calculated community ranks if the original set of seed nodes is small and introduce a boosting scheme that iteratively repeats seed oversampling to further improve rank quality when certain ranking algorithm properties are met. Finally, we demonstrate the effectiveness of our methods in improving community relevance ranks given only a few random seed nodes of real-world network communities. In our experiments, boosted and simple seed oversampling yielded better rank quality than the previous neighborhood inflation heuristic, which adds the neighborhoods of original seed nodes to seeds.

Automated programs (bots) are responsible for a large percentage of website traffic. These bots can either be used for benign purposes, such as Web indexing, Website monitoring (validation of hyperlinks and HTML code), feed fetching Web content and data extraction for commercial use or for malicious ones, including, but not limited to, content scraping, vulnerability scanning, account takeover, distributed denial of service attacks, marketing fraud, carding and spam. To ensure their security, Web servers try to identify bot sessions and apply special rules to them, such as throttling their requests or delivering different content. The methods currently used for the identification of bots are based either purely on rule-based bot detection techniques or a combination of rule-based and machine learning techniques. While current research has developed highly adequate methods for Web bot detection, these methods' adequacy when faced with Web bots that try to remain undetected hasn't been studied. For this reason, we created and evaluated a Web bot detection framework on its ability to detect conspicuous bots separately from its ability to detect advanced Web bots. We assessed the proposed framework performance using real HTTP traffic from a public Web server. Our experimental results show that the proposed framework has significant ability to detect Web bots that do not try to hide their bot identity using HTTP Web logs (balanced accuracy in a false-positive intolerant server > 95%). However, detecting advanced Web bots that present a browser fingerprint and may present a humanlike behaviour as well is considerably more difficult.

In this paper we propose a new method for EEG source localization. Solving the above problem usually requires choosing an appropriate regularization term. Our method is based on the bayesian approach, hence, the regularization term is closely connected to the prior distribution. The proposed prior distribution has sparse properties favoring focal EEG sources. In order to obtain an efficient algorithm we use the Variational Bayesian framework which provides us a tractable iterative algorithm of closed form equations. We compare the proposed method with well-known approaches of EEG source localization and the results, using synthetic and real EEG data, have shown that our method presents state-of-the-art performance, especially in cases where we expect few activated brain regions.

A significant limitation on the wide use of a Motor Imagery (MI) Brain-Computer Interfaces is the acquisition of electroenchephalogram (EEG) data over a significant amount of EEG trials to achieve accurate classification. In this work, we propose a new sparse representation classification scheme to overcome the above limitation by training the proposed model using only a limited amount of EEG trials. Our algorithm extends current sparse representation classification schemes by exploiting the group sparsity of EEG features. We have evaluated the proposed algorithm on two MI EEG datasets, showing state-of-the-art performance against well known classification methods of MI BCI literature.

In this paper, we present work in progress on the development of an intelligent interconnected infrastructure for public security and protection domain. In contrast to other Internet of Things (IoT) frameworks, the proposed system aims to effectively combine device and human awareness to achieve situational awareness, so as to provide a protection and security environment for citizens. The emphasis is placed on tourists, by creating the appropriate infrastructure to address a set of urgent situations, such as health-related problems and missing children in overcrowded environments, supporting smart links between humans and entities on the basis of goals, and adapting device operation to comply with human objectives, profiles and privacy. The framework effectively combines state-of-the-art technologies on IoT data collection and analytics, knowledge representation and interoperability, crowdsourcing, data fusion and decision-making.

In this paper we introduce ViSiL, a Video Similarity Learning architecture that considers fine-grained Spatio-Temporal relations between pairs of videos -- such relations are typically lost in previous video retrieval approaches that embed the whole frame or even the whole video into a vector descriptor before the similarity estimation. By contrast, our Convolutional Neural Network (CNN)-based approach is trained to calculate video-to-video similarity from refined frame-to-frame similarity matrices, so as to consider both intra- and inter-frame relations. In the proposed method, pairwise frame similarity is estimated by applying Tensor Dot (TD) followed by Chamfer Similarity (CS) on regional CNN frame features - this avoids feature aggregation before the similarity calculation between frames. Subsequently, the similarity matrix between all video frames is fed to a four-layer CNN, and then summarized using Chamfer Similarity (CS) into a video-to-video similarity score - this avoids feature aggregation before the similarity calculation between videos and captures the temporal similarity patterns between matching frame sequences. We train the proposed network using a triplet loss scheme and evaluate it on five public benchmark datasets on four different video retrieval problems where we demonstrate large improvements in comparison to the state of the art. The implementation of ViSiL is publicly available.

This paper deals with the classification of steady-state visual evoked potentials (SSVEP), which is a multiclass classification problem addressed in SSVEP-based brain–computer interfaces. In particular, our method named MultiLRM_MKL uses multiple linear regression models under a Sparse Bayesian Learning (SBL) framework to discriminate between the SSVEP classes. The regression coefficients of each model are learned using the Variational Bayesian (VB) framework and the kernel trick is adopted not only for reducing the computational cost of our method, but also for enabling the combination of different kernel spaces. We verify the effectiveness of our method in handling different kernel spaces by evaluating its performance with a new kernel based on canonical correlation analysis. In particular, we prove the benefit of combining multiple kernels by outperforming several state-of-the-art methods in two SSVEP datasets, reaching an information transfer rate of 93 b/min using only three channels from the occipital area ($O_z$,$O_1$, and$O_2$).

This paper introduces the problem of Fine-grained Incident Video Retrieval (FIVR). Given a query video, the objective is to retrieve all associated videos, considering several types of associations that range from duplicate videos to videos from the same incident. FIVR offers a single framework that contains several retrieval tasks as special cases. To address the benchmarking needs of all such tasks, we construct and present a large-scale annotated video dataset, which we call FIVR-200K, and it comprises 225,960 videos. To create the dataset, we devise a process for the collection of YouTube videos based on major news events from recent years crawled from Wikipedia and deploy a retrieval pipeline for the automatic selection of query videos based on their estimated suitability as benchmarks. We also devise a protocol for the annotation of the dataset with respect to the four types of video associations defined by FIVR. Finally, we report the results of an experimental study on the dataset comparing five state-of-the-art methods developed based on a variety of visual descriptors, highlighting the challenges of the current problem.

A novel work for Ambient Assisted Living applications is presented here. More specifically, this paper focuses on activity recognition from recordings of daily living captured by wearable cameras. It constructs a discriminant object centric motion descriptor for representing the micro-actions within the viewpoint of the action maker so as to later define the activity that he/she performs. The accumulation of these activities build patterns over time that can be used to study the behavior of the end-users, which is very useful for health application and monitoring of patients inside their own dwellings or their behavior inside a controlled environment.

Oil spills pose a major threat of the oceanic and coastal environments, hence, an automatic detection and a continuous monitoring system comprises an appealing option for minimizing the response time of relevant operations. Numerous efforts have been conducted towards such solutions by exploiting a variety of sensing systems such as satellite Synthetic Aperture Radar (SAR) which can identify oil spills over sea surfaces in any environmental conditions and operational time. Such approaches include the use of artificial neural networks which effectively identify the polluted areas. Considering their remarkable abilities in many applications, deep Convolutional Neural Networks (DCNN) could surpass limitations and performances of previously proposed methods. This paper describes the application of an approach that combines the merits of a DCNN with SAR imagery in order to provide a fully automated oil spill identification system. The model semantically segments the input SAR images into multiple areas of interest. The deployed DCNN was trained using multiple SAR images acquired from the Sentinel-1 satellite provided by ESA and based on EMSA records for maritime pollution events. Experiments on such challenging benchmark datasets for such an abstract problem demonstrate that the algorithm can accurately identify oil spills leading to an effective detection solution.

Machine learning bias and fairness have recently emerged as key issues due to the pervasive deployment of data-driven decision making in a variety of sectors and services. It has often been argued that unfair classifications can be attributed to bias in training data, but previous attempts to 'repair' training data have led to limited success. To circumvent shortcomings prevalent in data repairing approaches, such as those that weight training samples of the sensitive group (e.g. gender, race, financial status) based on their misclassification error, we present a process that iteratively adapts training sample weights with a theoretically grounded model. This model addresses different kinds of bias to better achieve fairness objectives, such as trade-offs between accuracy and disparate impact elimination or disparate mistreatment elimination. We show that, compared to previous fairness-aware approaches, our methodology achieves better or similar trades-offs between accuracy and unfairness mitigation on real-world and synthetic datasets.

Location extraction, also called “toponym extraction,” is a field covering geoparsing, extracting spatial representations from location mentions in text, and geotagging, assigning spatial coordinates to content items. This article evaluates five “best-of-class” location extraction algorithms. We develop a geoparsing algorithm using an OpenStreetMap database, and a geotagging algorithm using a language model constructed from social media tags and multiple gazetteers. Third-party work evaluated includes a DBpedia-based entity recognition and disambiguation approach, a named entity recognition and Geonames gazetteer approach, and a Google Geocoder API approach. We perform two quantitative benchmark evaluations, one geoparsing tweets and one geotagging Flickr posts, to compare all approaches. We also perform a qualitative evaluation recalling top N location mentions from tweets during major news events. The OpenStreetMap approach was best (F1 0.90+) for geoparsing English, and the language model approach was best (F1 0.66) for Turkish. The language model was best (F1@1km 0.49) for the geotagging evaluation. The map database was best (R@20 0.60+) in the qualitative evaluation. We report on strengths, weaknesses, and a detailed failure analysis for the approaches and suggest concrete areas for further research.

Public awareness of and concerns about companies’ social and environmental impacts have seen a marked increase over recent decades. In parallel, the quantity of relevant information has increased, as states pass laws requiring certain forms of reporting, researchers investigate companies’ performance, and companies themselves seek to gain a competitive advantage by being seen to operate fairly and transparently. However, this information is typically dispersed and non-standardized, making it complicated to collect and analyze. To address this challenge, the WikiRate platform aims to collect this information and store it in a standardized format within a centralized public repository, making it much more amenable to analysis. In the context of WikiRate, this article introduces easIE, an easy-to-use information extraction (IE) framework that leverages general Web IE principles for building datasets with environmental, social, and governance information from the Web. To demonstrate the flexibility and value of easIE, we built a large-scale corporate social responsibility database comprising 654,491 metrics related to 49,009 companies spending less than 16 hours for data engineering, collection, and indexing. Finally, a data collection exercise involving 12 subjects was performed to showcase the ease of use of the developed framework.

Error-Related Potentials (ErrPs) have been used lately in order to improve several existing Brain-Computer Interface (BCI) applications. In our study we investigate the contribution of ErrPs in a Steady State Visual Evoked Potential (SSVEP) based BCI. An extensive study is presented in order to discover the limitations of the proposed scheme. Using Common Spatial Patterns and Random Forests we manage to show encouraging results regarding the incorporation of ErrPs in a SSVEP system. Finally, we provide a novel methodology (Inverse Correct Response Time) that can measure the gain of a BCI system by incorporating ErrPs in terms of time efficiency.

Brain-computer interfaces (BCIs) have been gaining momentum in making human-computer interaction more natural, especially for people with neuro-muscular disabilities. Among the existing solutions the systems relying on electroencephalograms (EEG) occupy the most prominent place due to their non-invasiveness. In this work, we provide a review of various existing techniques for the identification of motor imagery (MI) tasks. More specifically, we perform a comparison between Common Spatial Patterns (CSP) related features and features based on Power Spectral Density (PSD) techniques. Furthermore, for the identification of MI tasks, two well-known classifiers are used, the Linear Discriminant Analysis (LDA) and the Support Vector Machines (SVM). Our results confirm that PSD features demonstrate the most consistent robustness and effectiveness in extracting patterns for accurately discriminating between left and right MI tasks.

The publication and spread of misleading content is a problem of increasing magnitude, complexity and consequences in a world that is increasingly relying on user-generated content for news sourcing. To this end, multimedia analysis techniques could serve as an assisting tool to spot and debunk misleading content on the Web. In this paper, we tackle the problem of misleading multimedia content detection on Twitter in real time. We propose a number of new features and a new semi-supervised learning event adaptation approach that helps generalize the detection capabilities of trained models to unseen content, even when the event of interest is of different nature compared to that used for training. Combined with bagging, the proposed approach manages to outperform previous systems by a significant margin in terms of accuracy. Moreover, in order to communicate the verification process to end users, we develop a web-based application for visualizing the results.

Focused crawlers enable the automatic discovery of Web resources about a given topic by automatically navigating through the Web link structure and selecting the hyperlinks to follow by estimating their relevance to the topic of interest. This work proposes a generic focused crawling framework for discovering resources on any given topic that reside on the Surface or the Dark Web. The proposed crawler is able to seamlessly navigate through the Surface Web and several darknets present in the Dark Web (i.e., Tor, I2P, and Freenet) during a single crawl by automatically adapting its crawling behavior and its classifier-guided hyperlink selection strategy based on the destination network type and the strength of the local evidence present in the vicinity of a hyperlink. It investigates 11 hyperlink selection methods, among which a novel strategy proposed based on the dynamic linear combination of a link-based and a parent Web page classifier. This hybrid focused crawler is demonstrated for the discovery of Web resources containing recipes for producing homemade explosives. The evaluation experiments indicate the effectiveness of the proposed focused crawler both for the Surface and the Dark Web.

The large-scale availability of user-generated content in social media platforms has recently opened up new possibilities for studying and understanding the geospatial aspects of real-world phenomena and events. Yet, the large majority of user-generated content lacks proper geographic information (in the form of latitude and longitude coordinates). As a result, the problem of multimedia geotagging, i.e., extracting location information from user-generated text items when this is not explicitly available, has attracted increasing research interest. Here, we present a highly accurate geotagging approach for estimating the locations alluded by text annotations based on refined language models that are learned from massive corpora of social media annotations. We further explore the impact of different feature selection and weighting techniques on the performance of the approach. In terms of evaluation, we employ a large benchmark collection from the MediaEval Placing Task over several years. We demonstrate the consistently superior geotagging accuracy and low median distance error of the proposed approach using various data sets and comparing it against a number of state-of-the-art systems.

This work proposes a generic focused crawling framework for discovering resources on any given topic that reside on the Surface or the Dark Web. The proposed crawler is able to seamlessly traverse the Surface Web and several darknets present in the Dark Web (i.e. Tor, I2P and Freenet) during a single crawl by automatically adapting its crawling behavior and its classifier-guided hyperlink selection strategy based on the network type. This hybrid focused crawler is demonstrated for the discovery of Web resources containing recipes for producing homemade explosives. The evaluation experiments indicate the effectiveness of the proposed ap-proach both for the Surface and the Dark Web.

Effective multimedia retrieval requires the combination of the heterogeneous media contained within multimedia objects and the features that can be extracted from them. To this end, we extend a unifying framework that integrates all well-known weighted, graph-based, and diffusion-based fusion techniques that combine two modalities (textual and visual similarities) to model the fusion of multiple modalities. We also provide a theoretical formula for the optimal number of documents that need to be initially selected, so that the memory cost in the case of multiple modalities remains the same as in the case of two modalities. Experiments using two test collections and three modalities (similarities based on visual descriptors, visual concepts, and textual concepts) indicate improvements in the effectiveness over bimodal fusion under the same memory complexity.

This paper describes a multimodal graph-based approach to address the problem of event detection and summarization in large scale image collections. A first version of our system was presented in the Yahoo-Flickr Event Summarization Challenge of ACM Multimedia 2015 [6]. The objective of the approach is to automatically detect events within millions of photos and summarizing them efficiently for user consumption. The presented approach uses a moving time window over the collection of multimedia items to build a same-event image graph and applies graph clustering to detect events. In addition, it makes use of a graph-based diversity-oriented ranking algorithm to summarize instances of the detected events. A demo of the system is online at: http://mklab.iti.gr/acmmm2015-gc/.

Information sharing in online social networks is a daily practice for billions of users. The sharing process facilitates the maintenance of users' social ties but also entails privacy disclosure in relation to other users and third parties. Depending on the intentions of the latter, this disclosure can become a risk. It is thus important to propose tools that empower the users in their relations to social networks and third parties connected to them. As part of USEMP, a coordinated research effort aimed at user empowerment, we introduce a system that performs privacy-aware classification of images. We show that generic privacy models perform badly with real-life datasets in which images are contributed by individuals because they ignore the subjective nature of privacy. Motivated by this, we develop personalized privacy classification models that, utilizing small amounts of user feedback, provide significantly better performance than generic models. The proposed semi-personalized models lead to performance improvements for the best generic model ranging from 4%, when 5 user-specific examples are provided, to 18% with 35 examples. Furthermore, by using a semantic representation space for these models we manage to provide intuitive explanations of their decisions and to gain novel insights with respect to individuals' privacy concerns stemming from image sharing. We hope that the results reported here will motivate other researchers and practitioners to propose new methods of exploiting user feedback and of explaining privacy classifications to users.

This tutorial brings together a number of recent advances at the nexus of multimedia analysis, online privacy, and social media mining. Our goal is to offer a multidisciplinary view of the emerging field of Multimedia Privacy: the study of privacy issues arising in the context of multimedia sharing in online platforms, and the pursuit of new approaches to mitigating those issues within multimedia computer science.

Research on event-based processing and analysis of media is receiving an increasing attention from the scientific community due to its relevance for an abundance of applications, from consumer video management and video surveillance to lifelogging and social media.Events have the ability to semantically encode relationships of different informational modalities, such as visual-audio-text, time, involved agents and objects, with the spatio-temporal component of events being a key feature for contextual analysis.This unveils an enormous potential for exploiting new information sources and opening new research directions.In this paper, we survey the existing literature in this field.We extensively review the employed conceptualization of the notion of event in multimedia, the techniques for event representation and modeling, the feature representation and event inference approaches for the problems of event detection in audio, visual, and textual content.Furthermore, we review some key event-based multimedia applications, and various benchmarking activities that provide solid frameworks for measuring the performance of different event processing and analysis systems.We provide an in-depth discussion of the insights obtained from reviewing the literature and identify future directions and challenges.

Combining multimodal concept streams from heterogeneous sensors is a problem superficially explored for activity recognition. Most studies explore simple sensors in nearly perfect conditions, where temporal synchronization is guaranteed. Sophisticated fusion schemes adopt problem-specific graphical representations of events that are generally deeply linked with their training data and focused on a single sensor. This paper proposes a hybrid framework between knowledge-driven and probabilistic-driven methods for event representation and recognition. It separates semantic modeling from raw sensor data by using an intermediate semantic representation, namely concepts. It introduces an algorithm for sensor alignment that uses concept similarity as a surrogate for the inaccurate temporal information of real life scenarios. Finally, it proposes the combined use of an ontology language, to overcome the rigidity of previous approaches at model definition, and a probabilistic interpretation for ontological models, which equips the framework with a mechanism to handle noisy and ambiguous concept observations, an ability that most knowledge-driven methods lack. We evaluate our contributions in multimodal recordings of elderly people carrying out IADLs. Results demonstrated that the proposed framework outperforms baseline methods both in event recognition performance and in delimiting the temporal boundaries of event instances.

In this paper, we present SALIC, an active learning method for selecting the most appropriate user tagged images to expand the training set of a binary classifier. The process of active learning can be fully automated in this social context by replacing the human oracle with the images' tags. However, their noisy nature adds further complexity to the sample selection process since, apart from the images' informativeness (i.e., how much they are expected to inform the classifier if we knew their label), our confidence about their actual label should also be maximized (i.e., how certain the oracle is on the images' true contents). The main contribution of this work is in proposing a probabilistic approach for jointly maximizing the two aforementioned quantities. In the examined noisy context, the oracle's confidence is necessary to provide a contextual-based indication of the images' true contents, while the samples' informativeness is required to reduce the computational complexity and minimize the mistakes of the unreliable oracle. To prove this, first, we show that SALIC allows us to select training data as effectively as typical active learning, without the cost of manual annotation. Finally, we argue that the speed-up achieved when learning actively in this social context (where labels can be obtained without the cost of human annotation) is necessary to cope with the continuously growing requirements of large-scale applications. In this respect, we demonstrate that SALIC requires ten times less training data in order to reach the same performance as a straightforward informativeness-agnostic learning approach.

This work investigates the effectiveness of a state-of-the-art concept detection framework for the automatic classification of multimedia content, namely images and videos, embedded in publicly available Web resources containing recipes for the synthesis of Home Made Explosives (HMEs), to a set of predefined semantic concepts relevant to the HME domain. The concept detection framework employs advanced methods for video (shot) segmentation, visual feature extraction (using SIFT, SURF, and their variations), and classification based on machine learning techniques (logistic regression). The evaluation experiments are performed using an annotated collection of multimedia HME content discovered on the Web, and a set of concepts, which emerged both from an empirical study, and were also provided by domain experts and interested stakeholders, including Law Enforcement Agencies personnel. The experiments demonstrate the satisfactory performance of our framework, which in turn indicates the significant potential of the adopted approaches on the HME domain.

The phenomenal increase in the use of social media in recent years has raised a number of issues related to privacy. In this paper, we propose a framework for raising the awareness of Online Social Network (OSN) users with respect to the information about them that is disclosed and that can be inferred by OSN service operators as well as by third parties that can access their data. This framework takes the form of a semantic, hierarchical scoring structure, that enables users to easily browse over different privacy-related aspects of their presence in a social network. Contrary to previous privacy scoring approaches, the proposed framework provides a finer and more intuitive organization of privacy information. Importantly, it also takes into account both information that is explicitly mentioned in users' shared content, as well as implicit information, that may be inferred from it. We make available an open source implementation of the framework.

This work proposes a novel framework that integrates diverse state-of-the-art technologies for the discovery, analysis, retrieval, and recommendation of heterogeneous Web resources containing multimedia information about homemade explosives (HMEs), with particular focus on HME recipe information. The framework corresponds to a knowledge management platform that enables the interaction with HME information, and consists of three major components: (i) a discovery component that allows for the identification of HME resources on the Web, (ii) a content-based multimedia analysis component that detects HME-related concepts in multimedia content, and (iii) an indexing, retrieval, and recommendation component that processes the available HME information to enable its (semantic) search and provision of similar information. The proposed framework is being developed in a user-driven manner, based on the requirements of law enforcement and security agencies personnel, as well as HME domain experts. In addition, its development is guided by the characteristics of HME Web resources, as these have been observed in an empirical study conducted by HME domain experts. Overall, this framework is envisaged to increase the operational effectiveness and efficiency of law enforcement and security agencies in their quest to keep the citizen safe.

Online Social Networks (OSN) allow their users to organize their friends into groups, also known as social circles. These social circles can be used to better manage who has access to users' posted content and also to control the content posted from other users that they view. Unfortunately, these social circles are generated manually and this can be a laborious process for users with more than a few friends. In this paper, we propose an approach for automatically generating social circles that takes into account both the profile information of the friends to be grouped and the social network connectivity between them, while it allows multiple membership of friends in social circles. The approach is based on an adaptation of the widely used Latent Dirichlet Allocation model and, despite the fact that it does not explicitly model social network connectivity, as other state of the art methods do, it manages to achieve results that are competitive and even better than those obtained from such methods, at a considerably lower computational cost.

• Depth descriptors (HOSNP) achieves high acc. rate on location based ADLs such Hand-Shake/ Read Paper • Motion descriptors (MBH /HOF) achieve high accuracy rate for ADLs that feature significant motion such as: Drink, Eat, but at a much higher computational cost. • Appearance descriptors (HOG) give higher recognition rates for static ADLs such as: Read, Talk and discriminate between activities with similar motion : Drink/Eat and Prepare Snack/Prepare Drink Appearance and Depth for Rapid Human Activity Recognition in Real Applications

The rise of big data, which need computationally demanding manipulation has posed unprecedented challenges in the machine learning community. In this context, a variety of dimensionality reduction methods has been introduced in order to deal with the large-scale aspect of the data. However, their employment in very large scales often becomes impractical due to memory and computation limitations. In parallel, Compressive Sampling (CS) has recently emerged as a powerful mathematical framework providing a suite of conditions and methods that allow for an almost lossless and efficient compression of sparse data. Given that the majority of big data problems entail the existence of sparse datasets, our goal in this paper is to investigate the potential of CS as a dimensionality reduction method in very large scales. Towards this end, we propose a novel Product Compressive Sampling (PCS) method that is used for scalable image annotation. The new method displays robustness equal to the typical CS method, while decreases the computational complexity dramatically. Another novel characteristic of our work consists in establishing a connection between the sparsity level of the data and the effectiveness of PCS as a dimensionality reduction method for image annotation. For this purpose, a new metric for estimating the data sparsity is proposed. Finally, in comparison with the state-of-the-art, we show that PCS displays competitive classification performance, while at the same moment proves to be orders of magnitude superior in terms of computational efficiency.

Ontology alignment is the process where two different ontologies that usually describe similar domains are ’aligned’, i.e. a set of correspondences between their entities, regarding semantic equivalence, is determined. In order to identify these correspondences several methods and metrics that measure semantic equivalence have been proposed in literature. The most common features that these metrics employ are string-, lexical-, structure- and semantic-based similarities for which several approaches have been developed. However, what hasn’t been investigated is the usage of visual-based features for determining entity similarity in cases where images are associated with concepts. Nowadays the existence of several resources (e.g. ImageNet) that map lexical concepts onto images allows for exploiting visual similarities for this purpose. In this paper, a novel approach for ontology matching based on visual similarity is presented. Each ontological entity is associated with sets of images, retrieved through ImageNet or web-based search, and state of the art visual feature extraction, clustering and indexing for computing the similarity between entities is employed. An adaptation of a popular Wordnet-based matching algorithm to exploit the visual similarity is also proposed. Our method is compared with traditional metrics against a standard ontology alignment benchmark dataset and demonstrates promising results.

Cognitive Rehabilitation aims to enable people with cognitive impairments to optimize their cognitive functioning in the everyday context, in order to prevent or reduce excess disability, and thus improve quality of life and well-being. This paper presents the main elements of a sensor-based system to support cognitive skills of people suffering from Alzheimer disease (AD) and dementia. The system monitors the patients at their homes during the performance of daily activities. Different types of sensors are used for monitoring the patients, while a graphical user interface a) enables the clinicians to access and visualize the results and b) supports the patients through specific interventions. The sensor data is semantically integrated and analyzed using knowledge-driven interpretation techniques based on Semantic Web technologies. Overall, this paper presents a) the system, b) cognitive rehabilitation scenarios and c) the relevant evaluation methods.

Camera motion is often present in videos, due to shaking, jitter or ego-motion. This creates the need for a reliable motion compensation algorithm that will lead to accurate spatial activity localization and activity recognition. We detect background areas using SLIC superpixels and multiple homographies, and introduce motionplanes, defined as background areas that are undergoing diverse camera motions. Dense trajectories are sampled from these regions and histograms of appearance and motion are extracted to build a representation scheme to be used for activity recognition. Experiments on two publicly available unconstrained video datasets of human activities recorded with a moving camera show that our algorithm not only outperforms most activity localization state-of-the-art algorithms but also leads to high activity recognition accuracy rates.

In this work, we propose a computationally efficient method for the recognition of human activities of daily living. Our method uses trajectories of tracked visual features extracted on dense grids and performs recognition via Support Vector Machines (SVMs). In contrast to State-of-the-Art approaches, which are based on dense optical flow (OF), we use fast block matching motion estimation, resulting in increased computational efficiency, with minimal loss in terms of recognition accuracy. To prove the effectiveness of our approach, we have conducted experiments on benchmark datasets of videos of human activities of daily living, demonstrating the trade-offs between recognition accuracy and computational efficiency.

Based on the Aristotelian concept of potentiality vs. actuality allowing for the study of energy and dynamics in language, we propose a field approach to lexical analysis. Falling back on the distributional hypothesis to statistically model word meaning, we used evolving fields as a metaphor to express time-dependent changes in a vector space model by a combination of random indexing and evolving self-organizing maps (ESOM). To monitor semantic drifts within the observation period, an experiment was carried out on the term space of a collection of 12.8 million Amazon book reviews. For evaluation, the semantic consistency of ESOM term clusters was compared with their respective neighbourhoods in WordNet, and contrasted with distances among term vectors by random indexing. We found that at 0.05 level of significance, the terms in the clusters showed a high level of semantic consistency. Tracking the drift of distributional patterns in the term space across time periods, we found that consistency decreased, but not at a statistically significant level. Our method is highly scalable, with interpretations in philosophy.

In this paper we present FastAR, a software component capable of transforming Joomla based websites into AR-channels compatible with the most popular augmented reality browsers (i.e. Junaio, Layar, Wikitude). FastAR exploits the consistency of the data structure across multiple sites that have been developed using the same content management system, so as to automate the transformation process of an internet website to an augmented reality channel. The proposed component abstracts all related programming tasks and significantly reduces the time required to generate and publish AR-content, making the entire process manageable by non-experts. In verifying the usefulness and effectiveness of FastAR, we conducted a survey to solicit the opinion of users who carried out the installation and transformation process.

Due to the increasing popularity of microblogging platforms, the amount of messages (posts) related to public events, especially posts encompassing multimedia content, is steadily increasing. The inclusion of images can convey much more information about the event, compared to their text, which is typically very short (e.g., tweets). Although such messages can be quite informative regarding different aspects of the event, there is a lot of spam and redundancy making it challenging to extract pertinent insights. In this work, we describe a summarization framework that, given a set of social media messages about an event, aims to select a subset of images derived from them, that, at the same time, maximizes the relevance of the selected images and minimizes their redundancy. To this end, we propose a topic modeling technique to capture the relevance of messages to event topics and a graph-based algorithm to produce a diverse ranking of the selected high-relevance images. A user-centered evaluation on a large Twitter dataset around several real-world events demonstrates that the proposed method considerably outperforms a number of state-of-the-art summarization algorithms in terms of result relevance, while at the same time it is also highly competitive in terms of diversity. Namely, we get an improvement of 25% in terms of precision compared to the second best result, and 7% in terms of diversity.

Results returned by commercial image search engines should include relevant and diversified depictions of queries in order to ensure good coverage of users' information needs. While relevance has drastically improved in recent years, diversity is still an open problem. In this paper we propose a reranking method that could be implemented on top of such engines in order to provide a better balance between relevance and diversity. Our method formulates the reranking problem as an optimization of a utility function that jointly considers relevance and diversity. Our main contribution is the replacement of the unsupervised definition of relevance that is commonly used in this formulation with a supervised classification model that strives to capture a query and application-specific notion of relevance. This model provides more accurate relevance scores that lead to significantly improved diversification performance. Furthermore, we propose a stacking-type ensemble learning approach that allows combining multiple features in a principled way when computing the relevance of an image. An empirical evaluation carried out on the datasets of the MediaEval 2013 and 2014 "Retrieving Diverse Social Images" (RDSI) benchmarks confirms the superior performance of the proposed method compared to other participating systems as well as a state-of-the-art, unsupervised reranking method.

The paper describes a multimodal graph-based system for addressing the Yahoo-Flickr Event Summarization Challenge of ACM Multimedia 2015. The objective is to automatically uncover structure within a collection of 100 million photos/videos in the form of detecting and identifying events, and summarizing them succinctly for consumer consumption. The presented system uses a sliding window over the stream of multimedia items to build and maintain a multimodal same-event image graph and applies a graph clustering algorithm to detect events. In addition, it makes use of a graph-based diversity-oriented ranking approach and a versatile event retrieval mechanism to access summarized instances of the events of interest. A demo of the system is online at http://mklab.iti.gr/acmmm2015-gc/.

Abstract Environmental and meteorological conditions are of utmost importance for the population, as they are strongly related to the quality of life. Citizens are increasingly aware of this importance. This awareness results in an increasing demand for environmental information tailored to their specific needs and background. We present an environmental information platform that supports submission of user queries related to environmental conditions and orchestrates results from complementary services to generate personalized suggestions. The system discovers and processes reliable data in the Web in order to convert them into knowledge. At runtime, this information is transferred into an ontology‐structured knowledge base, from which then information relevant to the specific user is deduced and communicated in the language of their preference. The platform is demonstrated with real world use cases in the south area of Finland, showing the impact it can have on the quality of everyday life.

This paper focuses on detecting and localizing anomalous events in videos of crowded scenes, i.e., divergences from a dominant pattern. Both motion and appearance information are considered, so as to robustly distinguish different kinds of anomalies, for a wide range of scenarios. A newly introduced concept based on swarm theory, histograms of oriented swarms (HOS), is applied to capture the dynamics of crowded environments. HOS, together with the well-known histograms of oriented gradients, are combined to build a descriptor that effectively characterizes each scene. These appearance and motion features are only extracted within spatiotemporal volumes of moving pixels to ensure robustness to local noise, increase accuracy in the detection of local, nondominant anomalies, and achieve a lower computational cost. Experiments on benchmark data sets containing various situations with human crowds, as well as on traffic data, led to results that surpassed the current state of the art (SoA), confirming the method's efficacy and generality. Finally, the experiments show that our approach achieves significantly higher accuracy, especially for pixel-level event detection compared to SoA methods, at a low computational cost.

Motivated by the recently introduced scalable concept detection challenge that requires classifiers for hundreds or even thousands of concepts, the objective of this work is to predict the cases where the enhancement of an initial classifier with additional training images is not expected to provide significant improvements. To facilitate this objective, we need a model for predicting the performance gain of a bootstrapping process prior to actually applying it. In order to train this model, we propose two features; the initial classifier's maturity (i.e. how close is the current hyperplane to the optimal) and the oracle's reliability (i.e. how reliable is the oracle in providing the correct labels of new training data). Thus, the contribution of our work is on proposing a method that is able to exploit the correlation between the expected performance boost and these two indicators. As a result, we can considerably improve the scalability properties of such bootstrapping processes by concentrating on the most prominent models and thus reducing the overall processing load.

In this work we propose a novel approach to the detection of anomalous events occurring in crowded scenes. Swarm theory is applied for the creation of a motion feature first introduced in this work, the Histograms of Oriented Swarm Accelerations (HOSA), which are shown to effectively capture a scene's motion dynamics. The HOSA, together with the well known Histograms of Oriented Gradients (HOGs) describing appearance, are combined to provide a final descriptor based on both motion and appearance, to effectively characterize a crowded scene. Appearance and motion features are only extracted within spatiotemporal volumes of moving pixels (regions of interest) to ensure robustness to local noise and allow the detection of anomalies occurring only in a small region of the frame. Experiments and comparisons with the State of the Art (SoA) on a variety of benchmark datasets demonstrate the effectiveness of the proposed method, its flexibility and applicability to different crowd environments, and its superiority over currently existing approaches.

In this paper, we deal with the problem of video concept detection to use the concept detection results toward a more effective concept-based video retrieval. The key novelties of this paper are as follows: 1) the use of spatio-temporal video slices (tomographs) in the same way that visual keyframes are typically used in video concept detection schemes. These spatio-temporal slices capture in a compact way motion patterns that are useful for detecting semantic concepts and are used for training a number of base detectors. The latter augment the set of keyframe-based base detectors that can be trained using different frame representations. 2) The introduction of a generic methodology, built upon a genetic algorithm, for controlling which subset of the available base detectors (consequently, which subset of the possible shot representations) should be combined for developing an optimal detector for each specific concept. This methodology is directly applicable to the learning of hundreds of diverse concepts, while diverging from the one-size-fits-all approach that is typically used in problems of this size. The proposed techniques are evaluated on the datasets of the 2011 and 2012 Semantic Indexing Task of TRECVID, each comprising several hundred hours of heterogeneous video clips and ground-truth annotations for tens of concepts that exhibit significant variation in terms of generality, complexity, and human participation. The experimental results manifest the merit of the proposed techniques.

This paper deals with content-based large-scale image retrieval using the state-of-the-art framework of VLAD and Product Quantization proposed by Jegou as a starting point. Demonstrating an excellent accuracy-efficiency trade-off, this framework has attracted increased attention from the community and numerous extensions have been proposed. In this work, we make an in-depth analysis of the framework that aims at increasing our understanding of its different processing steps and boosting its overall performance. Our analysis involves the evaluation of numerous extensions (both existing and novel) as well as the study of the effects of several unexplored parameters. We specifically focus on: a) employing more efficient and discriminative local features; b) improving the quality of the aggregated representation; and c) optimizing the indexing scheme. Our thorough experimental evaluation provides new insights into extensions that consistently contribute, and others that do not, to performance improvement, and sheds light onto the effects of previously unexplored parameters of the framework. As a result, we develop an enhanced framework that significantly outperforms the previous best reported accuracy results on standard benchmarks and is more efficient.

In this paper, a new feature selection method is used, in combination with a semantic model vector video representation, in order to enumerate the key semantic evidences of an event in a video signal. In particular, a set of semantic concept detectors is firstly used for estimating a model vector for each video signal, where each element of the model vector denotes the degree of confidence that the respective concept is depicted in the video. Then, a novel feature selection method is learned for each event of interest. This method is based on exploiting the first two eigenvectors derived using the eigenvalue formulation of the mixture subclass discriminant analysis. Subsequently, given a video-event pair, the proposed method jointly evaluates the significance of each concept for the detection of the given event and the degree of confidence with which this concept is detected in the given video, in order to decide which concepts provide the strongest evidence in support of the provided video-event link. Experimental results using a video collection of TRECVID demonstrate the effectiveness of the proposed video event recounting method.

Environmental data are considered of utmost importance for human life, since weather conditions, air quality and pollen are strongly related to health issues and affect everyday activities. This paper addresses the problem of discovery of air quality and pollen forecast Web resources, which are usually presented in the form of heatmaps (i.e. graphical representation of matrix data with colors). Towards the solution of this problem, we propose a discovery methodology, which builds upon a general purpose search engine and a novel post processing heatmap recognition layer. The first step involves generation of domain-specific queries, which are submitted to the search engine, while the second involves an image classification step based on visual low level features to identify Web sites including heatmaps. Experimental results comparing various visual features combinations show that relevant environmental sites can be efficiently recognized and retrieved.

In this work we deal with the problem of video concept detection, for the purpose of using the detection results towards more effective concept-based video retrieval. In order to handle this task, we propose using spatio-temporal video slices, called video tomographs, in the same way that visual keyframes are typically used in traditional keyframe-based video concept detection schemes. Video tomographs capture in a compact way motion patterns that are present in the video, and are used in this work for training a number of base detectors. The latter augment the set of keyframe-based base detectors that can be trained on different image representations. Combining the keyframe-based and tomograph-based detectors, improved concept detection accuracy can be achieved. The proposed approach is evaluated on a dataset that is extensive both in terms of video duration and concept variation. The experimental results manifest the merit of the proposed approach.

In this paper, complex video events are learned and detected using a novel subclass recoding error-correcting outputs (SRECOC) design. In particular, a set of pre-trained concept detectors along different low-level visual feature types are used to provide a model vector representation of video signals. Subsequently, a subclass partitioning algorithm is used to divide only the target event class to several subclasses and learn one subclass detector for each event subclass. The pool of the subclass detectors is then combined under a SRECOC framework to provide a single event detector. This is achieved by first exploiting the properties of the linear loss-weighted decoding measure in order to derive a probability estimate along the different event subclass detectors, and then utilizing the sum probability rule along event subclasses to retrieve a single degree of confidence for the presence of the target event in a particular test video. Experimental results on the large-scale video collections of the TRECVID Multimedia Event Detection (MED) task verify the effectiveness of the proposed method. Moreover, the effect of weak or strong concept detectors on the accuracy of the resulting event detectors is examined.

The paper describes the approach, the experimental settings, and the results obtained by the proposed methodology at the ACM Yahoo! Multimedia Grand Challenge. Its main contribution is the use of fast and efficient features with a highly scalable semi-supervised learning approach, the Approximate Laplacian Eigenmaps (ALEs), and its extension, by computing the test set incrementally for learning concepts in time linear to the number of images (both labelled and unlabelled). A combination of two local visual features combined with the VLAD feature aggregation method and PCA is used to improve the efficiency and time complexity. Our methodology achieves somewhat better accuracy compared to the baseline (linear SVM) in small training sets, but improves the performance as the training data increase. Performing ALE fusion on a training set of 50K/concept resulted in a MiAP score of 0.4223, which was among the highest scores of the proposed approach.

ImproveMyCity is an open source platform that enables residents to directly report to their public administration local issues about their neighborhood such as discarded trash bins, faulty street lights, broken tiles on sidewalks, illegal advertising boards, etc. The reported issues are automatically transmitted to the appropriate office in public administration so as to schedule their settlement. Reporting is feasible both through a web- and a smartphone-based front-end that adopt a map-based visualization, which makes reporting a user-friendly and intriguing process. The management and routing of incoming issues is performed through a back-end infrastructure that serves as an integrated management system with easy to use interfaces. Apart from reporting a new issue, both front-ends allow the citizens to add comments or vote on existing issues, which adds a social dimension on the collected content. Finally, the platform makes also provision for informing the citizens about the progress status of the reported issue and in this way facilitate the establishment of a two-way dialogue between the citizen and public administration.

In this paper, a new method that exploits related videos for the problem of event detection is proposed, where related videos are videos that are closely but not fully associated with the event of interest. In particular, the Weighted Margin SVM formulation is modified so that related class observations can be effectively incorporated in the optimization problem. The resulting Relevance Degree SVM is especially useful in problems where only a limited number of training observations is provided, e.g., for the EK10Ex subtask of TRECVID MED, where only ten positive and ten related samples are provided for the training of a complex event detector. Experimental results on the TRECVID MED 2011 dataset verify the effectiveness of the proposed method.

This paper presents the 2012 Social Event Detection dataset (SED2012). The dataset constitutes a challenging benchmark for methods that detect social events in large collections of multimedia items. More specifically, the dataset comprises more than 160 thousands of Flickr photos and their accompanying metadata, as well as a list of 149 manually selected and annotated target events, each of which is defined as a set of relevant photos. This paper discusses the challenges defined as part of SED 2012, the data collection process, the dataset and its basic statistics, the ground truth creation and the suggested evaluation methodology.

Online social and news media generate rich and timely information about real-world events of all kinds. However, the huge amount of data available, along with the breadth of the user base, requires a substantial effort of information filtering to successfully drill down to relevant topics and events. Trending topic detection is therefore a fundamental building block to monitor and summarize information originating from social sources. There are a wide variety of methods and variables and they greatly affect the quality of results. We compare six topic detection methods on three Twitter datasets related to major events, which differ in their time scale and topic churn rate. We observe how the nature of the event considered, the volume of activity over time, the sampling procedure and the pre-processing of the data all greatly affect the quality of detected topics, which also depends on the type of detection method used. We find that standard natural language processing techniques can perform well for social streams on very focused topics, but novel techniques designed to mine the temporal distribution of concepts are needed to handle more heterogeneous streams containing multiple stories evolving in parallel. One of the novel topic detection methods we propose, based on -grams cooccurrence and topic ranking, consistently achieves the best performance across all these conditions, thus being more reliable than other state-of-the-art techniques.

In this paper, a theoretical link between mixture subclass discriminant analysis (MSDA) and a restricted Gaussian model is first presented. Then, two further discriminant analysis (DA) methods, i.e., fractional step MSDA (FSMSDA) and kernel MSDA (KMSDA) are proposed. Linking MSDA to an appropriate Gaussian model allows the derivation of a new DA method under the expectation maximization (EM) framework (EM-MSDA), which simultaneously derives the discriminant subspace and the maximum likelihood estimates. The two other proposed methods generalize MSDA in order to solve problems inherited from conventional DA. FSMSDA solves the subclass separation problem, that is, the situation in which the dimensionality of the discriminant subspace is strictly smaller than the rank of the inter-between-subclass scatter matrix. This is done by an appropriate weighting scheme and the utilization of an iterative algorithm for preserving useful discriminant directions. On the other hand, KMSDA uses the kernel trick to separate data with nonlinearly separable subclass structure. Extensive experimentation shows that the proposed methods outperform conventional MSDA and other linear discriminant analysis variants.

No abstract available.

The automated analysis of crowd behavior from videos has been a rather challenging problem to address due to the complexity and density of the motion, occlusions and local noise. A novel approach for the fast and reliable detection and characterization of abnormal events in crowd motions is proposed, based on particle advection and accurate optical flow estimation. Experiments on benchmark datasets show that changes are detected reliably and faster than existing methods. Also, regions of change are localized spatially, and the events occurring in the video are characterized with accuracy.

This paper presents a new method for human action recognition which exploits advantages of both trajectory and space-time based approaches in order to identify action patterns in given sequences. Videos with both a static and moving camera can be tackled, where camera motion effects are overcome via motion compensation. Only pixels undergoing changing motion, found by extracting motion boundary-based activity areas, are processed in order to introduce robustness to camera motion and reduce computational complexity. In these regions, densely sampled grid points on multiple scales are tracked using a KLT tracker, leading to dense multi-scale trajectories, on which HOGHOF descriptors are estimated. The length of each trajectory is determined by detecting changes in the tracked points' motion or appearance using sequential change detection techniques, namely the CUSUM approach. A vocabulary is created for each video's features using Hierarchical K-means, and the resulting fast search trees are used to describe the actions in the videos. SVMs are used for classification, using a kernel based on the similarity scores between training and testing videos. Experiments are carried out with new and challenging datasets for which the proposed method is shown to lead to recognition results that are comparable to or better than existing state of the art methods.

Our purpose in this work is to boost the performance of object classifiers learned using the self-training paradigm. We exploit the multi-modal nature of tagged images found in social networks, to optimize the process of region selection when retraining the initial model. More specifically, the proposed approach uses a small number of manually labelled regions to train the initial object detection classifiers. Then, a large number of loosely tagged images, pre-segmented by an automatic segmentation algorithm, is used to enhance the initial training set with additional image regions. However, in contrast to the typical case of self-training where the image regions are selected based solely on how well they fit to the original classification model, our approach aims at optimizing this selection by making combined use of both visual and textual information. The experimental results show that the object detection classifiers generated using the proposed approach outperform the classifiers generated using the typical self-training paradigm.

We present a technical demonstration of a novel smartphone application that enables efficient browsing and tagging of landmark and event photos. The application employs a hierarchical mode of exploration enabling zooming from the level of a city, through the level of an area/neighbourhood, down to the level of a specific spot. This navigation mechanism combined with photo clustering, and a dual map-list viewing mechanism, enables efficient browsing of hundreds of thousands of publicly available geotagged photos on a smartphone. Due to the large number of markers on the map, an adaptive clustering strategy is employed to reduce clutter on the screen. In addition to the advanced browsing capabilities, the application enables users to import their personal photos from Flickr and to easily annotate them by propagating to them metadata (location, tags) from the currently viewed object (area, landmark). The application currently supports more than 30 cities worldwide.

A large variety of features can be extracted from raw multimedia items. Moreover, in many contexts, like in the case of multimedia uploaded by users of social media platforms, items may be linked to metadata that can be very useful for a variety of analysis tasks. Nevertheless, such features are typically heterogeneous and are difficult to combine in a unified representation that would be suitable for analysis. In this paper, we discuss the problem of clustering collections of multimedia items with the purpose of detecting social events. In order to achieve this, a novel multimodal clustering algorithm is proposed. The proposed method uses a known clustering in the currently examined domain, in order to supervise the multimodal fusion and clustering procedure. It is tested on the MediaEval social event detection challenge data and is compared to a multimodal spectral clustering approach that uses early fusion. By taking advantage of the explicit supervisory signal, it achieves superior clustering accuracy and additionally requires the specification of a much smaller number of parameters. Moreover, the proposed approach has wider scope; it is not only applicable to the task of social event detection, but to other multimodal clustering problems as well.

In this letter, linear subclass support vector machines (LSSVMs) are proposed that can efficiently learn a piecewise linear decision function for binary classification problems. This is achieved using a nongaussianity criterion to derive the subclass structure of the data, and a new formulation of the optimization problem that exploits the subclass information. LSSVMs provide low computation cost during training and evaluation, and offer competitive recognition performance in comparison to other popular SVM-based algorithms. Experimental results on various datasets confirm the advantages of LSSVMs.

In this paper, a novel approach to evaluating video temporal decomposition algorithms is presented. The evaluation measures typically used to this end are nonlinear combinations of precision-recall or coverage-overflow, which are not metrics and additionally possess undesirable properties, such as nonsymmetricity. To alleviate these drawbacks, we introduce a novel unidimensional measure that is proven to be metric and satisfies a number of qualitative prerequisites that previous measures do not. This measure is named differential edit distance (DED), since it can be seen as a variation of the well-known edit distance. After defining DED, we further introduce an algorithm that computes it in less than cubic time. Finally, DED is extensively compared with state-of-the-art measures, namely, the harmonic means (F-score) of precision-recall and coverage-overflow. The experiments include comparisons of qualitative properties, the time required for optimizing the parameters of scene segmentation algorithms with the help of these measures, and a user study gauging the agreement of these measures with the users' assessment of the segmentation results. The results confirm that the proposed measure is a unidimensional metric that is effective in evaluating scene segmentation techniques and in helping to optimize their parameters.

We present a technical demonstration of an online city exploration application that helps users identify interesting spots in a city by use of spatio-temporal analysis and clustering of user contributed photos. Our framework analyzes the spatial distribution of large city-centered collections of user contributed photos at different time scales in order to index the most popular spots of a city in a time-aware manner. Subsequently, the photo sets belonging to the same spatiotemporal context are clustered in order to extract representative photos for each spot. The resulting application enables users to obtain flexible summaries of the most important spots in a city given a temporal slice (time of the day, month, season). The demonstration will be based on a photo dataset covering major European cities.

This paper demonstrates a new approach to detecting high-level events that may be depicted in images or video frames. Given a non-annotated content item, a large number of previously trained visual concept detectors are applied to it and their responses are used for representing the content item with a model vector in a high-dimensional concept space. Subsequently, an improved subclass discriminant analysis method is used for identifying a concept subspace within the aforementioned concept space, that is most appropriate for detecting and recognizing the target high-level events. In this subspace, the nearest neighbor rule is used for comparing the non-annotated content item with a few known example instances of the target events. The high-level events used as target events in the present version of the system are those defined for the TRECVID 2010 Multimedia Event Detection (MED) task.

This paper investigates the role of gaze movements as implicit user feedback during interactive video retrieval tasks. In this context, we use a content-based video search engine to perform an interactive video retrieval experiment, during which, we record the user gaze movements with the aid of an eye-tracking device and generate features for each video shot based on aggregated past user eye fixation and pupil dilation data. Then, we employ support vector machines, in order to train a classifier that could identify shots marked as relevant to a new query topic submitted by new users. The positive results provided by the classifier are used as recommendations for future users, who search for similar topics. The evaluation shows that important information can be extracted from aggregated gaze movements during video retrieval tasks, while the involvement of pupil dilation data improves the performance of the system and facilitates interactive video search.

The Third Workshop on Social Media (WSM2011) continues the series of Workshops on Social Media in 2009 and 2010 and has been established as a platform for the presentation and discussion of the latest, key research issues in social media analysis, exploration, search, mining, and emerging new social media applications. It is held in conjunction with the ACM International Multimedia Conference (MM'11) at Scottsdale, Arizona, USA, 2011 and has attracted contributions on various aspects of social media including data mining from social media, content organization, geo-localization, personalization, recommendation systems, user experience, machine learning and social media approaches and architectures for large-scale data processing.

In this letter, mixture subclass discriminant analysis (MSDA) that alleviates two shortcomings of subclass discriminant analysis (SDA) is proposed. In particular, it is shown that for data with Gaussian homoscedastic subclass structure a) SDA does not guarantee to provide the discriminant subspace that minimizes the Bayes error, and, b) the sample covariance matrix can not be used as the minimization metric of the discriminant analysis stability criterion (DSC). Based on this analysis MSDA modifies the objective function of SDA and utilizes a novel partitioning procedure to aid discrimination of data with Gaussian homoscedastic subclass structure. Experimental results confirm the improved classification performance of MSDA.

In this paper, a novel approach to video temporal decomposition into semantic units, termed scenes, is presented. In contrast to previous temporal segmentation approaches that employ mostly low-level visual or audiovisual features, we introduce a technique that jointly exploits low-level and high-level features automatically extracted from the visual and the auditory channel. This technique is built upon the well-known method of the scene transition graph (STG), first by introducing a new STG approximation that features reduced computational cost, and then by extending the unimodal STG-based temporal segmentation technique to a method for multimodal scene segmentation. The latter exploits, among others, the results of a large number of TRECVID-type trained visual concept detectors and audio event detectors, and is based on a probabilistic merging process that combines multiple individual STGs while at the same time diminishing the need for selecting and fine-tuning several STG construction parameters. The proposed approach is evaluated on three test datasets, comprising TRECVID documentary films, movies, and news-related videos, respectively. The experimental results demonstrate the improved performance of the proposed approach in comparison to other unimodal and multimodal techniques of the relevant literature and highlight the contribution of high-level audiovisual features toward improved video segmentation to scenes.

Computer vision techniques have made considerable progress in recognizing object categories by learning models that normally rely on a set of discriminative features. However, in contrast to human perception that makes extensive use of logic-based rules, these models fail to benefit from knowledge that is explicitly provided. In this paper, we propose a framework that can perform knowledge-assisted analysis of visual content. We use ontologies to model the domain knowledge and a set of conditional probabilities to model the application context. Then, a Bayesian network is used for integrating statistical and explicit knowledge and performing hypothesis testing using evidence-driven probabilistic inference. In addition, we propose the use of a focus-of-attention (FoA) mechanism that is based on the mutual information between concepts. This mechanism selects the most prominent hypotheses to be verified/tested by the BN, hence removing the need to exhaustively test all possible combinations of the hypotheses set. We experimentally evaluate our framework using content from three domains and for the following three tasks: 1) image categorization; 2) localized region labeling; and 3) weak annotation of video shot keyframes. The results obtained demonstrate the improvement in performance compared to a set of baseline concept classifiers that are not aware of any context or domain knowledge. Finally, we also demonstrate the ability of the proposed FoA mechanism to significantly reduce the computational cost of visual inference while obtaining results comparable to the exhaustive case.

This paper proposes the use of feature tracks for the detection of concepts in video, particularly dynamic concepts. Feature tracks are defined as sets of local interest points found in different frames of a video shot that exhibit spatio-temporal and visual continuity, defining a trajectory in the 2D+Time space. The extraction of feature tracks and the selection and representation of an appropriate subset of them allow the generation of a Bag-of-Spatiotemporal-Words model for the shot, which facilitates capturing the dynamics of video content. The experimental evaluation of the proposed approach highlights how the selection of such feature tracks for the definition of the Bag-of-Spatiotemporal-Words model enhances the results of traditional keyframe-based concept detection techniques.

In this paper, a probabilistic approach to combining spatial context with visual and co-occurrence information for semantic image analysis is presented. Overall, the examined image is segmented and subsequently an initial classification of the resulting image regions to semantic concepts is performed based solely on visual information. Then, a Genetic Algorithm (GA) is introduced for deciding on the optimal semantic image interpretation, realizing image analysis as a global optimization problem. The fundamental novelty of this work is that the GA incorporates in its evolutionary procedure a set of Bayesian Networks (BNs), which probabilistically learn the impact of the available spatial, visual and co-occurrence information on the final outcome for every possible pair of semantic concepts. Experimental results on two publicly available datasets demonstrate the efficiency of the proposed approach.

The wide adoption of photo sharing applications such as Flickr©and the massive amounts of user-generated content uploaded to them raises an information overload issue for users. An established technique to overcome such an overload is to cluster images into groups based on their similarity and then use the derived clusters to assist navigation and browsing of the collection. In this paper, we present a community detection (i.e. graph-based clustering) approach that makes use of both visual and tagging features of images in order to efficiently extract groups of related images within large image collections. Based on experiments we conducted on a dataset comprising publicly available images from Flickr©, we demonstrate the efficiency of our method, the added value of combining visual and tag features and the utility of the derived clusters for exploring an image collection.

In this paper, a statistical learning approach to spatial context exploitation for semantic image analysis is presented. The proposed method constitutes an extension of the key parts of the authors' previous work on spatial context utilization, where a Genetic Algorithm (GA) was introduced for exploiting fuzzy directional relations after performing an initial classification of image regions to semantic concepts using solely visual information. In the extensions reported in this work, a more elaborate approach is followed during the spatial knowledge acquisition and modeling process. Additionally, the impact of every resulting spatial constraint on the final outcome is adaptively adjusted. Experimental results as well as comparative evaluation on three datasets of varying complexity in terms of the total number of supported semantic concepts demonstrate the efficiency of the proposed method.

We present a technical demonstration of an online city exploration application that helps users identify interesting spots in a city by use of photo clusters corresponding to landmarks and events. Our application, called ClustTour, is based on an efficient landmark and event detection scheme for tagged photo collections. The proposed scheme relies on the combination of a graph-based photo clustering algorithm, making use of both visual and tag information of photos, with a cluster classification and merging module. ClustTour creates a map-based visualization of the identified photo clusters that are classified in prominent categories and are filterable by time and tag. We believe that such an application can greatly facilitate the task of knowing a city through its landmarks and events. So far, the demo has been based on a large photo dataset focused on Barcelona, and it is gradually expanding to contain photo clusters of several major cities of Europe. Furthermore, an Android application is developed that complements the web-based version of ClustTour.

In this paper, a graphical modeling-based approach to semantic video analysis is presented for jointly realizing modality fusion and temporal context exploitation. Overall, the examined video sequence is initially segmented into shots and for every resulting shot appropriate color, motion and audio features are extracted. Then, Hidden Markov Models (HMMs) are employed for performing an initial association of each shot with the semantic classes that are of interest separately for every modality. Subsequently, an integrated Bayesian Network (BN) is introduced for simultaneously performing information fusion and temporal contextual knowledge exploitation, contrary to the usual practice of performing each task separately. The final outcome of the overall video analysis approach is the association of a semantic class with every shot. Experimental results as well as comparative evaluation from the application of the proposed approach in the domain of news broadcast video are presented.

In this work the problem of automatic decomposition of video into elementary semantic units, known in the literature as scenes, is addressed. Two multi-modal automatic scene segmentation techniques are proposed, both building upon the Scene Transition Graph (STG). In the first of the proposed approaches, speaker diarization results are used for introducing a post-processing step to the STG construction algorithm, with the objective of discarding scene boundaries erroneously identified according to visual-only dissimilarity. In the second approach, speaker diarization and additional audio analysis results are employed and a separate audio-based STG is constructed, in parallel to the original STG based on visual information. The two STGs are subsequently combined. Preliminary results from the application of the proposed techniques to broadcast videos reveal their improved performance over previous approaches.

In this paper we present a hybrid recommendation system that combines ontological knowledge with content-extracted linguistic information, derived from pre-trained lexical graphs, in order to produce high quality, personalized recommendations. In the described approach, such recommendations are exemplified in an advertising scenario. We propose a distributed system architecture that uses semantic knowledge, based on terminologically enriched domain ontologies, to learn ontological user profiles and consequently infer recommendations through fuzzy semantic reasoning. A real world user study demonstrates the improvements attained in providing user-relevant recommendations with the aid of semantic profiles.

By analyzing explicit & implicit feedback information retrieval systems can determine topical relevance and tailor search criteria to the user's needs. In this paper we investigate whether it is possible to infer what is relevant by observing user affective behaviour. The sensory data employed range between facial expressions and peripheral physiological signals. We extract a set of features from the signals and analyze the data using classification methods, such as SVM and KNN. The results of our initial evaluation indicate that prediction of relevance is possible, to a certain extent, and implicit feedback models can benefit from taking into account user affective behavior.

In this paper, an approach to semantic video analysis that is based on the statistical processing and representation of the motion signal is presented. Overall, the examined video is temporally segmented into shots and for every resulting shot appropriate motion features are extracted; using these, hidden Markov models (HMMs) are employed for performing the association of each shot with one of the semantic classes that are of interest. The novel contributions of this paper lie in the areas of motion information processing and representation. Regarding the motion information processing, the kurtosis of the optical flow motion estimates is calculated for identifying which motion values originate from true motion rather than measurement noise. Additionally, unlike the majority of the approaches of the relevant literature that are mainly limited to global- or camera-level motion representations, a new representation for providing local-level motion information to HMMs is also presented. It focuses only on the pixels where true motion is observed. For the selected pixels, energy distribution-related information, as well as a complementary set of features that highlight particular spatial attributes of the motion signal, are extracted. Experimental results, as well as comparative evaluation, from the application of the proposed approach in the domains ofTennis,NewsandVolleyballbroadcast video, andHuman Actionvideo demonstrate the efficiency of the proposed method.

A robust, theoretically founded approach for the extraction of temporal templates corresponding to areas of motion in video, is presented. Higher order statistics (kurtosis) are employed to extract activity areas, i.e., binary masks indicating which pixels in a video are active. The application of the kurtosis on illumination changes modeled as Gaussians and mixture of Gaussians is shown to be sensitive to outliers for both models, thus correctly localizing active pixels. Activity areas are compared to existing, difference-based temporal templates, known as motion energy images, and the robustness of both categories of temporal templates to additive noise is analyzed theoretically. Experiments with numerous real videos with additive noise, both indoors and outdoors, are conducted to compare the robustness of the activity areas and motion energy images, and their temporal extensions, the activity history areas, and motion history images. As expected from the theoretical analysis, the kurtosis-based activity areas prove to be more robust than the difference-based templates. Challenging videos containing occlusions, varying backgrounds, and shadows are also examined, and it is shown that the proposed approach outperforms the difference-based method for these cases, as well, consistently providing reliable localization of activity under a wide range of difficult circumstances. The proposed approach provides good results at a very low computational cost, and without requiring prior knowledge about the scene, nor training of any kind.

This work presents a novel approach to the extraction of trajectories in video. It has the advantage of simultaneously processing all spatiotemporal information, i.e. all the video frames, and thus overcoming disadvantages of local approaches. A projection of the video frames over time is used to create a frequency modulated signal, which is then processed with the continuous wavelet transform (CWT). The CWT's power is concentrated around the prominent time-varying frequencies, which are proportional to the object trajectory. Consequently it can extract the time-varying motion trajectories, which can be used in future extensions to fully characterize the type motion taking place. This approach is robust to local measurement noise and occlusions, since it processes the available data in a global, integrated manner. Experiments take place with synthetic and real sequences demonstrate the capabilities of this approach.

In this paper, a motion-based approach for detecting high-level semantic events in video sequences is presented. Its main characteristic is its generic nature, i.e. it can be directly applied to any possible domain of concern without the need for domain-specific algorithmic modifications or adaptations. For realizing event detection, the examined video sequence is initially segmented into shots and for every resulting shot appropriate motion features are extracted. Then, Hidden Markov Models (HMMs) are employed for performing the association of each shot with one of the high-level semantic events that are of interest in any given domain. Regarding the motion feature extraction procedure, a new representation for providing local-level motion information to HMMs is presented, while motion characteristics from previous frames are also exploited. Experimental results as well as comparative evaluation from the application of the proposed approach in the domain of news broadcast video are presented.

Shot segmentation provides the basis for almost all high-level video content analysis approaches, validating it as one of the major prerequisites for efficient video semantic analysis, indexing and retrieval. The successful detection of both gradual and abrupt transitions is necessary to this end. In this paper a new gradual transition detection algorithm is proposed, that is based on novel criteria such as color coherence change that exhibit less sensitivity to local or global motion than previously proposed ones. These criteria, each of which could serve as a standalone gradual transition detection approach, are then combined using a machine learning technique, to result in a meta-segmentation scheme. Besides significantly improved performance, advantage of the proposed scheme is that there is no need for threshold selection, as opposed to what would be the case if any of the proposed features were used by themselves and as is typically the case in the relevant literature. Performance evaluation and comparison with four other popular algorithms reveals the effectiveness of the proposed technique.

Recently, advanced video processing systems have been developed for numerous applications, such as surveillance, tracking, monitoring, object and event detection. The large size of this data and the wide range of detectable events or objects render the automation of various video processing procedures necessary. The proposed system first finds the pixels where activities occur by processing the higher order statistics of luminance changes between accumulated video frames. Once these pixels are extracted, sequential likelihood ratio based change detection techniques are applied to find at which frames activities occur. The resulting system is novel, as it can localize actions both in space and in time, in a theoretically sound manner. Testing takes place with real video sequences, to demonstrate its effectiveness.

When a user performs a web search, the first query entered will frequently not return the required information. Thus, one needs to review the initial set of links and then to modify the query or construct a new one. This incremental process is particularly frustrating and difficult to manage for a mobile user due to the device limitations (e.g. keyboard, display). We present a query formulation architecture that employs the notion of context in order to automatically construct queries, where context refers to the article currently being viewed by the user. The proposed system uses semantic metadata extracted from the web page being consumed to automatically generate candidate queries. Novel methods are proposed to create and validate candidate queries. Further two variants of query expansion and a post-expansion validation technique are described. Finally, insights into the effectiveness of our system are provided based on evaluation tests of its individual components.

Under social tagging systems, a typical Web 2.0 application, users label digital data sources by using freely chosen textual descriptions (tags). Poor retrieval in the aforementioned systems remains a major problem mostly due to questionable tag validity and tag ambiguity. Earlier clustering techniques have shown limited improvements, since they were based mostly on tag co-occurrences. In this paper, a co-clustering approach is employed, that exploits joint groups of related tags and social data sources, in which both social and semantic aspects of tags are considered simultaneously. Experimental results demonstrate the efficiency and the beneficial outcome of the proposed approach in correlating relevant tags and resources.

Intravascular ultrasound (IVUS) constitutes a valuable technique for the diagnosis and management of coronary atherosclerosis. The detection of lumen and media-adventitia borders in IVUS images represents a necessary step in the utilization of the IVUS data for the reliable quantitative assessment of atherosclerotic lesions. In this paper, a fully automated technique for the detection of lumen and media-adventitia boundaries is developed. This comprises two different steps for contour initialization, one for each corresponding contour of interest, and a procedure for the refinement of the detected contours based on Radial Basis Function approximation. The proposed approach is shown to be capable of performing quick and reliable automated segmentation of IVUS images.

In recent years, digital multimedia technologies have evolved significantly, and are finding numerous applications, over the internet, and even over mobile networks. Thus, the video processing community has started focusing more intensively on the extraction of higher level information from multimedia data. This paper proposes a novel two-stage video processing system that aims to segment and extract semantically meaningful information, which can help achieve higher level interpretation of video. The flow fields present in the video are accumulated over several frames and their statistics are processed to derive an "activity area", that is characteristic of the type of events taking place. The color information complements the motion data, and is used for the accurate segmentation of the moving entities in each frame. The joint use of the activity area and accurate segmentation can serve as a first step to the further semantic interpretation of the video, including the recognition and accurate localization of moving objects of interest. We present experiments that demonstrate the effectiveness of our method for real videos.

Vast amounts of digital multimedia data are being produced and distributed today, so methods for the efficient and reliable extraction of information from video data are becoming necessary. We present a novel motion segmentation algorithm, which accurately extracts moving objects from a video, and also provides a likelihood map, for each object pixel assignment. The flow is estimated, and accumulated over several frames, to give action masks. Color segmentation clusters regions of similar color in each frame. A novel, likelihood ratio-based method for the statistical comparison of color layers in the regions of activity and the background is presented and compared with an Earth mover's distance-based approach. Our method also gives the likelihood with which each pixel is assigned to a moving object in each frame. Experiments with real sequences illustrate the advantages of our method, namely that it gives overall more reliable results, and also provides the likelihood map for the segmented object.

In large organizations the resources needed to solve challenging problems are typically dispersed over systems within and beyond the organization, and also in different media. However, there is still the need, in knowledge environments, for extraction methods able to combine evidence for a fact from across different media. In many cases the whole is more than the sum of its parts: only when considering the different media simultaneously can enough evidence be obtained to derive facts otherwise inaccessible to the knowledge worker via traditional methods that work on each single medium separately. In this paper, we present a cross-media knowledge extraction framework specifically designed to handle large volumes of documents composed of three types of media text, images and raw data and to exploit the evidence across the media. Our goal is to improve the quality and depth of automatically extracted knowledge.

In this special session overview paper, we present an introduction to the advances in semantic multimedia analysis for personalised content access. We firstly identify the importance of semantic multimedia analysis for personalised multimedia content applications, from a user and business perspective. We then summarise of the state of the art in these novel, and emerging domains. Finally, we describe some applications under development and evaluate their possible future commercial success

In this article, an approach to semantic image analysis is presented. Under the proposed approach, ontologies are used to capture general, spatial, and contextual knowledge of a domain, and a genetic algorithm is applied to realize the final annotation. The employed domain knowledge considers high-level information in terms of the concepts of interest of the examined domain, contextual information in the form of fuzzy ontological relations, as well as low-level information in terms of prototypical low-level visual descriptors. To account for the inherent ambiguity in visual information, uncertainty has been introduced in the spatial relations definition. First, an initial hypothesis set of graded annotations is produced for each image region, and then context is exploited to update appropriately the estimated degrees of confidence. Finally, a genetic algorithm is applied to decide the most plausible annotation by utilizing the visual and the spatial concepts definitions included in the domain ontology. Experiments with a collection of photographs belonging to two different domains demonstrate the performance of the proposed approach.

Efficient video content management and exploitation requires extraction of the underlying semantics, a non-trivial task associating low-level features of the image domain and high-level semantic descriptions. In this paper, a knowledge-assisted approach for extracting semantics of domain-specific video content is presented. Domain knowledge considers both low-level features (color, motion, shape) and spatial behavior (topological and directional information). During the preprocessing step, a set of over-segmented homogenous atom-regions is generated and their low-level and spatial descriptions are extracted. A genetic algorithm is then applied in order to find the optimal interpretation according to a specific domain conceptualization. The proposed approach was tested on the formula one, tennis and beach vacations domains showing promising results.

An approach to knowledge-assisted semantic video object detection based on a multimedia ontology infrastructure is presented. Semantic concepts in the context of the examined domain are defined in an ontology, enriched with qualitative attributes (e.g., color homogeneity), low-level features (e.g., color model components distribution), object spatial relations, and multimedia processing methods (e.g., color clustering). Semantic Web technologies are used for knowledge representation in the RDF(S) metadata standard. Rules in F-logic are defined to describe how tools for multimedia analysis should be applied, depending on concept attributes and low-level features, for the detection of video objects corresponding to the semantic concepts defined in the ontology. This supports flexible and managed execution of various application and domain independent multimedia analysis tasks. Furthermore, this semantic analysis approach can be used in semantic annotation and transcoding systems, which take into consideration the users environment including preferences, devices used, available network bandwidth and content identity. The proposed approach was tested for the detection of semantic objects on video data of three different domains.

A novel approach to domain-specific video analysis is proposed. The proposed approach is based on exploiting domain-specific knowledge in the form of an ontology to detect video objects corresponding to the semantic concepts defined in the ontology. The association between the visual objects and the defined semantic concepts is performed by taking into account both qualitative attributes of the semantic objects (e.g., color homogeneity), indicating necessary preprocessing methods (color clustering, respectively), and numerical data generated via training (e.g., color models, also defined in the ontology). To enable fast and efficient processing, this methodology is applied to MPEG-2 video, requiring only its partial decoding. The proposed approach is demonstrated in the domain of Formula-1 racing video and shows promising results.

The SCHEMA Network of Excellence aims to bring together a critical mass of universities, research centers, industrial partners and end users, in order to design a reference system for content-based semantic scene analysis, interpretation and understanding. In this paper, advances in the development of the SCHEMA reference system are reported, focusing on the application of region-based image retrieval using automatic segmentation. More specifically, the integration of four segmentation algorithms and the MPEG-7 eXperimentation Model with the reference system are discussed, along with the motivation behind these and various other choices that were made during the development of the reference system. Experimental results for this system, as well as results for an earlier version of it employing proprietary descriptors, are shown using a common collection of images. Comparative evaluation of these versions, both in terms of retrieval accuracy and in terms of time-efficiency, allows the evaluation of the reference system as a whole as well as the evaluation of the usability of different components integrated with it, such as the MPEG-7 eXperimentation Model. These results illustrate the efficiency of the proposed system, as well as its suitability in serving as a test-bed for evaluating and comparing different algorithms and approaches pertaining to the content-based and semantic manipulation of visual information.

In this paper, a color image segmentation algorithm and an approach to large-format image segmentation are presented, both focused on breaking down images to semantic objects for object-based multimedia applications. The proposed color image segmentation algorithm performs the segmentation in the combined intensity–texture–position feature space in order to produce connected regions that correspond to the real-life objects shown in the image. A preprocessing stage of conditional image filtering and a modified K-Means-with-connectivity-constraint pixel classification algorithm are used to allow for seamless integration of the different pixel features. Unsupervised operation of the segmentation algorithm is enabled by means of an initial clustering procedure. The large-format image segmentation scheme employs the aforementioned segmentation algorithm, providing an elegant framework for the fast segmentation of relatively large images. In this framework, the segmentation algorithm is applied to reduced versions of the original images, in order to speed-up the completion of the segmentation, resulting in a coarse-grained segmentation mask. The final fine-grained segmentation mask is produced with partial reclassification of the pixels of the original image to the already formed regions, using a Bayes classifier. As shown by experimental evaluation, this novel scheme provides fast segmentation with high perceptual segmentation quality.

In this paper, a novel algorithm is presented for the real-time, compressed-domain, unsupervised segmentation of image sequences and is applied to video indexing and retrieval. The segmentation algorithm uses motion and color information directly extracted from the MPEG-2 compressed stream. An iterative rejection scheme based on the bilinear motion model is used to effect foreground/background segmentation. Following that, meaningful foreground spatiotemporal objects are formed by initially examining the temporal consistency of the output of iterative rejection, clustering the resulting foreground macroblocks to connected regions and finally performing region tracking. Background segmentation to spatiotemporal objects is additionally performed. MPEG-7 compliant low-level descriptors describing the color, shape, position, and motion of the resulting spatiotemporal objects are extracted and are automatically mapped to appropriate intermediate-level descriptors forming a simple vocabulary termed object ontology. This, combined with a relevance feedback mechanism, allows the qualitative definition of the high-level concepts the user queries for (semantic objects, each represented by a keyword) and the retrieval of relevant video segments. Desired spatial and temporal relationships between the objects in multiple-keyword queries can also be expressed, using the shot ontology. Experimental results of the application of the segmentation algorithm to known sequences demonstrate the efficiency of the proposed segmentation approach. Sample queries reveal the potential of employing this segmentation algorithm as part of an object-based video indexing and retrieval scheme.

A novel unsupervised video object segmentation algorithm is presented, aiming to segment a video sequence to objects: spatiotemporal regions representing a meaningful part of the sequence. The proposed algorithm consists of three stages: initial segmentation of the first frame using color, motion, and position information, based on a variant of the K-means-with-connectivity-constraint algorithm; a temporal tracking algorithm, using a Bayes classifier and rule-based processing to reassign changed pixels to existing regions and to efficiently handle the introduction of new regions; and a trajectory-based region merging procedure that employs the long-term trajectory of regions, rather than the motion at the frame level, so as to group them to objects with different motion. As shown by experimental evaluation, this scheme can efficiently segment video sequences with fast moving or newly appearing objects. A comparison with other methods shows segmentation results corresponding more accurately to the real objects appearing on the image sequence.

In this paper, an image retrieval methodology suited for search in large collections of heterogeneous images is presented. The proposed approach employs a fully unsupervised segmentation algorithm to divide images into regions. Low-level features describing the color, position, size and shape of the resulting regions are extracted and are automatically mapped to appropriate intermediate-level descriptors forming a simple vocabulary termed object ontology. The object ontology is used to allow the qualitative definition of the high-level concepts the user queries for (semantic objects, each represented by a keyword) in a human-centered fashion. When querying, clearly irrelevant image regions are rejected using the intermediate-level descriptors; following that, a relevance feedback mechanism employing the low-level features is invoked to produce the final query results. The proposed approach bridges the gap between keyword-based approaches, which assume the existence of rich image captions or require manual evaluation and annotation of every image of the collection, and query-by-example approaches, which assume that the user queries for images similar to one that already is at his disposal.

A novel approach to large-format image segmentation is presented, focused on usage in content-based multimedia applications. The proposed framework aims at facilitating the time-efficient segmentation of large-format images while maintaining the high perceptual quality of the segmentation result. For this to be achieved, the employed segmentation algorithm is applied to reduced versions of the large-format images, in order to speed-up its execution, resulting in a coarse-grained segmentation mask. The final fine-grained segmentation mask is produced by an enhancement stage that involves partial reclassification of the pixels of the original image using a Bayes classifier. As shown by experimental evaluation, this novel scheme provides fast segmentation with high perceptual segmentation quality.

A procedure is described for the spatiotemporal segmentation and tracking of objects in colour image sequences. For this purpose, we propose the novel procedure of the k-means with a connectivity constraint algorithm as a general segmentation algorithm combining several types of information including colour, motion and compactness. A new colour distance is also defined for this algorithm. The regularisation parameters are evaluated automatically using the min-max criterion. In this algorithm, the use of spatiotemporal regions is introduced since a number of frames is analyzed simultaneously and as a result the same region is present in consequent frames. Experimental results on real and synthetic colour data demonstrate the performance of the data.

A region-based color image indexing and retrieval algorithm is presented. As a basis for the indexing, a novel K-Means segmentation algorithm is used, modified so as to take into account the coherence of the regions. A new color distance is also defined for this algorithm. Based on the extracted regions, characteristic features are estimated using color, texture and shape information. An important and unique aspect of the algorithm is that, in the context of similarity-based querying, the user is allowed to view the internal representation of the submitted image and the query results. Experimental results demonstrate the performance of the algorithm. The development of an intelligent image content-based search engine for the World-Wide Web is also presented, as a direct application of the presented algorithm.

This paper presents a novel procedure for the representation and coding of three-dimensional (3-D) surfaces using hierarchical adaptive triangulation. The proposed procedure is based on pyramidal analysis using the quincunx sampling minimum variance interpolation (QMVINT) filters. These are reduced pyramids with quincunx sampling applied to the parametric representation of the surface, chosen so as to minimize the variance of the interpolation error, and thus, when combined with the appropriate encoding of the coefficients, optimize the compression of the mesh information transmitted. At the same time, it produces a hierarchy of meshes based on quincunx sampling where coarse meshes are as similar to their finer versions as possible. This is very much desirable in progressive transmission. Depending on its interpolation error and the available bitrate, each filtered sample is a candidate for becoming a vertex of the mesh. The result is a progressive sequence of meshes consisting of more triangles wherever large variations exist and fewer in uniform regions. Complete correspondence between triangles at each level is identified, resulting in an efficient hierarchical representation of the mesh. The algorithm can be also used for the triangulation of a specific region of interest. Experimental results demonstrate that the proposed scheme provides an improvement in quality (MSE) by a factor of two when compared with other well known adaptive triangulation schemes.

This paper presents a novel procedure for the representation and coding of 3D surfaces using hierarchical adaptive triangulation. The proposed procedure is based on pyramidal analysis using the quincunx sampling minimum variance interpolation (QMVINT) filters. These are reduced pyramids with quincunx sampling applied to the parametric representation of the surface, chosen so as to minimize the variance of the interpolation error, and thus optimize the compression of the mesh information transmitted. At the same time, it produces a hierarchy of meshes based on quincunx sampling where coarse meshes are as similar to their finer versions as possible. This is very much desirable in progressive transmission. The result is a progressive sequence of meshes consisting of more triangles wherever large variations exist and fewer in uniform regions. Complete correspondence between triangles at each level is identified, resulting to an efficient hierarchical representation of the mesh. Experimental results demonstrate the efficient performance of the algorithm.

In this paper a procedure is described for the segmentation and tracking of objects in color image sequences. For this purpose, we propose the novel procedure of K-Means with a connectivity constraint algorithm as a general segmentation algorithm combining several types of information including color, motion and compactness. In this algorithm, the use of spatiotemporal regions is introduced since a number of frames is analyzed simultaneously and as a result the same region is present in consequent frames. Experimental results in real image sequences evaluate the performance of the algorithm.

This paper introduces an optimal hierarchical adaptive mesh construction algorithm using the face-centered orthorhombic lattice (FCO) sampling which is a natural extension of the quincunx lattice to the 3-dimensional case. A scheme for construction of adaptive meshes is presented. Initially, a highly detailed and densely sampled regular mesh is obtained from geometry scanning or from a nonoptimal polygon mesh. The adaptive triangle mesh is constructed by using fixed position vertices along with an efficient adaptive triangulation technique. The decimation is based on FCO sampling and surface estimation filters. The result is a progressive sequence of meshes consisting of more triangles wherever sharp edges exist and fewer in uniform plane regions. Experimental results demonstrate the usage and performance of the algorithm.

A procedure is described for the segmentation, content-based coding, and visualization of videoconference image sequences. First, image sequence analysis is used to estimate the shape and motion parameters of the person facing the camera. A spatiotemporal filter, taking into account the intensity differences between consequent frames, is applied, in order to separate the moving person from the static background. The foreground is segmented in a number of regions in order to identify the face. For this purpose, we propose the novel procedure of K-means with connectivity constraint algorithm as a general segmentation algorithm combining several types of information including intensity, motion and compactness. In this algorithm, the use of spatiotemporal regions is introduced since a number of frames are analyzed simultaneously and as a result, the same region is present in consequent frames. Based on this information, a 3-D ellipsoid is adapted to the person's face using an efficient and robust algorithm. The rigid 3-D motion is estimated next using a least median of squares approach, Finally, a virtual reality modeling language (VRML) file is created containing all the above information; this file may be viewed by using any VRML 2.0 compliant browser.

In this paper, a procedure is described for deformable boundary detection of medical tools, called stents, in angiographic images. A stent is a surgical stainless steel coil that is placed in the artery in order to improve blood circulation in regions where a stenosis has appeared. Assuming initially a set of three-dimensional (3-D) models of stents and using perspective projection of various deformations of the 3-D model of the stent, a large set of synthetic two-dimensional (2-D) images of stents is constructed. These synthetic images are then used as a training set for deriving a multivariate Gaussian density estimate based on eigenspace decomposition and formulating a maximum-likelihood estimation framework in order to reach an initial rough estimate for automatic object recognition. The silhouette of the detected stent is then refined by using a 2-D active contour (snake) algorithm integrated with a novel iterative initialization technique, which takes into consideration the geometry of the stent. The algorithm is experimentally evaluated using real angiographic images containing stents.

In this paper a procedure is described for the segmentation of image sequences. For this purpose, we propose the novel procedure of K-Means with connectivity constraint algorithm as a general segmentation algorithm combining several types of information including intensity, motion and compactness. The algorithm is extended so as to separate and track objects appearing in consequent frames of an image sequence. In this algorithm, the use of spatiotemporal regions is introduced since a number of frames is analyzed simultaneously and as a result the same region is present in consequent frames. Experimental results demonstrate the usage and performance of the algorithm.

This paper describes a 3D model-based unsupervised procedure for the segmentation of multiview image sequences using multiple sources of information. The articulation procedure is based on the homogeneity of parameters, such as rigid 3D motion, color and depth, estimated for each sub-object, which consists of a number of interconnected triangles of the 3D model. The rigid 3D motion of each sub-object for subsequent frames is estimated using a Kalman filtering algorithm taking into account the temporal correlation between consecutive frames. Information from all cameras is combined during the formation of the equations for the rigid 3D motion parameters. The parameter estimation for each sub-object and the 3D model segmentation procedures are interleaved and repeated iteratively until a satisfactory object segmentation emerges. The performance of the resulting segmentation method is evaluated experimentally.

This paper describes a three-dimensional (3-D) model-based unsupervised procedure for the segmentation of multiview image sequences using multiple sources of information. The 3-D model is initialized by accurate adaptation of a two-dimensional wireframe model to the foreground object of one of the views. The articulation procedure is based on the homogeneity of parameters, such as rigid 3-D motion, color, and depth, estimated for each subobject, which consists of a number of interconnected triangles of the 3-D model. The rigid 3-D motion of each subobject for subsequent frames is estimated using a Kalman filtering algorithm, taking into account the temporal correlation between consecutive frames. Information from all cameras is combined during the formation of the equations for the rigid 3-D motion parameters. The threshold used in the object segmentation procedure is updated at each iteration using the histogram of the subobject parameters. The parameter estimation for each subobject and the 3-D model segmentation procedures are interleaved and repeated iteratively until a satisfactory object segmentation emerges. The performance of the resulting segmentation method is evaluated experimentally.

This paper describes a procedure for model-based coding of all the channels of a multiview image sequence. The scheme is initialized by the adaptation of a wireframe model to the consistent depth information. Robust classification techniques are then used to obtain an articulated description of the foreground of the scene (head, neck, shoulders). The object articulation procedure is based on a novel scheme for the segmentation of the rigid 3D motion fields of the triangle patches of the 3D model object. Spatial neighborhood constraints are used to improve the reliability of the original triangle motion estimation. The motion estimation and motion field segmentation procedures are repeated iteratively until a satisfactory object articulation emerges. Rigid 3D motion estimation is performed next for each resulting sub-object. The performance of the resulting articulation method is evaluated experimentally.