Abstract The aim of this work is to provide a systematic literature review of techniques for taxonomy generation across the cybersecurity domain. Cybersecurity taxonomies can be classified into manual and dynamic, each one of which focuses on different characteristics and tails different goals. Under this premise, we investigate the current state of the art in both categories with respect to their characteristics, applications and methods. To this end, we perform a systematic literature review in accordance with an extensive analysis of the tremendous need for dynamic taxonomies in the cybersecurity landscape. This analysis provides key insights into the advantages and limitations of both techniques, and it discusses the datasets which are most commonly used to generate cybersecurity taxonomies.

We present the 3D Media Search Engine (3DMSE), which is designed to facilitate the exploration and retrieval of 3D models and images. 3DMSE incorporates unimodal, cross-modal and multimodal retrieval, using any combinations of mesh, point-cloud and multi-image representations. The 3DMSE system is built on the recently proposed MuseHash approach for multimodal representation, and offers a user-friendly web interface that enables formulating queries, presenting search results, and visualising 3D information in an accessible manner.

We face up to the challenge of explainability in Multimodal Artificial Intelligence (MMAI). At the nexus of neuroscience-inspired and quantum computing, interpretable and transparent spin-geometrical neural architectures for early fusion of large-scale, heterogeneous, graph-structured data are envisioned, harnessing recent evidence for relativistic quantum neural coding of (co-)behavioral states in the self-organizing brain, under competitive, multidimensional dynamics. The designs draw on a self-dual classical description – via special Clifford-Lipschitz operations – of spinorial quantum states within registers of at most 16 qubits for efficient encoding of exponentially large neural structures. Formally ‘trained’, Lorentz neural architectures with precisely one lateral layer of exclusively inhibitory interneurons accounting for anti-modalities, as well as their co-architectures with intra-layer connections are highlighted. The approach accommodates the fusion of up to 16 time-invariant interconnected (anti-)modalities and the crystallization of latent multidimensional patterns. Comprehensive insights are expected to be gained through applications to Multimodal Big Data, under diverse real-world scenarios.

Object detection appears to be omnipresent nowadays with detectors being available for every problem available, covering solutions from extra-light to ultra resource demanding models. Yet, the vast majority of these approaches are based on large datasets to provide the required feature diversity. This work focuses on object detection solutions which do not rely heavily on abundant training datasets but rather on medium-sized data collections. It uses Efficientdet object detector as base for the application of novel modifications which achieve better performance both in efficiency as well in effectiveness. The focus on medium-sized datasets aim at representing more commonplace datasets which can be accumulated and compiled with relative ease.

This paper presents a novel method for supporting multiple modalities in the field of image retrieval, called Multimodal Bayesian Supervised Hashing (MuseHash). The method takes into consideration the semantic information of the training data through the use of Bayesian regression to estimate the semantic probabilities and statistical properties in the retrieval process. MuseHash is an extension of the previously proposed Bayesian ridge-based Semantic Preserving Hashing (BiasHash) method. Experimentation on various domain-specific and benchmark datasets demonstrates that MuseHash outperforms seven existing state-of-the-art methods in image retrieval performance, regardless of the feature extractor type, code length, and visual or textual descriptors used. This highlights the robustness and adaptability of MuseHash, making it a promising solution for multimodal image retrieval.

The identification of key actors in complex networks has gathered significant interest by virtue of their importance in modern applications. Several of the existing methods employ standard centrality measures to achieve their goal and as a result, one of the main challenges is identifying key actor nodes with high relevance across all such measures. In this work, we propose a model based on the use of graph convolutional networks (GeNs) that retrieves the key actors in a network based on a centrality measure ranking aggregation scheme. We experimentally demonstrate the effectiveness of our solution compared to baseline and state-of-the-art approaches in terms of: i) accuracy, ii) performance compared to standard machine learning approaches, and iii) influence propagation capabilities.

Unexploded Ordnance (UXO) classification is a challenging task which is currently tackled using electromagnetic induction devices that are expensive and may require physical presence in potentially hazardous environments. The limited availability of open UXO data has, until now, impeded the progress of image-based UXO classification, which may offer a safe alternative at a reduced cost. In addition, the existing sporadic efforts focus mainly on small scale experiments using only a subset of common UXO categories. Our work aims to stimulate research interest in image-based UXO classification, with the curation of a novel dataset that consists of over 10000 annotated images from eight major UXO categories. Through extensive experimentation with supervised deep learning we uncover key insights into the challenging aspects of this task. Finally, we set the baseline on our novel benchmark by training state-of-the-art Convolutional Neural Networks and a Vision Transformer that are able to discriminate between highly overlapping UXO categories with 84.33% accuracy.

This work presents a spatio-temporal activity detection and recognition framework for untrimmed surveillance videos consisting of a three-step pipeline: object detection, tracking, and activity recognition. The framework relies on the YOLO v4 architecture for object detection, Euclidean distance for tracking, while the activity recognizer uses a 3D Convolutional Deep learning architecture employing spatio-temporal boundaries and addressing it as multi-label classification. The evaluation experiments on the VIRAT dataset achieve accurate detections of the temporal boundaries and recognitions of activities in untrimmed videos, with better performance for the multi-label compared to the multi-class activity recognition.

In this paper, we address the problem of global-scale image geolocation, proposing a mixed classification-retrieval scheme. Unlike other methods that strictly tackle the problem as a classification or retrieval task, we combine the two practices in a unified solution leveraging the advantages of each approach with two different modules. The first leverages the EfficientNet architecture to assign images to a specific geographic cell in a robust way. The second introduces a new residual architecture that is trained with contrastive learning to map input images to an embedding space that minimizes the pairwise geodesic distance of same-location images. For the final location estimation, the two modules are combined with a search-within-cell scheme, where the locations of most similar images from the predicted geographic cell are aggregated based on a spatial clustering scheme. Our approach demonstrates very competitive performance on four public datasets, achieving new state-of-the-art performance in fine granularity scales, i.e., 15.0% at 1km range on Im2GPS3k.

Personalized PageRank (PPR) is a popular scheme for scoring the relevance of network nodes to a set of seed ones through a random walk with restart process. Calculating the scores of all network nodes often involves the power method, which iterates the PPR formula until convergence to an empirically selected numerical tolerance. However, finding a tolerance that is not so lax as to impact pairwise node comparisons but not so strict as to require a high number of iterations to converge requires time-consuming empirical investigation. In this work we aim to avoid this investigation by stopping power method iterations when node score order is robust against subsequent changes. To do this, we analyse the expected fraction of random walks considered at a given iteration and identify a potential stopping point that depends on a (fixed) confidence level of future iterations preserving node order. Experiments on four real-world networks show that a confidence level of 98% runs in a fraction of the time and yields more than 0.999 Spearman correlation with the node order of 10-20numerical tolerance. Furthermore, that stopping point is comparable to empirically selecting a numerical tolerance that yields robust node order.

This work examines violence detection in video scenes of crowds and proposes a crowd violence detection framework based on a 3D convolutional deep learning architecture, the 3D-ResNet model with 50 layers. The proposed framework is evaluated on the Violent Flows dataset against several state-of-the-art approaches and achieves higher accuracy values in almost all cases, while also performing the violence detection activities in (near) real-time.

Local community detection is an emerging topic in network analysis that aims to detect well-connected communities encompassing sets of priorly known seed nodes. In this work, we explore the similar problem of ranking network nodes based on their relevance to the communities characterized by seed nodes. However, seed nodes may not be central enough or sufficiently many to produce high quality ranks. To solve this problem, we introduce a methodology we call seed oversampling, which first runs a node ranking algorithm to discover more nodes that belong to the community and then reruns the same ranking algorithm for the new seed nodes. We formally discuss why this process improves the quality of calculated community ranks if the original set of seed nodes is small and introduce a boosting scheme that iteratively repeats seed oversampling to further improve rank quality when certain ranking algorithm properties are met. Finally, we demonstrate the effectiveness of our methods in improving community relevance ranks given only a few random seed nodes of real-world network communities. In our experiments, boosted and simple seed oversampling yielded better rank quality than the previous neighborhood inflation heuristic, which adds the neighborhoods of original seed nodes to seeds.

Machine learning bias and fairness have recently emerged as key issues due to the pervasive deployment of data-driven decision making in a variety of sectors and services. It has often been argued that unfair classifications can be attributed to bias in training data, but previous attempts to 'repair' training data have led to limited success. To circumvent shortcomings prevalent in data repairing approaches, such as those that weight training samples of the sensitive group (e.g. gender, race, financial status) based on their misclassification error, we present a process that iteratively adapts training sample weights with a theoretically grounded model. This model addresses different kinds of bias to better achieve fairness objectives, such as trade-offs between accuracy and disparate impact elimination or disparate mistreatment elimination. We show that, compared to previous fairness-aware approaches, our methodology achieves better or similar trades-offs between accuracy and unfairness mitigation on real-world and synthetic datasets.

Location extraction, also called “toponym extraction,” is a field covering geoparsing, extracting spatial representations from location mentions in text, and geotagging, assigning spatial coordinates to content items. This article evaluates five “best-of-class” location extraction algorithms. We develop a geoparsing algorithm using an OpenStreetMap database, and a geotagging algorithm using a language model constructed from social media tags and multiple gazetteers. Third-party work evaluated includes a DBpedia-based entity recognition and disambiguation approach, a named entity recognition and Geonames gazetteer approach, and a Google Geocoder API approach. We perform two quantitative benchmark evaluations, one geoparsing tweets and one geotagging Flickr posts, to compare all approaches. We also perform a qualitative evaluation recalling top N location mentions from tweets during major news events. The OpenStreetMap approach was best (F1 0.90+) for geoparsing English, and the language model approach was best (F1 0.66) for Turkish. The language model was best (F1@1km 0.49) for the geotagging evaluation. The map database was best (R@20 0.60+) in the qualitative evaluation. We report on strengths, weaknesses, and a detailed failure analysis for the approaches and suggest concrete areas for further research.

The publication and spread of misleading content is a problem of increasing magnitude, complexity and consequences in a world that is increasingly relying on user-generated content for news sourcing. To this end, multimedia analysis techniques could serve as an assisting tool to spot and debunk misleading content on the Web. In this paper, we tackle the problem of misleading multimedia content detection on Twitter in real time. We propose a number of new features and a new semi-supervised learning event adaptation approach that helps generalize the detection capabilities of trained models to unseen content, even when the event of interest is of different nature compared to that used for training. Combined with bagging, the proposed approach manages to outperform previous systems by a significant margin in terms of accuracy. Moreover, in order to communicate the verification process to end users, we develop a web-based application for visualizing the results.

Effective multimedia retrieval requires the combination of the heterogeneous media contained within multimedia objects and the features that can be extracted from them. To this end, we extend a unifying framework that integrates all well-known weighted, graph-based, and diffusion-based fusion techniques that combine two modalities (textual and visual similarities) to model the fusion of multiple modalities. We also provide a theoretical formula for the optimal number of documents that need to be initially selected, so that the memory cost in the case of multiple modalities remains the same as in the case of two modalities. Experiments using two test collections and three modalities (similarities based on visual descriptors, visual concepts, and textual concepts) indicate improvements in the effectiveness over bimodal fusion under the same memory complexity.

This paper describes a multimodal graph-based approach to address the problem of event detection and summarization in large scale image collections. A first version of our system was presented in the Yahoo-Flickr Event Summarization Challenge of ACM Multimedia 2015 [6]. The objective of the approach is to automatically detect events within millions of photos and summarizing them efficiently for user consumption. The presented approach uses a moving time window over the collection of multimedia items to build a same-event image graph and applies graph clustering to detect events. In addition, it makes use of a graph-based diversity-oriented ranking algorithm to summarize instances of the detected events. A demo of the system is online at: http://mklab.iti.gr/acmmm2015-gc/.

Information sharing in online social networks is a daily practice for billions of users. The sharing process facilitates the maintenance of users' social ties but also entails privacy disclosure in relation to other users and third parties. Depending on the intentions of the latter, this disclosure can become a risk. It is thus important to propose tools that empower the users in their relations to social networks and third parties connected to them. As part of USEMP, a coordinated research effort aimed at user empowerment, we introduce a system that performs privacy-aware classification of images. We show that generic privacy models perform badly with real-life datasets in which images are contributed by individuals because they ignore the subjective nature of privacy. Motivated by this, we develop personalized privacy classification models that, utilizing small amounts of user feedback, provide significantly better performance than generic models. The proposed semi-personalized models lead to performance improvements for the best generic model ranging from 4%, when 5 user-specific examples are provided, to 18% with 35 examples. Furthermore, by using a semantic representation space for these models we manage to provide intuitive explanations of their decisions and to gain novel insights with respect to individuals' privacy concerns stemming from image sharing. We hope that the results reported here will motivate other researchers and practitioners to propose new methods of exploiting user feedback and of explaining privacy classifications to users.

Online Social Networks (OSN) allow their users to organize their friends into groups, also known as social circles. These social circles can be used to better manage who has access to users' posted content and also to control the content posted from other users that they view. Unfortunately, these social circles are generated manually and this can be a laborious process for users with more than a few friends. In this paper, we propose an approach for automatically generating social circles that takes into account both the profile information of the friends to be grouped and the social network connectivity between them, while it allows multiple membership of friends in social circles. The approach is based on an adaptation of the widely used Latent Dirichlet Allocation model and, despite the fact that it does not explicitly model social network connectivity, as other state of the art methods do, it manages to achieve results that are competitive and even better than those obtained from such methods, at a considerably lower computational cost.

The rise of big data, which need computationally demanding manipulation has posed unprecedented challenges in the machine learning community. In this context, a variety of dimensionality reduction methods has been introduced in order to deal with the large-scale aspect of the data. However, their employment in very large scales often becomes impractical due to memory and computation limitations. In parallel, Compressive Sampling (CS) has recently emerged as a powerful mathematical framework providing a suite of conditions and methods that allow for an almost lossless and efficient compression of sparse data. Given that the majority of big data problems entail the existence of sparse datasets, our goal in this paper is to investigate the potential of CS as a dimensionality reduction method in very large scales. Towards this end, we propose a novel Product Compressive Sampling (PCS) method that is used for scalable image annotation. The new method displays robustness equal to the typical CS method, while decreases the computational complexity dramatically. Another novel characteristic of our work consists in establishing a connection between the sparsity level of the data and the effectiveness of PCS as a dimensionality reduction method for image annotation. For this purpose, a new metric for estimating the data sparsity is proposed. Finally, in comparison with the state-of-the-art, we show that PCS displays competitive classification performance, while at the same moment proves to be orders of magnitude superior in terms of computational efficiency.

Ontology alignment is the process where two different ontologies that usually describe similar domains are ’aligned’, i.e. a set of correspondences between their entities, regarding semantic equivalence, is determined. In order to identify these correspondences several methods and metrics that measure semantic equivalence have been proposed in literature. The most common features that these metrics employ are string-, lexical-, structure- and semantic-based similarities for which several approaches have been developed. However, what hasn’t been investigated is the usage of visual-based features for determining entity similarity in cases where images are associated with concepts. Nowadays the existence of several resources (e.g. ImageNet) that map lexical concepts onto images allows for exploiting visual similarities for this purpose. In this paper, a novel approach for ontology matching based on visual similarity is presented. Each ontological entity is associated with sets of images, retrieved through ImageNet or web-based search, and state of the art visual feature extraction, clustering and indexing for computing the similarity between entities is employed. An adaptation of a popular Wordnet-based matching algorithm to exploit the visual similarity is also proposed. Our method is compared with traditional metrics against a standard ontology alignment benchmark dataset and demonstrates promising results.

Due to the increasing popularity of microblogging platforms, the amount of messages (posts) related to public events, especially posts encompassing multimedia content, is steadily increasing. The inclusion of images can convey much more information about the event, compared to their text, which is typically very short (e.g., tweets). Although such messages can be quite informative regarding different aspects of the event, there is a lot of spam and redundancy making it challenging to extract pertinent insights. In this work, we describe a summarization framework that, given a set of social media messages about an event, aims to select a subset of images derived from them, that, at the same time, maximizes the relevance of the selected images and minimizes their redundancy. To this end, we propose a topic modeling technique to capture the relevance of messages to event topics and a graph-based algorithm to produce a diverse ranking of the selected high-relevance images. A user-centered evaluation on a large Twitter dataset around several real-world events demonstrates that the proposed method considerably outperforms a number of state-of-the-art summarization algorithms in terms of result relevance, while at the same time it is also highly competitive in terms of diversity. Namely, we get an improvement of 25% in terms of precision compared to the second best result, and 7% in terms of diversity.

Results returned by commercial image search engines should include relevant and diversified depictions of queries in order to ensure good coverage of users' information needs. While relevance has drastically improved in recent years, diversity is still an open problem. In this paper we propose a reranking method that could be implemented on top of such engines in order to provide a better balance between relevance and diversity. Our method formulates the reranking problem as an optimization of a utility function that jointly considers relevance and diversity. Our main contribution is the replacement of the unsupervised definition of relevance that is commonly used in this formulation with a supervised classification model that strives to capture a query and application-specific notion of relevance. This model provides more accurate relevance scores that lead to significantly improved diversification performance. Furthermore, we propose a stacking-type ensemble learning approach that allows combining multiple features in a principled way when computing the relevance of an image. An empirical evaluation carried out on the datasets of the MediaEval 2013 and 2014 "Retrieving Diverse Social Images" (RDSI) benchmarks confirms the superior performance of the proposed method compared to other participating systems as well as a state-of-the-art, unsupervised reranking method.

Our purpose in this work is to boost the performance of object classifiers learned using the self-training paradigm. We exploit the multi-modal nature of tagged images found in social networks, to optimize the process of region selection when retraining the initial model. More specifically, the proposed approach uses a small number of manually labelled regions to train the initial object detection classifiers. Then, a large number of loosely tagged images, pre-segmented by an automatic segmentation algorithm, is used to enhance the initial training set with additional image regions. However, in contrast to the typical case of self-training where the image regions are selected based solely on how well they fit to the original classification model, our approach aims at optimizing this selection by making combined use of both visual and textual information. The experimental results show that the object detection classifiers generated using the proposed approach outperform the classifiers generated using the typical self-training paradigm.

We present a technical demonstration of a novel smartphone application that enables efficient browsing and tagging of landmark and event photos. The application employs a hierarchical mode of exploration enabling zooming from the level of a city, through the level of an area/neighbourhood, down to the level of a specific spot. This navigation mechanism combined with photo clustering, and a dual map-list viewing mechanism, enables efficient browsing of hundreds of thousands of publicly available geotagged photos on a smartphone. Due to the large number of markers on the map, an adaptive clustering strategy is employed to reduce clutter on the screen. In addition to the advanced browsing capabilities, the application enables users to import their personal photos from Flickr and to easily annotate them by propagating to them metadata (location, tags) from the currently viewed object (area, landmark). The application currently supports more than 30 cities worldwide.

A large variety of features can be extracted from raw multimedia items. Moreover, in many contexts, like in the case of multimedia uploaded by users of social media platforms, items may be linked to metadata that can be very useful for a variety of analysis tasks. Nevertheless, such features are typically heterogeneous and are difficult to combine in a unified representation that would be suitable for analysis. In this paper, we discuss the problem of clustering collections of multimedia items with the purpose of detecting social events. In order to achieve this, a novel multimodal clustering algorithm is proposed. The proposed method uses a known clustering in the currently examined domain, in order to supervise the multimodal fusion and clustering procedure. It is tested on the MediaEval social event detection challenge data and is compared to a multimodal spectral clustering approach that uses early fusion. By taking advantage of the explicit supervisory signal, it achieves superior clustering accuracy and additionally requires the specification of a much smaller number of parameters. Moreover, the proposed approach has wider scope; it is not only applicable to the task of social event detection, but to other multimodal clustering problems as well.

We present a technical demonstration of an online city exploration application that helps users identify interesting spots in a city by use of spatio-temporal analysis and clustering of user contributed photos. Our framework analyzes the spatial distribution of large city-centered collections of user contributed photos at different time scales in order to index the most popular spots of a city in a time-aware manner. Subsequently, the photo sets belonging to the same spatiotemporal context are clustered in order to extract representative photos for each spot. The resulting application enables users to obtain flexible summaries of the most important spots in a city given a temporal slice (time of the day, month, season). The demonstration will be based on a photo dataset covering major European cities.

This paper demonstrates a new approach to detecting high-level events that may be depicted in images or video frames. Given a non-annotated content item, a large number of previously trained visual concept detectors are applied to it and their responses are used for representing the content item with a model vector in a high-dimensional concept space. Subsequently, an improved subclass discriminant analysis method is used for identifying a concept subspace within the aforementioned concept space, that is most appropriate for detecting and recognizing the target high-level events. In this subspace, the nearest neighbor rule is used for comparing the non-annotated content item with a few known example instances of the target events. The high-level events used as target events in the present version of the system are those defined for the TRECVID 2010 Multimedia Event Detection (MED) task.

This paper investigates the role of gaze movements as implicit user feedback during interactive video retrieval tasks. In this context, we use a content-based video search engine to perform an interactive video retrieval experiment, during which, we record the user gaze movements with the aid of an eye-tracking device and generate features for each video shot based on aggregated past user eye fixation and pupil dilation data. Then, we employ support vector machines, in order to train a classifier that could identify shots marked as relevant to a new query topic submitted by new users. The positive results provided by the classifier are used as recommendations for future users, who search for similar topics. The evaluation shows that important information can be extracted from aggregated gaze movements during video retrieval tasks, while the involvement of pupil dilation data improves the performance of the system and facilitates interactive video search.

In this paper we present a hybrid recommendation system that combines ontological knowledge with content-extracted linguistic information, derived from pre-trained lexical graphs, in order to produce high quality, personalized recommendations. In the described approach, such recommendations are exemplified in an advertising scenario. We propose a distributed system architecture that uses semantic knowledge, based on terminologically enriched domain ontologies, to learn ontological user profiles and consequently infer recommendations through fuzzy semantic reasoning. A real world user study demonstrates the improvements attained in providing user-relevant recommendations with the aid of semantic profiles.

By analyzing explicit & implicit feedback information retrieval systems can determine topical relevance and tailor search criteria to the user's needs. In this paper we investigate whether it is possible to infer what is relevant by observing user affective behaviour. The sensory data employed range between facial expressions and peripheral physiological signals. We extract a set of features from the signals and analyze the data using classification methods, such as SVM and KNN. The results of our initial evaluation indicate that prediction of relevance is possible, to a certain extent, and implicit feedback models can benefit from taking into account user affective behavior.

Under social tagging systems, a typical Web 2.0 application, users label digital data sources by using freely chosen textual descriptions (tags). Poor retrieval in the aforementioned systems remains a major problem mostly due to questionable tag validity and tag ambiguity. Earlier clustering techniques have shown limited improvements, since they were based mostly on tag co-occurrences. In this paper, a co-clustering approach is employed, that exploits joint groups of related tags and social data sources, in which both social and semantic aspects of tags are considered simultaneously. Experimental results demonstrate the efficiency and the beneficial outcome of the proposed approach in correlating relevant tags and resources.

In large organizations the resources needed to solve challenging problems are typically dispersed over systems within and beyond the organization, and also in different media. However, there is still the need, in knowledge environments, for extraction methods able to combine evidence for a fact from across different media. In many cases the whole is more than the sum of its parts: only when considering the different media simultaneously can enough evidence be obtained to derive facts otherwise inaccessible to the knowledge worker via traditional methods that work on each single medium separately. In this paper, we present a cross-media knowledge extraction framework specifically designed to handle large volumes of documents composed of three types of media text, images and raw data and to exploit the evidence across the media. Our goal is to improve the quality and depth of automatically extracted knowledge.

In this article, an approach to semantic image analysis is presented. Under the proposed approach, ontologies are used to capture general, spatial, and contextual knowledge of a domain, and a genetic algorithm is applied to realize the final annotation. The employed domain knowledge considers high-level information in terms of the concepts of interest of the examined domain, contextual information in the form of fuzzy ontological relations, as well as low-level information in terms of prototypical low-level visual descriptors. To account for the inherent ambiguity in visual information, uncertainty has been introduced in the spatial relations definition. First, an initial hypothesis set of graded annotations is produced for each image region, and then context is exploited to update appropriately the estimated degrees of confidence. Finally, a genetic algorithm is applied to decide the most plausible annotation by utilizing the visual and the spatial concepts definitions included in the domain ontology. Experiments with a collection of photographs belonging to two different domains demonstrate the performance of the proposed approach.