The dAIEDGE Network of Excellence (NoE) seeks to strengthen and support the development of a dynamic European cutting-edge Artificial intelligence (AI) ecosystem under the umbrella of the European Lighthouse for AI, and to sustain the development of advanced AI. dAIEDGE fosters the exchange of ideas, concepts, and trends on cutting-edge next generation AI, creating links between ecosystem actors to help both the European Commission (EC) and the European Union (EU) and the peripheral AI constituency identify strategies for future developments in Europe. Our main objective is to advance Europe’s innovation and technology base by developing a comprehensive policy and governance approach to AI in order for the EU to become a world leader in innovation in the data economy and its applications.

Detecting anomalies in video surveillance, particularly for firearm detection, remains a critical challenge in public safety systems. Traditional methods often rely on human operators manually monitoring surveillance feeds, which is both inefficient and prone to error. Recent advances in deep learning (DL) offer a promising alternative by enabling models to identify anomalous events without relying on rare and difficult-to-obtain positive samples during training. In this paper, we propose an unsupervised firearm detection framework based on adversarial autoencoders (AAE) networks. By learning a robust representation of normal (negative) training data, the model is able to identify deviations indicative of firearm-related anomalies during inference. Anomaly scores for firearm detection are calculated using reconstruction errors, based on the probability that the test sample aligns with the prior distribution. Our approach enhances the interpretability of firearm related anomaly detection (AD) and demonstrates superior performance on benchmark firearm datasets. Experimental results demonstrate that the proposed method surpasses current state-of-the-art techniques by effectively identifying out-of-distribution (OOD) events in video frames, leveraging learned priors within the AAE architecture. Experimental results on benchmark firearm datasets, including VISILAB, UCF-Firearm, and YouTube demonstrate the effectiveness of our firearm detection approach, achieving an average precision (AP) of 95.2% and an average detection accuracy (ACC) of 95.6%.

ABSTRACT Deep learning (DL) has significantly transformed computer vision, demonstrating remarkable achievements and extensive real‐world applications. However, recent studies have highlighted a critical vulnerability of DL models to adversarial examples (AE), where slight perturbations in input data can lead to erroneous outputs. We observe that the behaviour of the AE is similar to a chaotic system, where a minor change in the input leads to a significantly different output. In response, we propose a novel approach for detecting and categorizing adversarial inputs encountered by classification neural networks. The proposed approach focuses on extracting statistical profiles, termed as chaotic feature vectors (CFVs), from a collection of features derived from ordinal patterns (OP). In this work, the proposed AE detection method is tested on seven attack methods and three image datasets including MNIST, FMNIST and CIFAR10. The results indicate that CFVs exhibit promising capabilities in discerning AE against various types of adversarial attacks on different datasets. This advancement lays the foundation for devising attack mitigation strategies, thereby enhancing the robustness and security of DL models in the face of adversarial threats.

Video surveillance systems can play a critical role in ensuring public safety by assisting in the early detection of potentially dangerous objects or actions, such as people carrying handguns or other weapons. Recent machine learning architectures based on multi-head self-attention modules have demonstrated their ability to process sequential data. In this work, we propose AWARE, a self-Attention based Weapon Activity Recognition architecture for video surveillance systems. The main idea behind this approach is to use a Transformer encoder module to extract relevant features from input video sequences and classify them as either weapon-related actions or non-dangerous actions. The input data is generated by combining 2D human pose keypoints and potential weapon locations generated by object detection models. We evaluate our proposed method on a new action recognition dataset composed of video sequences of gun-related actions. Experiments conducted show that the proposed method achieves better results than other similar methods in this context.

ABSTRACT Adversarial examples are an intriguing and critical topic in the field of machine learning. The impact of malignant perturbations on deep learning‐based systems, especially in safety‐critical applications, highlights a significant security concern. While most research has focused on artificially generated adversarial attacks–crafted through optimization algorithms and constrained perturbations, it is important to note that adversarial examples can also occur naturally, without any artificial manipulation, during the prediction of real‐world images. These naturally occurring adversarial examples pose unique challenges, as they are harder to detect and interpret. Despite their importance, the study of natural adversarial examples remains in its early stages. Fundamental questions remain unanswered: Do natural adversarial examples exhibit similar behaviours or properties as artificially generated ones? How should models be adapted to improve their robustness against such natural inputs? To address these questions, this work proposes an in‐depth analysis of activation maps to compare the internal behaviour of neural networks when processing clean images, artificially perturbed inputs and natural adversarial examples. A set of quantitative metrics is extracted from activation heatmaps at various network layers, including mean activation intensity, centroid displacement and standard reference image quality metrics. These measurements enable a systematic comparison of how the network attends to different image regions under varying conditions. The experimental results demonstrate that natural adversarial examples exhibit statistically significant differences in activation patterns compared to their artificial counterparts, suggesting that they may require distinct strategies for detection and defence.

Abstract The phenomenon of adversarial examples is one of the most attractive topics in machine learning research these days. These are particular cases that are able to mislead neural networks, with critical consequences. For this reason, different approaches are considered to tackle the problem. On the one side, defense mechanisms, such as AutoEncoder-based methods, are able to learn from the distribution of adversarial perturbations to detect them. On the other side, chaos theory and Lyapunov exponents (LEs) have also been shown to be useful to characterize them. This work proposes the combination of both domains. The proposed method employs these exponents to add more information to the loss function that is used during an AutoEncoder training process. As a result, this method achieves a general improvement in adversarial examples detection performance for a wide variety of attack methods.

Early detection of the presence of dangerous objects such as handguns in Closed-Circuit Television (CCTV) images is vital to reduce the potential damage. In this work, a novel method for automatic detection of handguns in CCTV-like images based on a combination architecture which leverages body pose estimation is proposed. Weapon appearance features along with body pose features are combined to perform robust detection in typical surveillance environments where appearance features alone are not sufficient (e.g., because the handgun may appear too small or dark). Both CNN and recent transformer-based architectures are applied for visual feature extraction. Experiments on multiple datasets show that this approach improves state-of-the-art pose-based handgun detectors. An ablation study is also performed to verify the contribution of the pose processing branch and the false positive filter.

Soil-transmitted helminth infections are one of the most common healthcare problems worldwide and they especially affect to the poorest communities in tropical and subtropical areas. Nowadays, diagnosis of intestinal parasites is performed by highly skilled medical staff, directly examining samples in the laboratory via a microscope, a laborious and time-consuming work. Automatic deep learning-based object detection methods can help to automatically detect and identify intestinal parasitic eggs, or at least reduce the workload. In this work, the application of novel Transformer-based architectures is proposed to solve the parasitic egg detection task in microscopic images. Several detection methods and backbones have been analyzed and compared, obtaining up to 0.875 mIoU score on the dataset used for testing.

Automating the analysis of soil parameters can optimize the fertilization process, saving time and reducing the costs of food production, leading to a more sustainable agriculture. The work presented in this paper is part of the HYPERVIEW Challenge: Seeing Beyond the Visible. Several methods are proposed, based both on traditional approaches such as Support Vector Regression (SVR) and k-Nearest Neighbors (k-NN), as well as modern neural networks. A parameterized preprocessing stage has been proposed to deal with the varying size of the input data. The best results have been obtained with the k-NN model and the grid division of the images.

Abstract In an object detection system, the main objective during training is to maintain the detection and false positive rates under acceptable levels when the model is run over the test set. However, this typically translates into an unacceptable rate of false alarms when the system is deployed in a real surveillance scenario. To deal with this situation, which often leads to system shutdown, we propose to add a filter step to discard part of the new false positive detections that are typical of the new scenario. This step consists of a deep autoencoder trained with the false alarm detections generated after running the detector over a period of time in the new scenario. Therefore, this step will be in charge of determining whether the detection is a typical false alarm of that scenario or whether it is something anomalous for the autoencoder and, therefore, a true detection. In order to decide whether a detection must be filtered, three different approaches have been tested. The first one uses the autoencoder reconstruction error measured with the mean squared error to make the decision. The other two use the k-NN (k-nearest neighbors) and one-class SVMs (support vector machines) classifiers trained with the autoencoder vector representation. In addition, a synthetic scenario has been generated with Unreal Engine 4 to test the proposed methods in addition to a dataset with real images. The results obtained show a reduction in the number of false positives between 22.5% and 87.2% and an increase in the system’s precision of 1.2% $$-47$$ - 47 % when the autoencoder is applied.

Abstract Fast automatic handgun detection can be very useful to avoid or mitigate risks in public spaces. Detectors based on deep learning methods have been proposed in the literature to trigger an alarm if a handgun is detected in the image. However, those detectors are solely based on the weapon appearance on the image. In this work, we propose to combine the detector with the individual’s pose information in order to improve overall performance. To this end, a model that integrates grayscale images from the output of the handgun detector and heatmap-like images that represent pose is proposed. The results show an improvement over the original handgun detector. The proposed network provides a maximum improvement of a 17.5% in AP of the proposed combinational model over the baseline handgun detector.

While action recognition has become an important line of research in computer vision, the recognition of particular events such as aggressive behaviors, or fights, has been relatively less studied. These tasks may be extremely useful in several video surveillance scenarios such as psychiatric wards, prisons or even in personal camera smartphones. Their potential usability has led to a surge of interest in developing fight or violence detectors. One of the key aspects in this case is efficiency, that is, these methods should be computationally fast. "Handcrafted" spatiotemporal features that account for both motion and appearance information can achieve high accuracy rates, albeit the computational cost of extracting some of those features is still prohibitive for practical applications. The deep learning paradigm has been recently applied for the first time to this task too, in the form of a 3D Convolutional Neural Network that processes the whole video sequence as input. However, results in human perception of other's actions suggest that, in this specific task, motion features are crucial. This means that using the whole video as input may add both redundancy and noise in the learning process. In this work, we propose a hybrid "handcrafted/learned" feature framework which provides better accuracy than the previous feature learning method, with similar computational efficiency. The proposed method is compared to three related benchmark datasets. The method outperforms the different state-of-the-art methods in two of the three considered benchmark datasets.

Responsible Research and Innovation (RRI) is an approach that anticipates and assesses potential implications and societal expectations with regard to research and innovation, with the aim to foster the design of inclusive and sustainable research and innovation. While RRI includes many aspects, in certain types of projects ethics and particularly privacy, is arguably the most sensitive topic. The objective in Horizon 2020 innovation project Eyes of Things (EoT) is to build a small high-performance, low-power, computer vision platform (similar to a smart camera) that can work independently and also embedded into all types of artefacts. In this paper, we describe the actions taken within the project related to ethics and privacy. A privacy-by-design approach has been followed, and work continues now in four platform demonstrators.

In this paper, we propose a new discriminative framework based on Hough forests that enables us to efficiently recognize and localize sequential data in the form of spatio-temporal trajectories. Contrary to traditional decision forest-based methods where predictions are made independently of its output temporal context, we introduce the concept of "transition", which enforces the temporal coherence of estimations and further enhances the discrimination between action classes. We start applying our proposed framework to the problem of recognizing and localizing fingertip written trajectories in mid-air using an egocentric camera. To this purpose, we present a new challenging dataset that allows us to evaluate and compare our method with previous approaches. Finally, we apply our framework to general human action recognition using local spatio-temporal trajectories obtaining comparable to state-of-the-art performance on a public benchmark.

This paper describes a specific tool for automatically segmenting and archiving of tissue microarray (TMA) cores in microscopy images at different magnifications. TMA enables researchers to extract the small cylinders of a single tissue (core sections) from histological sections and arrange them in an array on a paraffin block such that hundreds can be analyzed simultaneously. A crucial step to improve the speed and quality of this process is the correct localization of each tissue core in the array. However, usually the tissue cores are not aligned in the microarray, the TMA cores are incomplete and the images are noisy and with distorted colors. We develop a robust framework to handle core sections under these conditions. The algorithms are able to detect, stitch, and archive the TMA cores at different magnifications. Once the TMA cores are segmented they are stored in a relational database allowing their processing for further studies of benign-malignant classification. The method was shown to be reliable for handling the TMA cores and therefore enabling further large-scale molecular pathology research.

Automatic facial analysis abilities are commonly integrated in a system by a previous off-line learning stage. In this paper we argue that a facial analysis system would improve its facial analysis capabilities based on its own experience similarly to the way a biological system, i.e. the human system, does throughout the years. The approach described, focused on gender classification, updates its knowledge according to the classification results. The presented gender experiments suggest that this approach is promising, even when just a short simulation of what for humans would take years of acquisition experience was performed.

This paper describes a method based on an energy minimizing deformable model applied to the 3D biomechanical modeling of a set of organs considered as regions of interest (ROI) for radiotherapy. The initial model consists of a quadratic surface that is deformed to the exact contour of the ROI by means of the physical properties of a mass-spring system. The exact contour of each ROI is first obtained using a geodesic active contour model. The ROI is then parameterized by the vibration modes resulting from the deformation process. Once each structure has been defined, the method provides a 3D global model including the whole set of ROIs. This model allows one to describe statistically the most significant variations among its structures. Statistical ROI variations among a set of patients or through time can be analyzed. Experimental results are presented using the pelvic zone to simulate anatomical variations among structures and its application in radiotherapy treatment planning.

The widespread availability of portable computing power and inexpensive digital cameras are opening up new possibilities for retailers in some markets. One example is in optical shops, where a number of systems exist that facilitate eyeglasses selection. These systems are now more necessary as the market is saturated with an increasingly complex array of lenses, frames, coatings, tints, photochromic and polarizing treatments, etc. Research challenges encompass Computer Vision, Multimedia and Human-Computer Interaction. Cost factors are also of importance for widespread product acceptance. This paper describes a low-cost system that allows the user to visualize different glasses models in live video. The user can also move the glasses to adjust its position on the face. The system, which runs at 9.5 frames/s on general-purpose hardware, has a homeostatic module that keeps image parameters controlled. This is achieved by using a camera with motorized zoom, iris, white balance, etc. This feature can be specially useful in environments with changing illumination and shadows, like in an optical shop. The system also includes a face and eye detection module and a glasses management module.

Robotics researchers and cognitive scientists are becoming more and more interested in so-called sociable robots. These machines normally have expressive power (facial features, voice, …) as well as abilities for locating, paying attention to, and addressing people. The design objective is to make robots which are able to sustain natural interactions with people. This capacity falls within the range classed as social intelligence in humans. This position paper argues that the reproduction of social intelligence, as opposed to other types of human ability, may lead to fragile performance, in the sense that tested cases may produce rather different performances to future (untested) cases and situations. This limitation stems from the fact that our social abilities, which appear early in life, are mainly unconscious in origin. This is in contrast with other human abilities that we carry out using conscious effort, and for which we can easily conceive algorithms and representations. This novel perspective is deemed useful for defining the obstacles and limitations of a field that is generating increasing interest. Taking into account the mentioned issues, a development approach suited to the problem is proposed. The use of this approach is demonstrated in the development of CASIMIRO, a robotic head with basic interaction abilities.