Unsupervised image segmentation methods typically assume zero a-priori knowledge about the data semantics. This assumption does hold in many practical scenarios where, although the training data might not be annotated, the target semantic image region classes are known. In these settings, text-driven prompting methods for semantic image segmentation offer noticeable improvement in segmentation accuracy over purely unsupervised approaches. However, such approaches are still limited by: a) inherent text-prompt semantic ambiguity, b) ineffective adaptation to target domain distributions, and c) excessive computational and architectural complexity. To address these shortcomings, we propose the Multiclass Extreme Weak Supervision (MEWS) framework for semantic image segmentation. MEWS assumes the availability of extremely few class-based pixel-level image annotations, e.g., few annotated image pixels per class in very few training images. Such pixel-based image prompts are thereby employed to form image region class prototypes. They can be used to leverage low-complexity unsupervised image segmentation architectures to be trained by our novel prototype-based triplet loss that learns discriminative image features by promoting intra-class image feature compactness while enforcing inter-class feature vector separation. Consequently, the proposed MEWS image segmentation architecture leads to increased weakly supervised training efficiency, bridging the performance gap between supervised and unsupervised image segmentation methods. Our experimental results indicate that the proposed methods compare favorably against text-based prompting image segmentation methods. It yields superior image segmentation accuracy in publicly available image segmentation datasets (e.g., Cityscapes), as well as in Natural Disaster Management (NDM) ones. • A novel Multiclass Extreme Weakly Supervised (MEWS) semantic segmentation framework is proposed that generalizes the original binary EWS DNN architecture by utilizing only sparse, per-class, few-pixel per class labelling. • A class prototype-based triplet loss function is designed that pulls same-class prototype feature vectors together, while pushing mean prototype feature vectors belonging to different classes apart. • A multiclass dynamic thresholding mechanism improves contrastive learning, without additional supervision or manual hyperparameter tuning. • MEWS segmentation excels in accuracy, scaling with annotation, ablation study on loss, validated on NDM Sardinia Wildfire dataset.

This work explores the application of Federated Learning (FL) to Unsupervised Semantic image Segmentation (USS). Recent USS methods extract pixel-level features using frozen visual foundation models and refine them through self-supervised objectives that encourage semantic grouping. These features are then grouped to semantic clusters to produce segmentation masks. Extending these ideas to federated settings requires feature representation and cluster centroid alignment across distributed clients, an inherently difficult task under heterogeneous data distributions in the absence of supervision. To address this, we propose FUSS ( F ederated U nsupervised image S emantic S egmentation) which is, to our knowledge, the first framework to enable fully decentralized, label-free semantic segmentation training. FUSS introduces novel federation strategies that promote global consistency in feature and prototype space, jointly optimizing local segmentation heads and shared semantic centroids. Experiments on both benchmark and real-world datasets, including binary and multi-class segmentation tasks, show that FUSS consistently outperforms local-only client trainings as well as extensions of classical FL algorithms under varying client data distributions. To fully support reproducibility, the source code, data partitioning scripts, and implementation details are publicly available at: https://github.com/evanchar/FUSS • Problem definition of Federated Unsupervised Semantic Segmentation (FUSS). • Various federated aggregation strategies are examined. • FedCC: Novel prototype alignment strategies for heterogeneous clients. • Experimental results show improved performance over federated baselines.

Despite recent advancements, Unsupervised Semantic Segmentation (USS) methods still exhibit a significant performance deficit compared to supervised approaches, particularly in binary semantic segmentation. This limitation arises because, without supervision, USS methods struggle to distinguish foreground from background image regions, particularly when the foreground contains small or uncommon objects. This issue is addressed by our proposed Extremely Weakly Supervised Binary Semantic Segmentation (EWS) framework. EWS expects minimal supervision, consisting only of a small set of one-pixel annotations explicitly belonging to the foreground class across the entire image dataset. Our approach leverages these one-pixel annotations and employs two contrastive losses to map visual transformer features into well-separated foreground and background feature clusters. Additionally, we propose a novel loss function to eliminate the need for hyperparameter tuning of the contrastive loss threshold, by dynamically computing it based on the similarity between the input image features. Even if we employ a single one-pixel annotation, EWS achieves competitive results in binary segmentation tasks while maintaining low computational costs, making it an efficient solution for critical segmentation applications. GitHub Repo: https://github.com/matJTzimas/EWS

Concurrent flame and smoke image region segmentation is a challenging task, particularly when relying on a single imaging modality. Leveraging the combination of visible (RGB) and infrared (IR) modalities in wildfire imaging significantly enhances the accuracy and robustness of fire segmentation. In particular, during dense wildfire smoke incidents, certain image features are only imaged by one modality. Therefore, the two wildfire imaging modalities are inherently complementary. This paper evaluates the effectiveness of RGB and IR image fusion for flame and smoke region segmentation. A novel intermediate image fusion architecture is proposed, built upon a real-time, state-of-the-art DNN semantic segmentation model, augmented with attention mechanisms that promote efficient image modality fusion. Furthermore, a U-Net-like decoder enables accurate spatial reconstruction of the lower-dimensional encoded features. Practical challenges, such as segmentation robustness in the absence of image registration and sensor failures, are also efficiently addressed. Based on our experiments, the proposed DNN segmentation model greatly outperforms existing multimodal DNN architectures in wildfire scenarios in terms of accuracy, while also comparing favorably to stateof-the-art semantic image region segmentation architectures in general urban datasets. Its real-time capabilities and enhanced robustness render it suitable for robotic applications in dynamic, high-stakes segmentation tasks. The code is available at https://gitfront.io/r/dfotiou/eiTd3o9UURjn/RoboFireFuseNetprivate/.

Industrial pipeline inspection in petrochemical refineries is dangerous, expensive, time-consuming and prone to errors. Anomaly detection can play a crucial role towards its automation. Damages in this type of infrastructure are few and can be considered as anomalies (essentially outliers). This paper proposes a novel patch-based Anomaly Detection Network (PADNet), that employs deep learning for detecting insulated pipe damages. It consists of three main components: a) a pipeline segmentation module, b) an image patch proposal module, and c) an anomaly detection module. These components work sequentially first to localize insulated pipelines in the input UAV or ground camera images or video frames and then analyze image patches to detect and localize any damages. Importantly, the anomaly detection module can be trained using undamaged pipeline image data only, hence eliminating the need for costly damaged pipeline image annotation. Experimental results demonstrate the effectiveness of the proposed PADNet method in detecting pipeline damages, making it a promising solution for autonomous industrial infrastructure inspection.

Flood segmentation systems play a crucial role in natural disaster management, particularly for real-time flood monitoring, thus real-time lightweight deep neural network (DNN) models represent the state-of-the-art (SOTA) solution. A neglected aspect during the design of such solutions is that flood segmentation is a computer vision problem where the variance of visual appearance between the foreground (flood) and the background is imbalanced. This paper mitigates this imbalance using Knowledge Distillation (KD), enhancing the performance of real-time SOTA DNN models for flood segmentation in complex and challenging environments. The proposed method employs a Self-KD approach, where a Teacher model, trained on augmented inputs with reduced background variance by exploiting traditional image processing techniques (e.g., blur-ring), guides a Student model operating on real-world data. By consistently processing augmented inputs, the Teacher model facilitates the Student’s ability to learn robust representations, effectively suppressing noisy background elements. Experimental results on a flood dataset demonstrate an improvement of up to 2.5% in mean Intersection over Union (mIoU) over baseline SOTA models which scored 85% mIoU, highlighting the effectiveness of the proposed method. Furthermore, our approach is model-agnostic, consistently improving the performance of various SOTA DNN architectures across different models.

This paper introduces a training loss function tailored for object detection in transformer-based architectures. Our approach addresses the imbalance in ground-truth bounding box sizes during training by implementing a coordinate-based error-weighting mechanism for the L1loss. This modification stabilizes optimization and enhances detection performance, particularly in detection problems requiring bounding boxes of varying sizes within the same image, such as fire/smoke detection applications. By integrating this method into the Real-Time Detection Transformer (RT-DETR), we conduct extensive experiments across three fire/smoke detection datasets and compare our findings against leading real-time object detection algorithms, such as YOLO models. To further validate the generalizability of the proposed loss function, we incorporate it into various DETR-based architectures. Our experiments demonstrate the superior fire detection accuracy of RT-DETR trained with our method across all three datasets while ensuring its effectiveness on more complex datasets. This study not only enhances the capabilities of transformer-based architectures for real-time detection tasks but also contributes to the development of more efficient and reliable fire detection systems.

The security and trustworthiness of participating DNN nodes are often overlooked during the design of modern Decentralized Deep Neural Networks (D-DNN). This paper introduces a shard-based distributed consensus protocol specifically tailored for DNN nodes operating over unreliable communication links. The proposed approach enhances D-DNN scalability, by enabling D-DNN systems having a large number of DNN nodes. This is achieved through a hierarchical consensus mechanism that partitions the D-DNN network into sub-networks (shards), leveraging Out-of-Distribution (OOD) detectors to localize and isolate the consensus process within each shard. Rather than randomly allocating DNN nodes into shards, the OOD detector can be employed to identify and group nodes with similar domain knowledge. This approach improves the overall D-DNN system robustness, by identifying and isolating malicious DNN nodes or once that have poor performance for a specific DNN task. Experimental results demonstrate improvements in the D-DNN system’s classification accuracy and reliability.

In human societies, individuals make their own decisions and they may select if and who may influence it, by e.g., consulting with people of their acquaintance or experts of a field. At a societal level, the overall knowledge is preserved and enhanced by individual person empowerment, where complicated consensus protocols have been developed over time in the form of societal mechanisms to assess, weight, combine and isolate individual people opinions. In distributed machine learning environments however, individual AI agents are merely part of a system where decisions are made in a centralized and aggregated fashion or require a fixed network topology, a practice prone to security risks and collaboration is nearly absent. For instance, Byzantine Failures may tamper both the training and inference stage of individual AI agents, leading to significantly reduced overall system performance. Inspired by societal practices, we propose a decentralized inference strategy where each individual agent is empowered to make their own decisions, by exchanging and aggregating information with other agents in their network. To this end, a “Quality of Inference” consensus protocol (QoI) is proposed, forming a single commonly accepted inference rule applied by every individual agent. The overall system knowledge and decisions on specific manners can thereby be stored by all individual agents in a decentralized fashion, employing e.g., blockchain technology. Our experiments in classification tasks indicate that the proposed approach forms a secure decentralized inference framework, that prevents adversaries at tampering the overall process and achieves comparable performance with centralized decision aggregation methods. • In human societies, individuals make their own decisions taking expert advice into account. • This process is simulated in decentralized DNN Inference settings. • A novel consensus protocol working as a single inference rule was developed. • A fault-tolerant inference architecture in which misbehaving AI agents are penalized.

Abstract Visual object tracking is crucial for numerous applications ranging from smartphones to autonomous vehicles. However, the impact of input noise on tracking performance remains underexplored. This paper presents a lightweight neural network module designed to enhance the robustness of 2D tracking methods against various types of noise. By performing image-to-image translation, the proposed robust tracking module (RTM) standardizes the operational space of tracking algorithms, thereby improving their resilience. Experimental results on benchmark datasets demonstrate the effectiveness of RTM in mitigating performance degradation caused by noise. Additionally, we introduce an evaluation toolkit that facilitates the assessment of tracking robustness against common noise types. The source code of the proposed method is available at https://github.com/iason1907/RTM .

This paper presents a unified system tailored for autonomous pipe segmentation within an industrial setting. To this end, it is designed to analyze RGB images captured by Unmanned Aerial Vehicle (UAV)-mounted cameras to predict binary pipe segmentation maps. The overall proposed system consists of three main components: a) a Convolutional Neural Network (CNN) that is used to obtain initial estimates of the pipe segmentation maps, b) a point extraction module that acts on the outputs of the CNN to propose strong pipe class representatives in the input image space, and c) a foundation segmentation model, utilized to refine the initial estimations based on the proposed pipe class representatives. The architecture of the proposed system was specifically designed to ensure increased generalization ability in different, unknown environments, offering an effective solution to a well-known limitation of typical segmentation CNNs, at least in the pipe segmentation task. The effectiveness of the proposed system in this particular setting is evaluated by utilizing two pipe segmentation datasets, originating from two different industrial sites, which were manually annotated with the corresponding pipe segmentation maps. Experimental results demonstrate that the proposed system outperforms the baseline segmentation CNNs, demonstrating its remarkable generalization capabilities.

In the realm of machine learning systems, achieving consensus among networking nodes is a fundamental yet challenging task. This paper presents Proof of Quality Inference (PoQI), a novel consensus protocol designed to integrate deep learning inference under the basic format of the Practical Byzantine Fault Tolerant (P-BFT) algorithm. PoQI is applied to Deep Neural Networks (DNNs) to infer the quality and authenticity of produced estimations by evaluating the trustworthiness of the DNN node’s decisions. In this manner, PoQI enables DNN inference nodes to reach a consensus on a common DNN inference history in a fully decentralized fashion, rather than relying on a centralized inference decision-making process. Through PBFT adoption, our method ensures byzantine fault tolerance, permitting DNN nodes to reach an agreement on inference validity swiftly and efficiently. We demonstrate the efficacy of PoQI through theoretical analysis and empirical evaluations, highlighting its potential to forge trust among unreliable DNN nodes.

Recently, climate change has led to more frequent extreme weather events, introducing new challenges for Natural Disaster Management (NDM) organizations. This fact makes the employment of modern technological tools such as Deep Neural Networks-based fire detectors a necessity, as they can assist such organizations manage these extreme events more effectively. In this work, we argue that the mean Average Precision (mAP) metric that is commonly used to evaluate typical object detection algorithms can not be trusted for the fire detection task, due to its high dependence on the employed data annotation strategy. This means that the mAP score of a fire detection algorithm may be low even when it predicts fire bounding boxes that accurately enclose the depicted fires. In this direction, a new evaluation metric for fire detection is proposed, denoted as Image-level mean Average Precision (ImAP), which reduces the dependence on the bounding box annotation strategy by rewarding/penalizing bounding box predictions on image level, rather than on bounding box level. Experiments using different object detection algorithms have shown that the proposed ImAP metric reveals the true fire detection capabilities of the tested algorithms more effectively.

This work examines the problem of increasing the robustness of deep neural network-based image classification systems to adversarial attacks, without changing the neural architecture or employ adversarial examples in the learning process. We attribute their famous lack of robustness to the geometric properties of the deep neural network embedding space, derived from standard optimization options, which allow minor changes in the intermediate activation values to trigger dramatic changes to the decision values in the final layer. To counteract this effect, we explore optimization criteria that supervise the distribution of the intermediate embedding spaces, in a class-specific basis, by introducing and leveraging one-class classification objectives. The proposed learning procedure compares favorably to recently proposed training schemes for adversarial robustness in black-box adversarial attack settings.

Power line segmentation is a critical component of UAV intelligent inspection systems to ensure the safe and reliable operation of power grids. For challenging-to-label tasks like this, simulators can efficiently generate large amounts of labeled data. In this work, a large-scale annotated synthetic power lines dataset generated utilizing the unity game engine and the unity perception package1. To address domain shift between real and synthetic domain, input-level adaptation performed. Additionally, a new power line segmentation loss developed to mitigate the effects of unbalanced pixel distributions among power lines and background. Experiments demonstrate that our approach achieves state-of-the-art performance on power line segmentation task.

In recent years, the field of automated aerial cinematography has seen a significant increase in demand for real-time 3D target geopositioning for motion and shot planning. To this end, many of the existing cinematography plans require the use of complex sensors that need to be equipped on the subject or rely on external motion systems. This work addresses this problem by combining monocular visual target detection and tracking with a simple ground intersection model. Under the assumption that the targets to be filmed typically stand on the ground, 3D target localization is achieved by estimating the direction and the norm of the look-at vector. The proposed algorithm employs an error estimation model that accounts for the error in detecting the bounding box, the height estimation errors, and the uncertainties of the pitch and yaw angles. This algorithm has been fully implemented in a heavy-lifting aerial cinematography hexacopter, and its performance has been evaluated through experimental flights. Results show that typical errors are within 5 meters of absolute distance and 3 degrees of angular error for distances to the target of around 100 meters.

This work studies and defines the problem of providing extensive and opportunistic Edge AI-based area coverage in smart city application scenarios, by researching and determining the optimal configuration of sensing and computational resources for minimizing the environmental/technology footprint of the solution. A typical smart city computing continuum consists of statically installed multimodal sensing Internet-of-Things (IoT) nodes at various city locations, accompanied by interconnected computational Cloud/Edge/IoT nodes. This paper presents Optimal Trustworthy EdgeAI (OTE), an entirely novel research pipeline, that complements existing smart city infrastructure with intelligent drone Edge/IoT nodes (in the form of modularly equipped unmanned aerial vehicles), capable of autonomous repositioning according to individual/collective sensing and coverage criteria. Thereby, we envisage the emerging cutting-edge technologies of trustworthy sensing, perceiving, modelling technologies for predicting the behavior of moving targets (e.g., citizens/vehicles/objects), understanding natural phenomena (e.g., sea wave motion, urban flora/fauna, biodiversity) in order to anticipate events (people's bad habits, environmental changes), by exploiting novel continuous data processing services across the whole span of the enhanced Cloud-Edge-IoT computing continuum.

Powerline inspection operations involve capturing and inspecting visual footage of powerline elements from elevated positions above and around the powerline and are currently performed with the help of helicopters and/or Unmanned Aerial Vehicles (UAVs). Current technological advances in the areas of robotics and machine learning are towards enabling fully autonomous operations. To this end, one of the tasks to be addressed is the robust, precise and fast powerline object detection problem. Recently introduced Transformer-based object detection methods demonstrate time and accuracy advances with respect to previous works. In this work, we present an enhanced Transformer-based architecture that further improves the state-of-the-art by incorporating a content-specific object query generator and by substituting the original attention operation with a whitening-inspired transformation at certain stages of the architecture. We evaluate our method in a recently captured powerline detection dataset and we show that our novel contributions offer a significant boost regarding detection accuracy.

Online tracking model updating is typically addressed as a regression problem, involving the minimization of the dispersion between the obtained tracker model response maps in each consecutive frame and some target distribution (e.g., Gaussian), using a closed-form solution. Inspired by the recent applications of Generative Adversarial Networks (GANs), we propose to solve this problem with an adversarial optimization scheme, by employing a GeneratorDiscriminator network pair. That is, the role of the Generator is assigned to the tracking model so that it produces response maps belonging to some target distribution, while an additional discriminator network is trained to identify if the tracker response maps produced by the generator belong to this target distribution, or not. Therefore, the tracker model exploits the discriminator network as an additional information pool about the target distribution. It is shown that this simple addition improves tracking performance in standard benchmark datasets, without significantly hurting training complexity, thus rendering the proposed method suitable for embedded system application such as in autonomous cars and Unmanned Aerial Systems.

This paper presents a long-term 2D tracking framework for the coverage of live outdoor (e.g., sports) events that is suitable for embedded system application (e.g. Unmanned Aerial Vehicles). This application scenario requires 2D target (e.g., athlete, ball, bicycle, boat) tracking for visually assisting the UAV pilot (or cameraman) to maintain proper target framing, or even for actual 3D target following/localization when the drone flies autonomously. In these cases, it should be expected that the target to be tracked/followed, may disappear from the UAV camera field of view, due to fast 3D target motion, illumination changes, or due to visual target occlusions by obstacles, even if the actual UAV continues following it (either autonomously, by exploiting alternative target localization sensors, or by pilot maneuvering). Therefore, the 2D tracker should be able to recover from such situations. The proposed framework solves exactly this problem. Target occlusions are detected from the 2D tracker responses. Depending on the occlusion immensity, the proposed framework decides whether to not update the tracking model, or to employ target re-detection in a broader window. As a result, the proposed framework allows continued target tracking once the target re-appears in the video stream, without tracker re-initialization.

Synthetic 3D object models have been proven crucial in object pose estimation, as they are utilized to generate a huge number of accurately annotated data. The object pose estimation problem is usually solved for images originating from the real data domain by employing synthetic images for training data enrichment, without fully exploiting the fact that synthetic and real images may have different data distributions. In this work, we argue that 3D object pose estimation problem is easier to solve for images originating from the synthetic domain, rather than the real data domain. To this end, we propose a 3D object pose estimation framework consisting of a two-step process, where a novel pose-oriented image-to-image translation step is first employed to translate noisy real images to clean synthetic ones and then, a 3D object pose estimation method is applied on the translated synthetic images to finally predict the 3D object poses. A novel pose-oriented objective function is employed for training the image-to-image translation network, which enforces that pose-related object image characteristics are preserved in the translated images. As a result, the pose estimation network does not require real data for training purposes. Experimental evaluation has shown that the proposed framework greatly improves the 3D object pose estimation performance, when compared to state-of-the-art methods.

A classification method that emphasizes on learning the hyperplane that separates the training data with the maximum margin in a regularized space, is presented. In the proposed method, this regularized space is derived by exploiting multiple graph structures, in the SVM optimization process. Each of the employed graph structure carries some information concerning a geometric or semantic property about the training data, e.g., local neighborhood area and global geometric data relationships. The proposed method introduces information from each graph type to the standard SVM objective, as a projection of the SVM hyperplane to such a direction, where a specific property of the training data is highlighted. We show that each data property can be encoded in a regularized kernel matrix. Finally, response in the optimal classification space can be obtained by exploiting a weighted combination of multiple regularized kernel matrices. Experimental results in face recognition and object classification denote the effectiveness of the proposed method.

Autonomous UAV cinematography is an active research field with exciting potential for the media industry. It bears the promise of greatly facilitating UAV shooting for various applications, while significantly reducing the costs compared to manual shooting. However, the general problem has not been clearly defined and the challenges arising from current legislation and technology restrictions have not been fully charted. A complete overview of issues related to autonomous UAV cinematography is needed, pertaining to the current situation in the field, so as to guide immediate-future research. The purpose of this paper is to lay exactly this groundwork, with the expectation of providing a global perspective to multiple domain-specific research communities. The outlined issues are partitioned into challenges deriving from ethical/legal/safety considerations and from operational/production requirements. A brief survey of current technological solutions, including their limitations, is also provided for each issue.

In this paper, we apply One-Class Classification methods in facial image analysis problems. We consider the cases where the available training data information originates from one class, or one of the available classes is of high importance. We propose a novel extension of the One-Class Extreme Learning Machines algorithm aiming at minimizing both the training error and the data dispersion and consider solutions that generate decision functions in the ELM space, as well as in ELM spaces of arbitrary dimensionality. We evaluate the performance in publicly available datasets. The proposed method compares favourably to other state-of-the-art choices.

In this paper, we describe a one-class classification method based on Support Vector Data Description, which exploits multiple graph structures in its optimization process. We derive in a generic solution which can be employed for supervised one-class classification tasks. The devised method can produce linear or non-linear decision functions, depending on the adopted kernel function. In our experiments, we simultaneously adopted two graphs that describe local and global geometric training data relationships, respectively. We evaluated the proposed classifier in publicly available datasets, where its performance compared favorably against closely related methods.

In this paper, we focus on studying the effects of various image operations on sensor fingerprint camera identification. It is known that artifacts in the image processing pipeline, such as pixel defects or unevenness of the responses in the CCD array as well black current noise leave telltale footprints. Nowadays, camera identification based on the analysis of these artifacts is a well established technology for linking an image to a specific camera. The sensor fingerprint is estimated from images taken from a device. A similarity measure is deployed in order to associate an image with the camera. However, when the images used in the sensor fingerprint estimation have been processed using e.g. gamma correction, contrast enhancement, histogram equalization or white balance, the properties of the detection statistic change, hence affecting fingerprint detection. In this paper we study this effect experimentally, towards quantifying the robustness of fingerprint detection in the presence of image processing operations.

A novel OCC method for human action recognition namely the Laplacian One Class Extreme Learning Machines is presented. The proposed method exploits local geometric data information within the OC-ELM optimization process. It is shown that emphasizing on preserving the local geometry of the data leads to a regularized solution, which models the target class more efficiently than the standard OC-ELM algorithm. The proposed method is extended to operate in feature spaces determined by the network hidden layer outputs, as well as in ELM spaces of arbitrary dimensions. Its superior performance against other OCC options is consistent among five publicly available human action recognition datasets.

In this paper, we propose a method for video summarization based on human activity description. We formulate this problem as the one of automatic video segment selection based on a learning process that employs salient video segment paradigms. For this one-class classification problem, we introduce a novel variant of the One-Class Support Vector Machine (OC-SVM) classifier that exploits subclass information in the OC-SVM optimization problem, in order to jointly minimize the data dispersion within each subclass and determine the optimal decision function. We evaluate the proposed approach in three Hollywood movies, where the performance of the proposed SOC-SVM algorithm is compared with that of the OC-SVM. Experimental results denote that the proposed approach is able to outperform OC-SVM-based video segment selection.