Machine learning models have been extensively used for analyzing Earth Observation images and have played a crucial role in advancing the field. While most studies focus on improving the model’s performance, some aim to understand the model’s output. These explainable approaches provide reasoning behind the model’s output, establishing trust and confidence in the results. However, the evaluation of these models’ performance is mainly based on accuracy. To enhance the fairness and transparency of machine learning models, the evaluation of these models on Earth Observation images should also focus on explainability. This work reflects on existing research on explaninable AI in Remote Sensing and further outlines the desirable properties of the gold standard metric for evaluating explainable machine learning models on EO images.

A vital component of Intelligent Transportation Systems (ITS) is vehicle re-identification, which allows vehicles to be identified across surveillance devices. Re-identification of vehicles is usually done using information collected from standalone surveillance devices such as fixed surveillance cameras (CCTVs) or aerial devices (UAVs). Re-identifying vehicles across standalone surveillance systems is challenging when there is a severe illumination change, a change of viewpoint, or an occlusion. Cross platform surveillance (CCTV+UAV) based vehicle re-identification is yet to be explored and can mitigate some of the challenges faced during re-identifying vehicles with standalone surveillance systems. This paper proposes a novel cross platform vehicle identification dataset called MCU-VReID using 42 CCTVs and a UAV. A novel re-identification method called Multi-Scale Feature Fusion Transformer (MSFFT) is proposed to re-identify vehicles observed across the cross platform surveillance systems. The network consists of inception layers with transformer networks that enable it to learn the vehicle’s features at a variety of scales. The vehicles observed by two contrasting surveillance systems appear to be transformed representations of one another. Hence a two-stage training approach is facilitated for re-identifying vehicles observed across cross platform surveillance systems. The two-stage training approach aims to learn vehicle semantic transformations in the first stage using self-supervised approaches. The knowledge gained at the first stage relating to vehicle semantic transformations is transferred at the second stage of training to perform re-identification. Extensive experiments using the method demonstrate that MSFFT significantly improves over state-of-the-art methods to perform cross platform vehicle re-identification.

With artificial intelligence breakthroughs permeating the Earth Science domain, there is an immediate need to advance the data, tools, and resulting technologies to broader societal challenges. Different efforts are emerging with fragmented best practices for making Earth Observation (EO) data Artificial Intelligence (AI)-ready, availing computer vision and image analysis tools for broader reuse across the remote sensing community. This paper will revisit current best practices and outline a guideline for advancing EO data and derivative AI products for broader community use. We mainly discuss the Analysis Ready Data (ARD) essentials and aim to forge their evolution with Findable, Accessible, Interoperable, Reusable (FAIR) principles to support cross-modal/cross-sensor/cross-provider opportunities that appear to be central to solving complex EO challenges.

The Earth Observation data provides an effective tool to assess post-disaster damage for better relief and rescue efforts management. However, the longer satellite revisit time might delay the rescue efforts. In contrast, Unmanned Aerial Vehicles (UAV) can be rapidly deployed with a customized flight plan. This work focuses on analyzing images from UAV to identify flooded regions. Specifically, a Transformer based semantic segmentation method is proposed for flooded region identification. The proposed encoder-decoder model integrates the features of UNet (ResNet18 backbone) with that of Vision in Transformer (ViT). These fused features are fed to a decoder module to obtain the final segmentation map. The proposed work is evaluated on the FloodNet dataset containing post-disaster UAV images after Hurricane Harvey. A mIoU of 86.84% is obtained using the proposed approach compared to a mIoU of 74.95% using the traditional UNet model. The significant improvement in mIoU demonstrates the robustness of ViT in learning discriminant features for post-disaster scene understanding.

The severity of flooding in a given region is essential in-formation required for better planning and managing post-flood relief and rescue efforts. This work proposes an unsu-pervised segmentation-based approach to estimate the sever-ity of flooding by analyzing images acquired from Unmanned Aerial Vehicles (UAV). In this work, handcrafted texture feature (Local Binary Pattern) is integrated with k-means seg-mentation algorithm to obtain an accurate segmentation of the flooded region. Subsequently, the image is categorized as severely flooded, moderately flooded, minor flooding, and no flooding based on the percentage of pixels belonging to the flooded region in the image. The proposed approach is evaluated on FloodNet dataset containing the UAV aerial images acquired after hurricane Harvey. The experimental re-sults demonstrate that the severity of flooding was correctly estimated in 84.29% of the images illustrating the robustness of the proposed approach. Moreover, the use of handcrafted features along with unsupervised segmentation eliminates the need of manually annotated images. Besides, the proposed unsupervised segmentation approach performs competitively with the deep learning method (UNet) to identify the flooded regions. Therefore, the proposed method could be preferred for analysing the images on-board UAV for post-flood scene understanding.

Last few years have seen a tremendous increase in natural disasters such as earthquakes, floods, hurricanes, etc., primarily due to climate change. Apart from the steps taken to mitigate climate change, there is also a need for rapid and efficient planning and management of post-disaster relief and rescue efforts. Unmanned Aerial Vehicles (UAV) based system offers mobility, rapid deployment and customized flight path for surveying the disaster affected area. Therefore, images acquired from Unmanned Aerial Vehicles (UAV) may be utilized for post-disaster damage assessment. This work summarizes current methods for assessing damage after an earthquake and flood by analyzing optical UAV aerial images. Moreover, this work highlights the challenges encountered and discusses the possible way forward for a robust postdisaster scene understanding from UAV aerial images.

With the increase in natural disasters, there is a need for better management and planning of post-disaster relief and rescue efforts to minimize loss of lives and property. An Unmanned Aerial Vehicle (UAV)-based system offers the advantage of mobility and a customized flight path that could be utilized to survey areas affected by a disaster. However, the images acquired by UAV must be analysed rapidly with minimum user intervention. In this context, the present work compares the performance of traditional handcrafted feature-based classifiers with that of deep learning methods for classifying images as flooded/non-flooded. The pixels corresponding to water in the UAV aerial image exhibit a characteristic texture as compared to roads, greenery etc. This motivated the use of handcrafted texture features (gray-level co-occurrence matrix (GLCM), local binary patterns (LBP)), which were then used to train a Support Vector Machine (SVM) classifier. Besides, Supervised (ResNet18) and Self-Supervised (Sim-CLR) deep learning methods are also studied for classifying UAV aerial images as flooded/non-flooded. The traditional and deep learning methods are compared on FloodNet dataset containing images acquired after hurricane Harvey. An F1 score of 0.84 for flooded class was obtained with the LBP texture classifier, compared to 0.87 using the self-supervised deep learning method. This result demonstrates that a hand-crafted texture-based classifier performs competitively with deep learning methods. Therefore, a traditional texture classifier could be preferred over deep learning methods for a rapid post- flood scene understanding in UAV aerial images.

Existing traffic infrastructure management to handle traffic congestion in major cities is aging and is not effective for traffic monitoring. With an increasing demand for developing cities as smart cities, advanced Intelligent transportation systems (ITS) are in need to improve the safety and traffic movements in the cities. As an application of ITS, vehicle re-identification has gained a wide interest in the field of robotics and computer vision. Currently, these tasks are performed from the data acquired by either of the standalone surveillance systems such as CCTV or UAV. Such data acquired for re-identification poses several challenges namely viewpoints, scale, illumination change, occlusion, etc. To address these, a hybrid surveillance system approach is proposed whereby an algorithm is developed for vehicle re-identification. The re-identification algorithm is tested on a dataset containing 33 identical vehicles observed across 20 CCTV cameras and a UAV. Re-identification of vehicles is performed by estimating a transformation that maps a vehicle observed in one modality to another. The performance of vehicle re-identification is compared for a CNN network trained with the vehicle identities with and without the application of homography.

Video anomaly detection has gained much attention in the computer vision community due to its wide applications in security. Specifically, the focus has been on feature extraction and the design of inference algorithms. The extraction of features to model the normality is challenging due to the scarcity of data and supervision. To this end, current computer vision technologies use reconstruction based methods that relied on auto-encoders to reconstruct normal events in an unsupervised manner. Higher reconstruction errors are often used to detect anomalies. However, the use of multiple auto-encoders to extract features (temporal and appearance) is redundant and expensive for videos. In this context, the present study proposes a novel feature extractor that uses a single CNN architecture to extract both temporal and appearance features. Also, this model is trained for classification tasks which are adapted as feature extractors in anomaly detection. The training of this model is easy and can be deployed efficiently due to its lightweight architecture. Further, the proposed model has been quantitatively evaluated on the UCSD ped 2 dataset and found to perform competitively with an AUC of 0.958.

Natural disasters such as floods cause huge loss of life and property every year. Hence, it is imperative to detect and estimate the magnitude of a flood in a flood-affected area. Besides, it is essential to assess the damage caused by the flood as quickly as possible for an effective post-disaster relief and rescue effort. However, the longer frequency of data acquisition from the existing remote sensing-based methods for post-disaster damage assessment can delay relief. In this work, we propose an approach to estimate the magnitude of the flooded region by analyzing the aerial images acquired from unmanned aerial vehicles (UAV). The proposed method computes two parameters: one based on unsupervised image segmentation and another on image similarity between input and flooded images. These parameters are then utilized to develop a model to estimate the flood magnitude in the aerial image. The proposed approach is evaluated on the FloodNet dataset, and an Fl-score of 0.90 was obtained. demonstrating the proposed algorithm's robustness.

Fish farmers are looking for sustainable methods of fishing to meet the ever-increasing demand for quality aquatic products. However, the water quality parameters, such as temperature, Dissolved Oxygen (DO) and pH plays a significant role in the success of aquaculture. The Dissolved oxygen concentration in the fish farms has a greater influence on the outcome of the aquaculture. DO can vary drastically depending upon many external factors, such as feeding, stocking density, diseases etc. Sudden depletion in DO can result in mass mortality of fishes if the preventive actions are not prompt. To this end, computer vision-based behaviour detection plays a significant role. The present study proposes to develop a novel computer vision-based approach to detect swimming at the surface pattern. An experiment is a setup to capture and develop the dataset of fish movement patterns. The proposed method uses detections alone to identify the swimming at the surface pattern. These detections are clustered and the mean of the clusters are compared against the threshold for classifying the pattern as Swimming at the surface pattern. The threshold is identified using the position histogram from the dataset. The proposed method is efficient, lightweight and reliable making it suitable for deployment in smart systems. The proposed method is also compared with pattern detection using a tracking algorithm. The results highlight the reliability of the proposed method to detect the patterns in aquaculture.

One of the desirable features of a surveillance system is the automatic identification of anomalous events in surveillance videos. The recent approaches for anomalous events identification utilize the difference between the predicted future frame and the current frame to detect the frames with an anomalous event. However, these approaches fare poorly if there is an overlap between multiple objects present in the scene. This work proposes to incorporate two modules to the future frame prediction-based anomalous activity detection approach. The first module penalizes the frame-wise PSNR value if there is an overlap between a normal and an anomalous object. In contrast, the second module penalizes the PSNR value if there is a sudden deviation of the vehicles from its trajectory. This object-centric approach ensures that the anomalous events are correctly identified even in the presence of occlusion. The proposed method is evaluated on two standard datasets Ped 2 and CUHK Avenue. The proposed method outperforms the existing approaches, and an AUC of 96.2% and 85.22% is obtained on Ped2 and CUHK, respectively.

International audience