Existing approaches for the problem of ultrasound image segmentation, whether supervised or semi-supervised, are typically specialized for specific anatomical structures or tasks, limiting their practical utility in clinical settings. In this paper, we pioneer the task of universal semi-supervised ultrasound image segmentation and propose ProPL, a framework that can handle multiple organs and segmentation tasks while leveraging both labeled and unlabeled data. At its core, ProPL employs a shared vision encoder coupled with prompt-guided dual decoders, enabling flexible task adaptation through a prompting-upon-decoding mechanism and reliable self-training via an uncertainty-driven pseudo-label calibration (UPLC) module. To facilitate research in this direction, we introduce a comprehensive ultrasound dataset spanning 5 organs and 8 segmentation tasks. Extensive experiments demonstrate that ProPL outperforms state-of-the-art methods across various metrics, establishing a new benchmark for universal ultrasound image segmentation.

Multimodal retrieval enables effective access to information across diverse modalities such as text, images, audio, and video. Audio–Text Retrieval (ATR), which retrieves relevant text given an audio query, is a key component in real-world applications including voice-enabled AI assistants and audio-integrated retrieval systems. Existing ATR methods primarily focus on aligning audio and textual semantics in a shared embedding space through contrastive learning or multimodal large language model backbones, while their robustness under realistic acoustic conditions remains largely unexplored. To bridge this gap, we introduce RoATR, a diagnostic benchmark for evaluating the Robustness of ATR systems, inspired by prior robustness studies in automatic speech recognition. RoATR defines five perturbation categories that cover eleven realistic perturbation types, including background noise, speaker variability, and device distortion. Extensive experiments on state-of-the-art ATR models reveal substantial and heterogeneous performance degradation across perturbation types. For example, LCO-Embedding suffers up to a 28.0% absolute drop in Recall@1 while no degradation under city noise. To improve the robustness of ATR models, we further propose perturbation-augmented training based on perturbed data curated following the same pipeline as RoATR, which achieves an average improvement of 16.0 percentage points in Recall@1.

Weakly supervised anomaly detection methods (WADMs) can effectively utilize incomplete label data to address the issue of imbalanced samples, thereby reducing the reliance on the quantity of labeled data and demonstrating superior anomaly detection capabilities in network and service management. However, when confronted with significant label noise and missing labels, existing WADMs still struggle to adequately extract the deep feature information of samples, leading to a decline in model performance. To tackle this challenge, we propose a self-contrastive enhanced weakly supervised anomaly detection framework, called SEAD-Net, which enhances the feature representation capability of data in the model’s feature space, thereby improving the accuracy and robustness of anomaly detection. Specifically, we first design a personalized data enhancement module that augments data representation by applying various transformations to the raw data. Subsequently, a self-contrastive enhanced learning module is introduced to impose hybrid constraints on the augmented samples, constructing the overall distribution structure while learning deep sample feature spaces under complex scenario disturbances. Finally, we extract contrastive enhancement features within the deep sample feature space and perform probabilistic generation to enable effective decision-making via an anomaly probability generation module. Experimental results on a series of public benchmark datasets demonstrate that our SEAD-Net outperforms the second-best WADM by 5.95% in average AUC-ROC and 16.28% in average AUC-PR.

Recently, there has been considerable exploration of methods for generating 3D point clouds, which is crucial for numerous 3D vision applications. Though conditional generation methods show promising performance, it depends on the additional paired label. On the other hand, unconditional generation methods usually fail to annotate the generated 3D point cloud. In this paper, we introduce a novel self-conditional architecture that trains on unlabeled data and then generates high-quality labeled 3D point clouds. Specifically, we design a module to extract geometry and view features, and then use a feature fusion module to integrate them as a substitute for label embedding in conditional point cloud generation. Then the point cloud generator is trained using the fused features. LPCG also harnesses CLIP to handle the view features of point clouds for generating label information. Besides, we train two feature diffusion modules to capture the essence of multimodal features and obtain diverse fused features for use as conditions in generating point clouds. Experiments on the ShapeNet dataset demonstrate that LPCG achieves state-of-the-art performance for single class generation. Our experimental results show that the accuracy of our generated label annotations reaches around 97.44% for a two-class generation task.

In this work, we address the challenge of adaptive pediatric Left Ventricular Ejection Fraction (LVEF) assessment. While Test-time Training (TTT) approaches show promise for this task, they suffer from two significant limitations. Existing TTT works are primarily designed for classification tasks rather than continuous value regression, and they lack mechanisms to handle the quasi-periodic nature of cardiac signals. To tackle these issues, we propose a novel Quasi-Periodic Adaptive Regression with Test-time Training (Q-PART) framework. In the training stage, the proposed Quasi-Period Network decomposes the echocardiogram into periodic and aperiodic components within latent space by combining parameterized helix trajectories with Neural Controlled Differential Equations. During inference, our framework further employs a variance minimization strategy across image augmentations that simulate common quality issues in echocardiogram acquisition, along with differential adaptation rates for periodic and aperiodic components. Theoretical analysis is provided to demonstrate that our variance minimization objective effectively bounds the regression error under mild conditions. Furthermore, extensive experiments across three pediatric age groups demonstrate that Q-PART not only significantly outperforms existing approaches in pediatric LVEF prediction, but also exhibits strong clinical screening capability with high mAUROC scores (up to 0.9747) and maintains gender-fair performance across all metrics, validating its robustness and practical utility in pediatric echocardiography analysis. The project can be found in Q-PART.

Despite advancements in Computer-Aided-Design (CAD) generation, direct generation of complex Boundary Representation (B-rep) CAD models remains challenging. The difficulty arises from the parametric nature of B-rep data, complicating the encoding and generation of its geometric and topological information. In this paper, we introduce BrepGiff, a lightweight generation approach for high-quality and complex B-rep based on 3D Graph Diffusion. First, we transfer B-rep models into 3D graphs representation. Specifically, BrepGiff extracts and integrates topological and geometric features to construct a 3D graph where nodes correspond to face centroids in 3D space, preserving adjacency and degree information. Geometric features are derived by sampling points in the UV domain and extracting face and edge features. BrepGiff then applies Graph Attention Network (GAT) to enforce topological constraints from local to global during the degree-guided diffusion process. With the 3D graph representation and diffusion process, BrepGiff significantly reduces the computational cost and improves the quality, thus achieving lightweight generation of complex models. Experiments show that BrepGiff can generate complex B-rep models (>100 faces) using only 2 RTX4090 GPUs, achieving state-of-the-art performance in B-rep generation.

Unsupervised anomaly detection using deep learning has garnered significant research attention due to its broad applicability, particularly in medical imaging where labeled anomalous data are scarce. While earlier approaches leverage generative models like autoencoders and generative adversarial networks (GANs), they often fall short due to overgeneralization. Recent methods explore various strategies, including memory banks, normalizing flows, self-supervised learning, and knowledge distillation, to enhance discrimination. Among these, knowledge distillation, particularly reverse distillation, has shown promise. Following this paradigm, we propose a novel scale-aware contrastive reverse distillation model that addresses two key limitations of existing reverse distillation methods: insufficient feature discriminability and inability to handle anomaly scale variations. Specifically, we introduce a contrastive student-teacher learning approach to derive more discriminative representations by generating and exploring out-of-normal distributions. Further, we design a scale adaptation mechanism to softly weight contrastive distillation losses at different scales to account for the scale variation issue. Extensive experiments on benchmark datasets demonstrate state-of-the-art performance, validating the efficacy of the proposed method. The code will be made publicly available.

The prevalence of motion sickness among passengers using personal electronic devices, such as smartphones, during vehicle journeys has become a growing concern. This issue is expected to intensify with the increasing adoption of assistant or automated driving functions, which may lead to non-driving tasks (NDT) being performed by all on-board passengers, including the user in the “driver” seat during conditionally or fully automated driving modes. This trend presents challenges related to motion sickness, particularly in terms of specific performance requirements for non-driving tasks. In response to the need to alleviate passenger motion sickness, we have developed an easy-to-understand animation cue app that can be conveniently implemented on smartphones. The motion cue conveys information about vehicle accelerations, including their directions and magnitudes, using the metaphors of traffic signal colors and backward-moving lane lines, either in straight or curved lane driving. Following several rounds of improvements based on moving-base simulator and real car experiments, finally a successful cue design was found, which could significantly alleviate motion sickness of passengers while engaging in NDT, with minimal impact on their NDT performances. However, the study also revealed limitations to the sickness-alleviating capability of our motion cue design, including potential lack of universal acceptance among different users and reduced effectiveness in severely uncomfortable driving conditions. This work may provide valuable insights for further visual cue improvements that can contribute in future carsickness-proof vehicles.

Accurate extraction of building footprints from satellite imagery is of high value. Currently, deep learning methods are predominant in this field due to their powerful representation capabilities. However, they generally require extensive pixel-wise annotations, which constrains their practical application. Semi-supervised learning (SSL) significantly mitigates this requirement by leveraging large volumes of unlabeled data for model self-training (ST), thus enhancing the viability of building footprint extraction. Despite its advantages, SSL faces a critical challenge: the imbalanced distribution between the majority background class and the minority building class, which often results in model bias toward the background during training. To address this issue, this article introduces a novel method called DeBiased matching (DBMatch) for semi-supervised building footprint extraction. DBMatch comprises three main components: 1) a basic supervised learning module (SUP) that uses labeled data for initial model training; 2) a classical weak-to-strong ST module that generates pseudo-labels from unlabeled data for further model ST; and 3) a novel logit debiasing (LDB) module that calculates a global logit bias between building and background, allowing for dynamic pseudo-label calibration. To verify the effectiveness of the proposed DBMatch, extensive experiments are performed on three public building footprint extraction datasets covering six global cities in SSL setting. The experimental results demonstrate that our method significantly outperforms some advanced SSL methods in semi-supervised building footprint extraction. Our codes will be publicly provided athttps://github.com/zhu-xlab/SSL_Buildings.

Automatic building footprint extraction from optical remote sensing (RS) images is popular and crucial for various downstream applications. Current building footprint extraction methods are mainly based on deep learning, which requires large amounts of manually labeled data for model training, limiting their practical deployment. Semi-supervised semantic segmentation (SSS), which leverages limited labeled data for supervised learning and abundant unlabeled data for unsupervised self-training, offers a promising solution to reduce this reliance. Nonetheless, directly applying existing SSS methods to building footprint extraction with limited labels fails to fully exploit the geometric structural features of buildings—key characteristics that distinguish them from background. To tackle this challenge, we propose a semi-supervised learning framework, HeightMatch, which integrates real or synthetic height information with RS images to extract more comprehensive and discriminative feature representations of buildings, particularly in limited-label scenarios. During training, these height maps effectively enhance the model’s ability to capture geometric structures, leading to more accurate pseudo-labels for unlabeled data and thereby enabling more effective self-training. At inference, building predictions rely solely on RS images, ensuring the practicality of the proposed method. Extensive experimental results on five widely-used building footprint extraction datasets demonstrate the effectiveness and superiority of our method in comparison with multiple state-of-the-art SSS methods. Our code is available at https://github.com/zhu-xlab/HeightMatch.

This paper addresses the challenge of mapping polygonal buildings from remote sensing images and introduces a novel algorithm, the Global Collinearity-aware Polygonizer (GCP). GCP, built upon an instance segmentation framework, processes binary masks produced by any instance segmentation model. The algorithm begins by collecting polylines sampled along the contours of the binary masks. These polylines undergo a refinement process using a transformer-based regression module to ensure they accurately fit the contours of the targeted building instances. Subsequently, a collinearity-aware polygon simplification module simplifies these refined polylines and generate the final polygon representation. This module employs dynamic programming technique to optimize an objective function that balances the simplicity and fidelity of the polygons, achieving globally optimal solutions. Furthermore, the optimized collinearity-aware objective is seamlessly integrated into network training, enhancing the cohesiveness of the entire pipeline. The effectiveness of GCP has been validated on three public benchmarks for polygonal building mapping. Further experiments reveal that applying the collinearity-aware polygon simplification module to arbitrary polylines, without prior knowledge, enhances accuracy over traditional methods such as the Douglas-Peucker algorithm. This finding underscores the broad applicability of GCP. The code for the proposed method will be made available at https://github.com/zhu-xlab/GCP.

Earth observation (EO) has inevitably entered the Big Data era. The computational challenge associated with analyzing large EO data using sophisticated deep learning models has become a significant bottleneck. To address this challenge, there has been a growing interest in exploring quantum computing as a potential solution. However, the process of encoding EO data into quantum states for analysis potentially undermines the efficiency advantages gained from quantum computing. This article introduces a hybrid quantum deep learning model that effectively encodes and analyzes EO data for classification tasks. The proposed model uses an efficient encoding approach called superpixel encoding, which reduces the quantum resources required for large image representation by incorporating the concept of superpixels. To validate the effectiveness of our model, we conducted evaluations on multiple EO benchmarks, including Overhead-MNIST, So2Sat LCZ42, and SAT-6 datasets. In addition, we studied the impacts of different interaction gates and measurements on classification performance to guide model optimization. The experimental results suggest the validity of our model for accurate classification of EO data. Our models and code are available on https://github.com/zhu-xlab/SEQNN.

Urban land cover classification aims to derive crucial information from earth observation data and categorize it into specific land uses. To achieve accurate classification, sophisticated machine learning models trained with large earth observation data are employed, but the required computation power has become a bottleneck. Quantum computing might tackle this challenge in the future. However, representing images into quantum states for analysis with quantum computing is challenging due to the high demand for quantum resources. To tackle this challenge, we propose a hybrid quantum neural network that can effectively represent and classify remote sensing imagery with reduced quantum resources. Our model was evaluated on the Local Climate Zone (LCZ)-based land cover classification task using the TensorFlow Quantum platform, and the experimental results indicate its validity for accurate urban land cover classification.

Non-intrusive estimation of building energy efficiency has profound applications in advancing sustainability in the built environment. Recent studies often focus on predicting energy performance alone, neglecting the interplay between the performance and related building semantics. This paper investigates whether incorporating semantic attributes benefits energy efficiency estimation. We develop a neural network to estimate energy efficiency, with building age and usage type as additional supervision for multi-task learning. The neural network processes both aerial imagery and airborne LiDAR data to classify buildings as energy-efficient or inefficient. Our results demonstrate the effectiveness of the superimposed semantics, particularly with building age. With the multi-task model achieving a 63.78% F1 score and outperforming that supervised solely with energy efficiency by 2.86%, this paper reveals the potential of integrating semantic attributes in modeling building energy performance.

Recognizing building attributes from remote sensing images is crucial for various applications. Recent developments in deep learning have demonstrated promising results in identifying these attributes. Nonetheless, a major challenge is the requirement for extensive and accurate building attribute data. Two primary data sources are commonly considered: Open-StreetMap (OSM), which offers global building information but often lacks completeness and correctness, and cadastral data, known for its high quality but typically restricted to certain areas. These two sources enable comparison between deep learning models trained on noisy and incomplete OSM data and those trained on accurate and complete cadastral data. In this work, comprehensive experiments on buildings in Bavaria, Germany, are conducted, covering diverse attributes such as footprints, use, and height. A large building dataset with corresponding building attribute labels from OSM and cadastral data is created, with OSM data featuring varying levels of incompleteness and noise for different attributes and cadastral data serving as ground truth. Moreover, we evaluate the effectiveness of several prevailing methods designed to handle noisy and incomplete labels, assessing their applicability to real-world scenarios with incomplete and noisy OSM labels.

In Earth observation, semantic understanding of Remote Sensing (RS) images holds significant importance, yet it is hindered in practice by the need for extensive manual pixel-level labeling. Semi-supervised semantic segmentation (SSS) of RS images would be a promising solution, which fully utilizes unlabeled data for model self-training under the guidance of limited labeled data. The mainstream SSS methods use pseudo-labels of the unlabeled data for model training, however, their performance is bottlenecked because of confirmation bias, i.e., stubborn incorrect pseudo-labels. To counter this, our study introduces a novel disentanglement learning (DL) method tailored for RS-SSS. It separates the predictions of the labeled and unlabeled data by two individual prediction heads during current training, and then integrates them during follow-up training. The experimental results verify its effectiveness on two widely-used RS semantic segmentation datasets in semi-supervised setting.

Compared to optical sensors, Synthetic Aperture Radar (SAR) can acquire remote sensing imagery under all-weather and all-time conditions. This technology is extensively applied in maritime vessel detection. Recently, deep learning approaches have shown promising performance in ship detection in SAR images. However, accurate detection still poses challenges in coastal scenes, especially in areas with densely arranged ships. To address this issue, this paper presents a ship target detection method for nearshore areas based on cluster detection. This method employs a cluster detection module to focus the detector on densely arranged ship areas and performs refined secondary detection. Subsequently, local and global results are strategically combined to derive the final outcome. Experiments conducted on the public datasets SSDD and RSDD-SAR validate the efficiency of the proposed method in detecting ships in densely arranged areas.

In the remote sensing community, extracting buildings from remote sensing imagery has triggered great interest. While many studies have been conducted, a comprehensive review of these approaches that are applied to optical and synthetic aperture radar (SAR) imagery is still lacking. Therefore, we provide an in-depth review of both early efforts and recent advances, which are aimed at extracting geometrical structures or semantic attributes of buildings, including building footprint generation, building facade segmentation, roof segment and superstructure segmentation, building height retrieval, building type classification, building change detection, and annotation data correction. Furthermore, a list of corresponding benchmark datasets is given. Finally, challenges and outlooks of existing approaches as well as promising applications are discussed to enhance comprehension within this realm of research.

Deep neural networks based on unrolled iterative algorithms have achieved remarkable success in sparse reconstruction applications, such as synthetic aperture radar (SAR) tomographic inversion (TomoSAR). However, the currently available deep learning-based TomoSAR algorithms are limited to 3-D reconstruction. The extension of deep learning-based algorithms to 4-D imaging, i.e., differential TomoSAR (D-TomoSAR) applications, is impeded mainly due to the high-dimensional weight matrices required by the network designed for D-TomoSAR inversion, which typically contain millions of freely trainable parameters. Learning such huge number of weights requires an enormous number of training samples, resulting in a large memory burden and excessive time consumption. To tackle this issue, we propose an efficient and accurate algorithm called HyperLISTA-ABT. The weights in HyperLISTA-ABT are determined in an analytical way according to a minimum coherence criterion, trimming the model down to an ultra-light one with only three hyperparameters. Additionally, HyperLISTA-ABT improves the global thresholding by utilizing an adaptive blockwise thresholding (ABT) scheme, which applies block-coordinate techniques and conducts thresholding in local blocks, so that weak expressions and local features can be retained in the shrinkage step layer by layer. Simulations were performed and demonstrated the effectiveness of our approach, showing that HyperLISTA-ABT achieves superior computational efficiency with no significant performance degradation compared to the state-of-the-art methods. Real data experiments showed that a high-quality 4-D point cloud could be reconstructed over a large area by the proposed HyperLISTA-ABT with affordable computational resources and in a fast time.

Monocular height estimation (MHE) is key for generating 3-D city models, essential for swift disaster response. Moving beyond the traditional focus on performance enhancement, our study breaks new ground by probing the interpretability of MHE networks. We have pioneeringly discovered that neurons within MHE models demonstrate selectivity for both height and semantic classes. This insight sheds light on the complex inner workings of MHE models and inspires innovative strategies for leveraging elevation data more effectively. Informed by this insight, we propose a pioneering framework that employs MHE as a self-supervised pretraining method for remote sensing (RS) imagery. This approach significantly enhances the performance of semantic segmentation tasks. Furthermore, we develop a disentangled latent transformer (DLT) module that leverages explainable deep representations from pretrained MHE networks for unsupervised semantic segmentation. Our method demonstrates the significant potential of MHE tasks in developing foundation models for sophisticated pixel-level semantic analyses. Additionally, we present a new dataset designed to benchmark the performance of both semantic segmentation and height estimation tasks. The dataset and code will be publicly available athttps://github.com/zhu-xlab/DLT-MHE.pytorch.

Cloud removal is a significant and challenging problem in remote sensing, and in recent years, there have been notable advancements in this area. However, two major issues remain hindering the development of cloud removal: the unavailability of high-resolution imagery for existing datasets and the absence of evaluation regarding the semantic meaningfulness of the generated structures. In this paper, we introduce M3R-CR, a benchmark dataset for high-resolution Cloud Removal with Multi-Modal and Multi-Resolution data fusion. M3R-CR is the first public dataset for cloud removal to feature globally sampled high-resolution optical observations, paired with radar measurements and pixel-level land cover annotations. With this dataset, we consider the problem of cloud removal in high-resolution optical remote sensing imagery by integrating multi-modal and multi-resolution information. In this context, we have to take into account the alignment errors caused by the multi-resolution nature, along with the more pronounced misalignment issues in high-resolution images due to inherent imaging mechanism differences and other factors. Existing multi-modal data fusion based methods, which assume the image pairs are aligned accurately at pixel-level, are thus not appropriate for this problem. To this end, we design a new baseline named Align-CR to perform the low-resolution SAR image guided high-resolution optical image cloud removal. It gradually warps and fuses the features of the multi-modal and multi-resolution data during the reconstruction process, effectively mitigating concerns associated with misalignment. In the experiments, we evaluate the performance of cloud removal by analyzing the quality of visually pleasing textures using image reconstruction metrics and further analyze the generation of semantically meaningful structures using a well-established semantic segmentation task. The proposed Align-CR method is superior to other baseline methods in both areas. The project is available at https://github.com/zhu-xlab/M3R-CR.

Object detection is an essential and fundamental task in computer vision and satellite image processing. Existing deep learning methods have achieved impressive performance thanks to the availability of large-scale annotated datasets. Yet, in real-world applications the availability of labels is limited. In this context, few-shot object detection (FSOD) has emerged as a promising direction, which aims at enabling the model to detect novel objects with only few of them annotated. However, many existing FSOD algorithms overlook a critical issue: when an input image contains multiple novel objects and only a subset of them are annotated, the unlabeled objects will be considered as background during training. This can cause confusions and severely impact the model’s ability to recall novel objects. To address this issue, we propose a self-training-based FSOD (ST-FSOD) approach, which incorporates the self-training mechanism into the few-shot fine-tuning process. ST-FSOD aims to enable the discovery of novel objects that are not annotated, and take them into account during training. On the one hand, we devise a two-branch region proposal networks (RPN) to separate the proposal extraction of base and novel objects, On another hand, we incorporate the student-teacher mechanism into RPN and the region of interest (RoI) head to include those highly confident yet unlabeled targets as pseudo labels. Experimental results demonstrate that our proposed method outperforms the state-of-the- art in various FSOD settings by a large margin. The codes will be publicly available at https://github.com/zhu-xlab/ST-FSOD.

Image classification plays an important role in remote sensing. Earth observation (EO) has inevitably arrived in the big data era, but the high requirement on computation power has already become a bottleneck for analyzing large amounts of remote sensing data with sophisticated machine learning models. Exploiting quantum computing might contribute to a solution to tackle this challenge by leveraging quantum properties. This article introduces a hybrid quantum-classical convolutional neural network (QC-CNN) that applies quantum computing to effectively extract high-level critical features from EO data for classification purposes. Besides that, the adoption of the amplitude encoding technique reduces the required quantum bit resources. The complexity analysis indicates that the proposed model can accelerate the convolutional operation in comparison with its classical counterpart. The model's performance is evaluated with different EO benchmarks, including Overhead-MNIST, So2Sat LCZ42, PatternNet, RSI-CB256, and NaSC-TG2, through the TensorFlow Quantum platform, and it can achieve better performance than its classical counterpart and have higher generalizability, which verifies the validity of the QC-CNN model on EO data classification tasks.

Three-dimensional (3D) building models play a crucial role in shaping digital twin cities and enabling a wide range of urban applications. However, one challenge remains in obtaining a compact representation of buildings from remote sensing. This paper introduces a novel deep learning approach to reconstructing polygonal building models from LiDAR point clouds. Our method leverages a graph neural network to assemble the polyhedra generated through space partitioning, thereby formulating building surface reconstruction as a graph node classification problem. To facilitate network training, we construct a synthetic dataset by simulating aerial LiDAR point clouds on building surface meshes. Experimental results demonstrate the effectiveness of our method, achieving a polyhedral classification accuracy of 96.4%. Moreover, our approach offers high efficiency and interpretability through end-to-end optimization.

Monocular height estimation is of great importance in generating 3D city models from single remote sensing images, while it is a challenging task due to the ill-posed nature of the problem. To address the issue, we propose to adopt adaptive bins (AdaBins) for the network design, which enhances the representation capability of the network with the classification-regression paradigm and the incorporation of local features and global context via a vision transformer encoder. Besides, to weaken the biases of the trained networks caused by the long-tailed nature of the dataset, a head-tail cut is conducted for different treatments of head and tail pixels. Experiments show that improvements are expected with the proposed network on the proposed GBH dataset.

Identifying suitable building roofs for the installation of photovoltaic (PV) systems is important to sustainable energy planning. However, most existing approaches neglect roof superstructures that can obstruct the installation of PV systems. In this research, we propose a novel method, which can help to deal with this issue by detecting roof superstructures from aerial imagery. Considering that semantic information about roof masks is also informative, we propose to first learns roof segmentation maps that are further used to learn roof superstructure maps. Experiments are conducted on Roof Information Dataset (RID). Our method outperforms the state-of-the-art methods both quantitatively and qualitatively.

The majority of the Earth’s surface is covered by clouds, causing optical images to suffer serious degradation of ground information. Synthetic Aperture Radar (SAR) images with the cloud-penetration capability could provide supplementary information to optical images. Thus, the fusion of optical and SAR image can remarkably improve the interpretation accuracy under cloudy conditions. In this paper, we propose to exploit related cloud removal task for accurate multi-modal semantic segmentation of land cover. Towards this goal, we develop an end-to-end learnable architecture which solves the tasks of cloud removal and semantic segmentation of land cover jointly. The cloud removal task encourages to learn knowledgeable features to overcome negative effects of semantic ambiguity. Our experiments show that the proposed algorithm can effectively improve the semantic segmentation accuracy under cloudy conditions.

Disaster mapping is a critical task that often requires on-site experts and is time-consuming. To address this, a comprehensive framework is presented for fast and accurate recognition of disasters using machine learning, termed DisasterNets. It consists of two stages, space granulation and attribute granulation. The space granulation stage leverages supervised/semi-supervised learning, unsupervised change detection, and domain adaptation with/without source data techniques to handle different disaster mapping scenarios. Furthermore, the disaster database with the corresponding geographic information field properties is built by using the attribute granulation stage. The framework is applied to earthquake-triggered landslide mapping and large-scale flood mapping. The results demonstrate a competitive performance for high-precision, high-efficiency, and cross-scene recognition of disasters. To bridge the gap between disaster mapping and machine learning communities, we will provide an openly accessible tool based on DisasterNets. The framework and tool will be available at https://github.com/HydroPML/DisasterNets.

3D geo-information is of great significance for understanding the living environment; however, 3D perception from remote sensing data, especially on a large scale, is restricted, mainly due to the high costs of 3D sensors such as LiDAR. To tackle this problem, we propose a method for monocular height estimation from optical imagery, which is currently one of the richest sources of remote sensing data. As an ill-posed problem, monocular height estimation requires well-designed networks for enhanced representations to improve performance. Moreover, the distribution of height values is long-tailed with the low-height pixels, e.g., the background, as the head, and thus trained networks are usually biased and tend to underestimate building heights. To solve the problems, instead of formalizing the problem as a regression task, we propose HTC-DC Net following the classification-regression paradigm, with the head-tail cut (HTC) and the distribution-based constraints (DCs) as the main contributions. HTC-DC Net is composed of the backbone network as the feature extractor, the HTC-AdaBins module, and the hybrid regression process. The HTC-AdaBins module serves as the classification phase to determine bins adaptive to each input image. It is equipped with a vision transformer encoder to incorporate local context with holistic information and involves an HTC to address the long-tailed problem in monocular height estimation for balancing the performances of foreground and background pixels. The hybrid regression process does the regression via the smoothing of bins from the classification phase, which is trained via DCs. The proposed network is tested on datasets of different resolutions, namely, DFC19 (1.3 m) and GBH (3 m). Experimental results show the superiority of the proposed network over existing methods by large margins. Extensive ablation studies demonstrate the effectiveness of each design component. Codes and trained models are published at https://github.com/zhu-xlab/HTC-DC-Net.

There are various binary semantic segmentation tasks in remote sensing (RS) that aim to extract the foreground areas of interest, such as buildings and roads, from the background in satellite images. In particular, semi-supervised learning, which can use limited labeled data to guide a large amount of unlabeled data for model training, can significantly promote the fast applications of these tasks in practice. However, due to the predominance of the background in RS images, the foreground only accounts for a small proportion of the pixels. It poses a challenge: models are biased toward the majority class of the background, leading to poor performance on the minority class of the foreground. To address this issue, this paper proposes a novel and effective semi-supervised learning framework, Adaptive Matching (AdaptMatch), for RS binary segmentation. AdaptMatch calculates individual and adaptive thresholds of the foreground and background based on their convergence difficulty in an online manner at the training stage; the adaptive thresholds are then used to select the high-confidence pseudo-labeled data of the two classes for model self-training in turn. Extensive experiments are conducted on two widely-studied RS binary segmentation tasks, building footprint extraction and road extraction, to demonstrate the effectiveness and generalizability of the proposed method. The results show that the proposed AdaptMatch achieves superior performance compared with some state-of-the-art semi-supervised methods in RS binary segmentation tasks. The codes will be publicly available at https://github.com/zhu-xlab/AdaptMatch.

Change detection (CD) by comparing two bi-temporal images is a crucial task in remote sensing. With the advantages of requiring no cumbersome labeled change information, unsupervised CD has attracted extensive attention in the community. However, existing unsupervised CD approaches rarely consider the seasonal and style differences incurred by the illumination and atmospheric conditions in multi-temporal images. To this end, we propose a change detection with domain shift setting for remote sensing images. Furthermore, we present a novel unsupervised CD method using a light-weight transformer, called UCDFormer. Specifically, a transformer-driven image translation composed of a light-weight transformer and a domain-specific affinity weight is first proposed to mitigate domain shift between two images with real-time efficiency. After image translation, we can generate the difference map between the translated before-event image and the original after-event image. Then, a novel reliable pixel extraction module is proposed to select significantly changed/unchanged pixel positions by fusing the pseudo change maps of fuzzy c-means clustering and adaptive threshold. Finally, a binary change map is obtained based on these selected pixel pairs and a binary classifier. Experimental results on different unsupervised CD tasks with seasonal and style changes demonstrate the effectiveness of the proposed UCDFormer. For example, compared with several other related methods, UCDFormer improves performance on the Kappa coefficient by more than 12%. In addition, UCDFormer achieves excellent performance for earthquake-induced landslide detection when considering large-scale applications. The code is available at https://github.com/zhu-xlab/UCDFormer.

The domain adaptation (DA) approaches available to date are usually not well suited for practical DA scenarios of remote sensing image classification since these methods (such as unsupervised DA) rely on rich prior knowledge about the relationship between label sets of source and target domains, and source data are often not accessible due to privacy or confidentiality issues. To this end, we propose a practical universal DA (UniDA) setting for remote sensing image scene classification that requires no prior knowledge on the label sets. Furthermore, a novel UniDA method without source data is proposed for cases when the source data are unavailable. The architecture of the model is divided into two parts: the source data generation stage and the model adaptation stage. The first stage estimates the conditional distribution of source data from the pretrained model using the knowledge of class separability in the source domain and then synthesizes the source data. With this synthetic source data in hand, it becomes a UniDA task to classify a target sample correctly if it belongs to any category in the source label set or mark it as “unknown” otherwise. In the second stage, a novel transferable weight that distinguishes the shared and private label sets in each domain promotes the adaptation in the automatically discovered shared label set and recognizes the “unknown” samples successfully. Empirical results show that the proposed model is effective and practical for remote sensing image scene classification, regardless of whether the source data are available or not. The code is available athttps://github.com/zhu-xlab/UniDA.

Semantic segmentation is a fundamental and crucial task that is of great importance to real-world satellite image-based applications. Yet a widely acknowledged issue that occurs when applying the semantic segmentation models to unseen scenery is that the model will perform much poorer than when it was applied to scenery similar to the training data. This phenomenon is usually termed as the domain shift problem. To tackle it, this article presents a self-training-based unsupervised domain adaptation (UDA) method. Different from the previous self-training approaches which focus on rectifying and improving the quality of the pseudo labels, we instead seek to exploit feature-level relation among neighboring pixels to structure and regularize the prediction of the adapted model. Based on the assumption that spatial topological relation is maintained despite the impact of the domain shift, we propose a novel self-training mechanism to perform DA by exploiting local relation in the feature space spanned by the teacher model, from which the pseudo labels are generated. Quantitative experiments on four different public benchmarks demonstrate that the proposed method can outperform the other UDA methods. Besides, analytical experiments also intuitively verify the proposed assumption. Codes will be publicly available athttps://github.com/zhu-xlab/PFST.

Due to the rapid growth of earth observation (EO) data and the complexity of machine learning models, the high requirement on the computation power for EO data analysis becomes a bottleneck. Exploiting quantum computing might tackle this challenge in the future. In this paper, we present a hybrid quantum-classical convolutional neural network (QC-CNN) to classify EO data which can accelerate feature extraction compared with its classical counterpart and handle multi-category classification tasks with reduced quantum resources. The model's validity is verified with the Overhead-MNIST dataset through the TensorFlow Quantum platform.

Building footprint maps are important to urban planning and monitoring. However, most existing approaches that fall back on convolutional neural networks (CNNs), require massive annotated samples for network learning. In this research, we propose a novel semi-supervised network, which can help to deal with this issue by leveraging a large amount of unlabeled data. Considering that rich information is also encoded in feature maps, we propose to integrate the consistency of both features and outputs in the end-to-end network training of unlabeled samples on data perturbation, enabling to impose additional constraints. Experiments are conducted on Inria dataset. Our approach is much superior to the state-of-the-art methods in both quantitative and qualitative results.

To achieve super-resolution synthetic aperture radar (SAR) tomography (TomoSAR), compressive sensing (CS)-based algorithms are usually employed, which are, however, computationally expensive, and thus is not often applied in large-scale processing. Recently, deep unfolding techniques have provided a good combination of physical model-based algorithms and the ability of neural networks to learn from data. In this vein, iterative CS-based algorithms can usually be un-rolled as neural networks with only 10 to 20 layers. When trained, it shows great computational efficiency for further TomoSAR processing. However, the learning architecture of neural networks built in this approach tends to result in error propagation and information loss, thus degrading the performance. In this paper, we propose to employ complex-valued sparse long short-term memory (CV-SLSTM) units to tackle this problem by incorporating historically updating information into the optimization procedure and preserving full information. Simulations are carried out to validate the performance of the proposed algorithm.

Buildings can be distinguished by their form or function and maps of building types can be used by authorities for city planning. Training models to perform this classification requires appropriate training data. OpenStreetMap (OSM) data is globaly available and partly provides information on building types. However, this data can be incomplete or wrong. In this work a U-Net is trained to group buildings into one of the three major function classes (commercial/industrial, residential and other) using incomplete OSM data or ground-truth cadastral data. The model achieves overall accuracies of 72 and 75 percent. Given the OSM data has only around 20 percent of the ground truth labels this shows the incomplete data can be used to train for the building classification task.

Existing domain adaptation (DA) approaches are usually not well suited for practical DA scenarios of remote sensing image classification, since these methods (such as unsupervised DA) rely on rich prior knowledge about the relationship between label sets of source and target domains, and source data are usually not accessible in many cases due to the privacy or confidentiality issues. To this end, we propose a novel source data generation-based universal domain adaptation (SDG-UniDA) model, which includes two parts, i.e., the stage of source data generation and the stage of model adaptation. The first stage is to estimate the conditional distribution of source data from the pre-trained model using the knowledge of class-separability in the source domain and then to synthesize the source data. With this synthetic source data in hand, it becomes a universal DA task that requires no prior knowledge on the label sets. A novel transferable weight is proposed to distinguish the shared and private label sets to each domain, thereby promoting the adaptation in the automatically discovered shared label set and recognizing the "unknown" samples successfully. Empirical results show that SDG-UniDA is effective and practical in this challenging setting for remote sensing image scene classification.

For global range satellite imaging mission, images captured from different areas may have large distribution biases due to different illuminations, shooting angles and atmospheric conditions. A straightforward idea to mitigate this problem is to categorize the images into different domains according the cities they belong to, and apply domain adaptation approaches. However, categorization by cities becomes unreasonable with the increase of the city number, and the emergence of inter-city similarity and intra-city discrepancy. With such consideration, this paper proposes a novel domain adaptation method named domain-agnostic domain adaptation (DADA) to reduce the distribution biases without explicitly defining the domain each image belongs to. To implement this, we augment the images to the styles of different domains by Generative Adversarial Networks (GAN) and contrastive learning to increase the generalizability of down-stream tasks. Experiments on Planetscope building footprint extraction datasets verify the effectiveness of our method.

Building footprint generation is a vital task in a wide range of applications, including, to name a few, land use management, urban planning and monitoring, and geographical database updating. Most existing approaches addressing this problem fall back on convolutional neural networks (CNNs) to learn semantic masks of buildings. However, one limitation of their results is blurred building boundaries. To address this, we propose to learn attraction field representation for building boundaries, which is capable of providing an enhanced representation power. Our method comprises two elemental modules: an Img2AFM module and an AFM2Mask module. More specifically, the former aims at learning an attraction field representation conditioned on an input image, which is capable of enhancing building boundaries and suppressing the background. The latter module predicts segmentation masks of buildings using the learned attraction field map. The proposed method is evaluated on three datasets with different spatial resolutions: the ISPRS dataset, the INRIA dataset, and the Planet dataset. From experimental results, we find that the proposed framework can well preserve geometric shapes and sharp boundaries of buildings, which brings significant improvements over other competitors. The trained model and code are available at https://github.com/lqycrystal/AFM_building.

Accurate and reliable building footprint maps are vital to urban planning and monitoring, and most existing approaches fall back on convolutional neural networks (CNNs) for building footprint generation. However, one limitation of these methods is that they require strong supervisory information from massive annotated samples for network learning. State-of-the-art semi-supervised semantic segmentation networks with consistency training can help to deal with this issue by leveraging a large amount of unlabeled data, which encourages the consistency of model output on data perturbation. Considering that rich information is also encoded in feature maps, we propose to integrate the consistency of both features and outputs in the end-to-end network training of unlabeled samples, enabling to impose additional constraints. Prior semi-supervised semantic segmentation networks have established the cluster assumption, in which the decision boundary should lie in the vicinity of low sample density. In this work, we observe that for building footprint generation, the low-density regions are more apparent at the intermediate feature representations within the encoder than the encoder’s input or output. Therefore, we propose an instruction to assign the perturbation to the intermediate feature representations within the encoder, which considers the spatial resolution of input remote sensing imagery and the mean size of individual buildings in the study area. The proposed method is evaluated on three datasets with different resolutions: Planet dataset (3 m/pixel), Massachusetts dataset (1 m/pixel), and Inria dataset (0.3 m/pixel). Experimental results show that the proposed approach can well extract more complete building structures and alleviate omission errors.

Finding sparse solutions of underdetermined linear systems commonly requires the solving ofL1regularized least squares minimization problem, which is also known as the basis pursuit denoising (BPDN). They are computationally expensive since they cannot be solved analytically. An emerging technique known asdeep unrollingprovided a good combination of the descriptive ability of neural networks, explainable, and computational efficiency for BPDN. Many unrolled neural networks for BPDN, e.g. learned iterative shrinkage thresholding algorithm and its variants, employ shrinkage functions to prune elements with small magnitude. Through experiments on synthetic aperture radar tomography (TomoSAR), we discover the shrinkage step leads to unavoidable information loss in the dynamics of networks and degrades the performance of the model. We propose a recurrent neural network (RNN) with novel sparse minimal gated units (SMGUs) to solve the information loss issue. The proposed RNN architecture with SMGUs benefits from incorporating historical information into optimization, and thus effectively preserves full information to the final output. Taking TomoSAR inversion as an example, extensive simulations demonstrated that the proposed RNN outperforms the state-of-the-art deep learning-based algorithm in terms of super-resolution power as well as generalization ability. It achieved 10% to 20% higher double scatterers detection rate and is less sensitive to phase and amplitude ratio difference between scatterers. Test on real TerraSAR-X spotlight images also shows high-quality 3-D reconstruction of test site.

Synthetic aperture radar tomography (TomoSAR) has been extensively employed in 3-D reconstruction in dense urban areas using high-resolution SAR acquisitions. Compressive sensing (CS)-based algorithms are generally considered as the state-of-the art in super-resolving TomoSAR, in particular in the single look case. This superior performance comes at the cost of extra computational burdens, because of the sparse reconstruction, which cannot be solved analytically, and we need to employ computationally expensive iterative solvers. In this article, we propose a novel deep learning-based super-resolving TomoSAR inversion approach,$\boldsymbol {\gamma }$-Net, to tackle this challenge.$\boldsymbol {\gamma }$-Net adopts advanced complex-valued learned iterative shrinkage thresholding algorithm (CV-LISTA) to mimic the iterative optimization step in sparse reconstruction. Simulations show the height estimate from a well-trained$\boldsymbol {\gamma }$-Net approaches the Cramér-Rao lower bound (CRLB) while improving the computational efficiency by one to two orders of magnitude comparing to the first-order CS-based methods. It also shows no degradation in the super-resolution power comparing to the state-of-the-art second-order TomoSAR solvers, which are much more computationally expensive than the first-order methods. Specifically,$\boldsymbol {\gamma }$-Net reaches more than 90% detection rate in moderate super-resolving cases at 25 measurements at 6 dB SNR. Moreover, simulation at limited baselines demonstrates that the proposed algorithm outperforms the second-order CS-based method by a fair margin. Test on real TanDEM-X data with just six interferograms also shows high-quality 3-D reconstruction with high-density detected double scatterers.

Building extraction in VHR RSIs remains a challenging task due to occlusion and boundary ambiguity problems. Although conventional convolutional neural networks (CNNs) based methods are capable of exploiting local texture and context information, they fail to capture the shape patterns of buildings, which is a necessary constraint in the human recognition. To address this issue, we propose an adversarial shape learning network (ASLNet) to model the building shape patterns that improve the accuracy of building segmentation. In the proposed ASLNet, we introduce the adversarial learning strategy to explicitly model the shape constraints, as well as a CNN shape regularizer to strengthen the embedding of shape features. To assess the geometric accuracy of building segmentation results, we introduced several object-based quality assessment metrics. Experiments on two open benchmark datasets show that the proposed ASLNet improves both the pixel-based accuracy and the object-based quality measurements by a large margin. The code is available at: https://github.com/ggsDing/ASLNet.

Building footprint generation is a vital task of satellite imagery interpretation. However, the segmentation masks of buildings obtained by existing semantic segmentation networks often have blurred boundaries and irregular shapes. In this research, we propose a new boundary regularization network for building footprint generation in satellite images. More specifically, we consider semantic segmentation and boundary regularization in an end-to-end generative adversarial network (GAN). The learned building footprints are regularized by the interplay between the generator and discriminator. By doing so, the straight boundaries and geometric details of the building could be preserved. Experiments are conducted on a collected dataset of Planetscope satellite imagery (spatial resolution: 4.77 m/pixel). Our approach is much superior to the state-of-the-art methods in both quantitative and qualitative results.

Synthetic aperture radar tomography (TomoSAR) has been widely employed in 3-D urban mapping. However, state-of-the-art super-resolving TomoSAR algorithms are computationally expensive, because conventional numerical solvers need to solve the$l_{2^{-}}l_{1}$mix norm minimization. This paper proposes a computationally efficient super-resolving To-moSAR inversion algorithm based on deep learning. We studied the potential of deep learning to mimic a conventional$l_{2}-l_{1}$mix norm solver, i.e. iterative shrinkage thresholding algorithm (ISTA), and proposed several improvements of the complex-valued learned ISTA for TomoSAR inversion. Investigation on the super-resolution ability and estimator efficiency of the proposed algorithm shows that the proposed algorithm approaches the Cramer Rao lower bound (CRLB) with a computational efficiency more than 100 times better than the conventional solver.

Interferometric synthetic aperture radar (InSAR) techniques are powerful tool for reconstructing the 3-D position of scatterers, especially for the urban areas. Since the estimation accuracy depends on the inverse of number of interferograms and signal-to-noise ratio (SNR), it is necessary to use as many as possible interferograms in order to achieve more accurate result. However, the number of interferograms of TanDEM-X data is generally limited for most areas. Therefore, in order to maintain the estimation accuracy, one feasible way is to increase the SNR. In this work, we proposed a novel framework, which integrates the non-local procedure into SAR tomography inversion and combines the robust estimation. A large-scale demonstration has been carried out with five TanDEM-X bistatic data, which covers the entire city of Munich, Germany. Quantitative evaluation of the reconstructed result with the LiDAR reference exhibits the standard deviation of the height difference is within two meters, which implies the proposed framework has great potential for high quality large-scale 3-D urban modeling.

Building segmentation is of great importance in the task of remote sensing imagery interpretation. However, the existing semantic segmentation and instance segmentation methods often lead to segmentation masks with blurred boundaries. In this paper, we propose a novel instance segmentation network for building segmentation in high-resolution remote sensing images. More specifically, we consider segmenting an individual building as detecting several keypoints. The detected keypoints are subsequently reformulated as a closed polygon, which is the semantic boundary of the building. By doing so, the sharp boundary of the building could be preserved. Experiments are conducted on selected Aerial Imagery for Roof Segmentation (AIRS) dataset, and our method achieves better performance in both quantitative and qualitative results with comparison to the state-of-the-art methods. Our network is a bottom-up instance segmentation method that could well preserve geometric details.

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. Recent developments in deep convolutional neural networks (DCNNs) have enabled accurate pixel-level labeling tasks. One central issue remains, which is the precise delineation of boundaries. Deep architectures generally fail to produce fine-grained segmentation with accurate boundaries due to progressive downsampling. In this work, we we introduce a generic framework to overcome the issue, integrating the gated graph convolutional network (GGCN) and deep structured feature embedding (DSFE) into an end-to-end workflow.

Building footprint maps are vital to many remote sensing (RS) applications, such as 3-D building modeling, urban planning, and disaster management. Due to the complexity of buildings, the accurate and reliable generation of the building footprint from RS imagery is still a challenging task. In this article, an end-to-end building footprint generation approach that integrates convolution neural network (CNN) and graph model is proposed. CNN serves as the feature extractor, while the graph model can take spatial correlation into consideration. Moreover, we propose to implement the feature pairwise conditional random field (FPCRF) as a graph model to preserve sharp boundaries and fine-grained segmentation. Experiments are conducted on four different data sets: 1) Planetscope satellite imagery of the cities of Munich, Paris, Rome, and Zurich; 2) ISPRS Benchmark data from the city of Potsdam; 3) Dstl Kaggle data set; and 4) Inria Aerial Image Labeling data of Austin, Chicago, Kitsap County, Western Tyrol, and Vienna. It is found that the proposed end-to-end building footprint generation framework with the FPCRF as the graph model can further improve the accuracy of building footprint generation by using only CNN, which is the current state of the art.

Multibaseline interferometric synthetic aperture radar (InSAR) techniques are effective approaches for retrieving the 3-D information of urban areas. In order to obtain a plausible reconstruction, it is necessary to use more than 20 interferograms. Hence, these methods are commonly not appropriate for large-scale 3-D urban mapping using TanDEM-X data, where only a few acquisitions are available in average for each city. This article proposes a new SAR tomographic processing framework to work with those extremely small stacks, which integrates the nonlocal filtering into SAR tomography inversion. The applicability of the algorithm is demonstrated using a TanDEM-X multibaseline stack with five bistatic interferograms over the whole city of Munich, Germany. A systematic comparison of our result with TanDEM-X raw digital elevation models (DEMs) and airborne LiDAR data shows that the relative height accuracy of two-third buildings is within 2 m, which outperforms the TanDEM-X raw DEM. The promising performance of the proposed algorithm paved the first step toward high-quality large-scale 3-D urban mapping.

Multi-baseline synthetic aperture radar (SAR) interferometric techniques, such as SAR tomography, is well established for 3-D reconstruction in the urban area. These methods usually require fairly large interferometric stacks (> 20 images) for a reliable reconstruction. Hence, they are usually not directly applicable for large-scale 3-D urban mapping using TanDEM-X data where only a few acquisitions are available in average for each city. This work proposes a new SAR tomographic processing framework to those extremely small stacks. The applicability of the algorithm is demonstrated using a TanDEM-X multi-baseline stack with five bistatic interferograms over the whole city of Munich, Germany. Systematic comparison of our result with TanDEM-X raw digital elevation models (DEM) and airborne LiDAR data shows that the relative height accuracy is two meters, which outperforms the TanDEM-X raw DEM. The promising performance of the proposed algorithm paved the first step towards high quality large-scale 3-D urban mapping.

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. Recent developments in deep convolutional neural networks (DCNNs) have enabled accurate pixel-level labeling tasks. One central issue remains, which is the precise delineation of boundaries. Deep architectures generally fail to produce fine-grained segmentation with accurate boundaries due to progressive downsampling. In this work, we have proposed a end-to-end framework to overcome this issue, which uses the graph convolutional network (GCN) for building footprint extraction task. Our proposed framework outperforms state-of-the-art methods.

Building footprint information is an essential ingredient for 3-D reconstruction of urban models. The automatic generation of building footprints from satellite images presents a considerable challenge due to the complexity of building shapes. In this letter, we have proposed improved generative adversarial networks (GANs) for the automatic generation of building footprints from satellite images. We used a conditional GAN (CGAN) with a cost function derived from the Wasserstein distance and added a gradient penalty term. The achieved results indicated that the proposed method can significantly improve the quality of building footprint generation compared to CGANs, the U-Net, and other networks. In addition, our method nearly removes all hyperparameters tuning.

Tomographic synthetic aperture radar (TomoSAR) inversion of urban areas is an inherently sparse reconstruction problem and, hence, can be solved using compressive sensing (CS) algorithms. This paper proposes solutions for two notorious problems in this field. First, TomoSAR requires a high number of data sets, which makes the technique expensive. However, it can be shown that the number of acquisitions and the signal-to-noise ratio (SNR) can be traded off against each other, because it is asymptotically only the product of the number of acquisitions and SNR that determines the reconstruction quality. We propose to increase SNR by integrating nonlocal (NL) estimation into the inversion and show that a reasonable reconstruction of buildings from only seven interferograms is feasible. Second, CS-based inversion is computationally expensive and therefore, barely suitable for large-scale applications. We introduce a new fast and accurate algorithm for solving the NL L1-L2-minimization problem, central to CS-based reconstruction algorithms. The applicability of the algorithm is demonstrated using simulated data and TerraSAR-X high-resolution spotlight images over an area in Munich, Germany.

Conventionally, keys on a physical keyboard have only two states: "released'' and "pressed''. As such, various techniques, such as hotkeys, are designed to enhance the keyboard expressiveness. Realizing that user inevitably perform touch actions during keystrokes, we propose GestAKey, leveraging location and motion of the touch on individual keycaps to augment the functionalities of existing keystrokes. With a log study, we collected touch data for both normal usage (typing and hotkeys) and while performing touch gestures (location and motion), which are analyzed to assess the viability of augmenting keystrokes with simultaneous gestures. A controlled experiment was conducted to compare GestAKey with existing keyboard interaction techniques, in terms of efficiency and learnability. The results show that GestAKey has comparable performance with hotkey. We further discuss the insights of integrating such touch modality into existing keyboard interaction, and demonstrate several usage scenarios.

Synthetic Aperture Radar Tomography is an advanced remote sensing method that is able to reconstruct the 3D distribution of scatterers. One promising solution is to use sophisticated reconstruction algorithms, like the compressive sensing based algorithm. However, it suffers from the high demand of number of images and the computational expenses. Therefore, it is hard to apply for large scale practice. In this work, a complete work-flow for Non-Local Compressive Sensing based SAR Tomography method is presented. Moreover, the applicability of the algorithm is demonstrated by exploiting TerraSAR-X high resolution spotlight images over a test site in Munich, Germany.

$L_{1}$regularization is used for finding sparse solutions to an underdetermined linear system. As sparse signals are widely expected in remote sensing, this type of regularization scheme and its extensions have been widely employed in many remote sensing problems, such as image fusion, target detection, image super-resolution, and others, and have led to promising results. However, solving such sparse reconstruction problems is computationally expensive and has limitations in its practical use. In this paper, we proposed a novel efficient algorithm for solving the complex-valued$L_{1}$regularized least squares problem. Taking the high-dimensional tomographic synthetic aperture radar (TomoSAR) as a practical example, we carried out extensive experiments, both with the simulation data and the real data, to demonstrate that the proposed approach can retain the accuracy of the second-order methods while dramatically speeding up the processing by one or two orders. Although we have chosen TomoSAR as the example, the proposed method can be generally applied to any spectral estimation problems.

The standard TanDEM-X product meats HRTI-3 DEM specification and comes with a sample spacing of 12 m. We apply non-local means (NL) interferogram filtering to the TanDEM-X data. In this paper, we present modifications of the original NL filter which render it more appropriate and efficient for massive processing of TanDEM-X data. Further, we investigate the noise reduction properties as well as the resolution and the coherence estimation accuracy of the new NL filter. Simulations and tests with TanDEM-X data hint that the improved DEMs possess a quality close to the HRTI-4 standard. Also future global InSAR missions like Tandem-L will greatly benefit from this type of filters.

With the availability of meter resolution space-borne SAR systems, urban monitoring using SAR Tomography (TomoSAR) becomes increasingly popular, because of its layover separation capability. However, compared to Persistent Scatterer Interferometry (PSI), TomoSAR applications are much more computationally expensive. This article introduces a TomoSAR processing system for long-term large urban area mapping and monitoring. Two new features were introduced: 1. PSI was integrated into the currently available TomoSAR algorithms (e.g. TSVD, SVD-Wiener, and SL1MMER) to increase the overall computational efficiency; 2. results from multiple view angles were fused to provide full coverage of each building façade. This processing system handles an entire TerraSAR-X high resolution spotlight scene in an affordable time, achieving scatterer density up to 1 million scatterer/km2from a single stack, comparing to 60 thousand to 100 thousand scatterer/km2for PSI.