Extracting 3D building information from monocular satellite sensor imagery remains a formidable challenge in the field of remote sensing. Multitask frameworks based on deep learning, which particularly for simultaneously predicting 2D building outlines and their respective heights using ortho-rectified satellite imagery, have shown promise in addressing this challenge. Height estimation is notably complex due to the absence of explicit height indicators, limited interaction between semantic-height features, and inadequate representation of building relationships. Moreover, the availability of data sources is a limiting factor for broader application. To overcome these issues, this study introduces an innovative instance-level multitask learning model (named BDH-Net) that leverages off-nadir perspectives and roof-to-footprint offset vectors to enhance modeling. This model comprises four key components: a pixel-wise feature extraction image encoder-decoder, a query transformer decoder, a multitask decoder, and a height decoder that employs intra-instance and inter-instance attention for precise building height estimation. Additionally, we pioneer the use of Google Earth imagery to construct an off-nadir satellite dataset with roof-to-footprint offset vectors specifically designed for building instance segmentation and height prediction, known as the BDH dataset. Comprehensive experiments demonstrate that the proposed BDH-Net significantly improves the accuracy of monocular 3D building data extraction by integrating roof-to-footprint offset vectors and leveraging context specific to each building instance. With the extensive coverage and regular updates of Google Earth imagery, BDH-Net holds substantial potential for wide-ranging and long-term applications. The source code of the proposed BDH-Net and the BDH dataset are publicly available at https://github.com/wishx98/BDHNet.

Neuromorphic computing with spiking neural networks is promising for energy-efficient artificial intelligence (AI) applications. However, different from humans who continually learn different tasks in a lifetime, neural network models suffer from catastrophic forgetting. How could neuronal operations solve this problem is an important question for AI and neuroscience. Many previous studies draw inspiration from observed neuroscience phenomena and propose episodic replay or synaptic metaplasticity, but they are not guaranteed to explicitly preserve knowledge for neuron populations. Other works focus on machine learning methods with more mathematical grounding, e.g., orthogonal projection on high dimensional spaces, but there is no neural correspondence for neuromorphic computing. In this work, we develop a new method with neuronal operations based on lateral connections and Hebbian learning, which can protect knowledge by projecting activity traces of neurons into an orthogonal subspace so that synaptic weight update will not interfere with old tasks. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities and enable orthogonal projection. This provides new insights into how neural circuits and Hebbian learning can help continual learning, and also how the concept of orthogonal projection can be realized in neuronal systems. Our method is also flexible to utilize arbitrary training methods based on presynaptic activities/traces. Experiments show that our method consistently solves forgetting for spiking neural networks with nearly zero forgetting under various supervised training methods with different error propagation approaches, and outperforms previous approaches under various settings. Our method can pave a solid path for building continual neuromorphic computing systems. The code is available at https://github.com/pkuxmq/HLOP-SNN.

The extraction of building heights from monocular satellite sensor imagery poses a significant difficulty in remote sensing. Deep learning-based multi-task frameworks have recently emerged, showing potential in concurrently predicting 2D building shapes and heights from ortho-rectified satellite images. These techniques, however, have several limitations, including a lack of direct height markers, inherent uncertainties in height estimation, a heavy reliance on limited digital surface models, suboptimal integration of semantic-height feature, and inadequate depiction of the spatial relationships between buildings. To address these challenges, we develop a novel instance-level multi-task learning model that uses off-nadir imaging angles, roof-to-footprint as a proxy for height, and integrates certain geometric properties to increase height estimation accuracy. This method was validated using a self-constructed dataset, demonstrating significantly superior performance relative to two benchmarks.

Urban Villages (UVs) renovation has been incorporated into the Sustainable Development Goals (SDGs) as a result of the inequality issue among residents garnering substantial social attention. However, existing deep-learning techniques for UVs extraction have been limited to a single spatial scale (e.g., patch-level or pixel-level extraction), leading to inadequate precision and integrity in their extraction outcomes. To overcome this limitation, our study introduces HR-UVFormer, a top-down and multimodal hierarchical extraction approach that extracts UVs from a coarse scale (patch) to a fine granularity (pixel), aiming to enhance the internal completeness and boundary accuracy of the extraction results. The multimodal approach can effectively fuse multimodal features (e.g., building footprints (BF)) with remote sensing images (RSI) to enhance UVs extraction. The Shenzhen results indicate that the coarse-scale extraction accuracy achieves an overall accuracy (OA) of 98.79%, and the fine-grained extraction accuracy achieves a mean Intersection over Union (mIoU) of 93.60%. Furthermore, ablation experiments demonstrate a notable 7.14% improvement in mIoU with the hierarchical extraction strategy compared to the traditional pixel-based extraction strategy, and the fusion of BF and RSI yields further improvements of 2.78% and 0.65% in OA and mIoU, respectively. This finding confirms the synergistic effect between RSI and BF in UVs extraction, which has been further analyzed in this study. Additionally, the proposed model outperforms other deep learning models and exhibits the potential to support more modal features (e.g., POI). Finally, the experimental dataset and code can be publicly accessed at https://github.com/q1310546582/HR-UVFormer-code.

Research on integrating statistical knowledge into deep learning models for earthquake forecasting has been limited. Traditional deep learning models require extensive parameter learning from scratch. This study proposes a spatio-temporal convolutional (STC) model that incorporates spatio-temporal decay prior knowledge derived from the epidemic-type aftershock sequence (ETAS) into a convolutional kernel. This allows the STC model to have the prototype to learn the pattern of mainshocks to trigger aftershocks at the beginning of training, with only four neurons to fine-tune it. In California, the STC and the ETAS model are conducted for forecasting next-day earthquakes with magnitudes of${M} \ge 3$,${M} \ge 4$, and${M} \ge 5$. Both performances were assessed using the receiver operating characteristic (ROC) curve, the precision-recall (PR) curve, and the parimutuel gambling score (PGS). The evaluation results indicate that the STC model surpasses ETAS in forecasting next-day earthquakes not accidental. Furthermore, our analysis suggests that including earthquakes below the complete magnitudes can enhance the STC model’s classification performance, as small earthquakes also contain information about future earthquakes.

Change detection (CD) in high-resolution remote sensing has received large attention due to its wide range of applications. Many methods have been proposed in the literature and achieved excellent performance. However, they are often fully supervised, thus requiring abundant pixel-level labeled samples, which is time-consuming and labor-intensive. Especially compared to the common single-temporal interpretation, labeling bi-temporal images is often more complicated. Therefore, this study combines weakly supervised learning (WSL) to reduce label acquisition costs. But changed regions are small, fragmented, and similar to the background, which increase the gap between weakly supervised and fully supervised tasks. To address these difficulties, we explore self-supervised methods to construct a WSL framework based on image-level labels for general CD, termed WSLCD in this paper. First, we design a double-branch siamese network to derive embeddings and initial class attention maps (CAMs), which inputs the original image pair and the spatially transformed image pair. Second, mutual learning and equivariant regularization (MLER) is enforced on CAMs from different views, which implements consistency constraints in confusion regions and makes CAMs learn from each other based on saliency regions. Furthermore, prototype-based contrastive learning (PCL) is designed such that unreliable pixels can learn from prototypes computed from reliable pixels. PCL includes intra-view contrast and cross-view contrast depending on whether the prototypes and class embeddings are from the same view. With the above strategies, we narrow the gap between image-level weakly supervised CD and fully supervised CD. Experiments are conducted on three CD datasets, including CLCD, DSIFN and GCD. Our method achieves state-of-the-art performance on pseudo label generation and CD. The code is available at https://github.com/mfzhao1998/WSLCD.

Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. For training the non-differentiable SNN methods, the backpropagation through time (BPTT) with surrogate gradients (SG) method has achieved high performance. However, this method suffers from considerable memory cost and training time during training. In this paper, we propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency compared with BPTT. First, we show that the backpropagation of SNNs through the temporal domain contributes just a little to the final calculated gradients. Thus, we propose to ignore the unimportant routes in the computational graph during backpropagation. The proposed method reduces the number of scalar multiplications and achieves a small memory occupation that is independent of the total time steps. Furthermore, we propose a variant of SLTT, called SLTT-K, that allows backpropagation only at K time steps, then the required number of scalar multiplications is further reduced and is independent of the total time steps. Experiments on both static and neuromorphic datasets demonstrate superior training efficiency and performance of our SLTT. In particular, our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT. Our code is available at https://github.com/qymeng94/SLTT.

Pansharpening is a crucial image processing technique for numerous remote sensing downstream tasks, aiming to recover high spatial resolution multispectral (HRMS) images by fusing high spatial resolution panchromatic (PAN) images and low spatial resolution multispectral (LRMS) images. Most current mainstream pansharpening fusion frameworks directly learn the mapping relationships from PAN and LRMS images to HRMS images by extracting key features. However, we propose a novel pansharpening method based on the denoising diffusion probabilistic model (DDPM) called PanDiff, which learns the data distribution of the difference maps (DM) between HRMS and interpolated MS (IMS) images from a new perspective. Specifically, PanDiff decomposes the complex fusion process of PAN and LRMS images into a multi-step Markov process, and the U-Net is employed to reconstruct each step of the process from random Gaussian noise. Notably, the PAN and LRMS images serve as the injected conditions to guide the U-Net in PanDiff, rather than being the fusion objects as in other pansharpening methods. Furthermore, we propose a modal intercalibration module (MIM) to enhance the guidance effect of the PAN and LRMS images. The experiments are conducted on a freely available benchmark dataset, including GaoFen-2, QuickBird, and WorldView-3 images. The experimental results from the fusion and generalization tests effectively demonstrate the outstanding fusion performance and high robustness of PanDiff. Fig. 1 depicts the results of the proposed method performed on various scenes. Additionally, the ablation experiments confirm the rationale behind PanDiff’s construction.

Designed to quantify the value of alarm-based earthquake predictions, the Molchan diagram only compares the rate of false negatives (FNR) to the expected number of target events in the alerted time-space volume, while ignoring essential performance metrics such as the false discovery rate (FDR). The FDR is the ratio of alarms that do not correspond to any earthquake to the total number of alarms and quantifies the credibility of alarms. Strategies only minimizing the false-negative rate can lead to too many false positive alarms, leading to mistrusts in them. In this study, a new significance test for alarms based on their FDR is constructed. We construct a new 3-D error diagram, where the x-axis is the relative spatio-temporal coverage of alarms, the y-axis is the FNR and the z-axis is the FDR. The strategy of the 3D error diagram is to balance the cost of alarms, the losses brought by missed events, and the additional negative impact brought by alarms with low credibility. Using the new 3D error diagram and a simple time series analysis for extracting thermal infrared (TIR) anomalies, we show that the 11-year TIR anomalies in the Sichuan area are strongly statistically related to earthquakes with magnitude ≥4.0 from the view of events and alarms, and that the score provided by our 3D error diagram is more consistent and robust than the 2D original Molchan diagram, so that the new method effectively helps finding more balanced alarms.

With the development of high-resolution satellites, more and more attention has been paid to remote sensing (RS) scene classification. Convolutional neural networks (CNNs), which replace the traditional handcrafted features with a learning-based feature extraction mechanism, are widely used in scene classification. But CNNs are less effective in deriving long-range contextual relations, which limits the further improvement. Visual transformer (VT), an emerging image processing method, provides a new perspective for RS scene classification by directly acquiring long-range features. Although there have been limited works combining CNN and VT through simple concatenation, the collaborations between them are insufficient. To address these issues, we propose a local and long-range collaborative framework (L2RCF). First, we design a dual-stream structure to extract the local and long-range features. Second, a cross-feature calibration (CFC) module is designed for them to improve representation of the fusion features. Then, combining deep supervision (DS) and deep mutual learning (DML), a novel joint loss is proposed to enhance the dual-stream feature extractor and further improve the fused features. Finally, a two-stage semi-supervised training strategy is designed to improve performance with unlabeled samples. To demonstrate the effectiveness of L2RCF, we conducted experiments on three widely used RS scene classification data sets: RSSCN7, AID, and NWPU. The results show that L2RCF performs significantly better compared with some state-of-the-art scene classification methods.

Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. However, it is a challenge to efficiently train SNNs due to their non-differentiability. Most existing methods either suffer from high latency (i.e., long simulation time steps), or cannot achieve as high performance as Artificial Neural Networks (ANNs). In this paper, we propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance that is competitive to ANNs yet with low latency. First, we encode the spike trains into spike representation using (weighted) firing rate coding. Based on the spike representation, we systematically derive that the spiking dynamics with common neural models can be represented as some sub-differentiable mapping. With this viewpoint, our proposed DSR method trains SNNs through gradients of the mapping and avoids the common non-differentiability problem in SNN training. Then we analyze the error when representing the specific mapping with the forward computation of the SNN. To reduce such error, we propose to train the spike threshold in each layer, and to introduce a new hyperparameter for the neural models. With these components, the DSR method can achieve state-of-the-art SNN performance with low latency on both static and neuromorphic datasets, including CIFAR-10, CIFAR-100, ImageNet, and DVS-CIFAR10.

Land surface temperature (LST) is one of the parameters characterizing the energy balance of the Earth surface system. To reveal the dominant influence factors of LST is essential to mitigate the urban thermal environment. This study used a deep learning approach (boosted regression trees, BR T) to prioritize the quantified roles and contribution of the influencing factors to urban LST. The approach is applied in Beijing- Tianjin-Hebei (BTH) urban agglomeration. The conclusions are obtained as: (1) From 2000 to 2015, the regions with obvious rising LST are mainly concentrated in the southwest and east, especially in the southern part of Tangshan, the eastern part of Langfang and the coastal area in the northeast of Cangzhou, while the LST in Beijing and the surrounding Tangshan and Qinhuangdao decreased instead. (2) Among the cities, there are large differences in the effects of different factors on LST. Overall, the socio-economic factor explains the highest contribution of LST, followed by the humidity. (3) Central core functional areas such as Cangzhou, Langfang and Tangshan near the Bohai Sea show a more consistent contribution distribution. For the northwestern ecological cultured area, the contribution of DEM is highest and can reach 42.93%.

Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. Particularly, backpropagation through time (BPTT) with surrogate gradients (SG) is popularly used to enable models to achieve high performance in a very small number of time steps. However, it is at the cost of large memory consumption for training, lack of theoretical clarity for optimization, and inconsistency with the online property of biological learning rules and rules on neuromorphic hardware. Other works connect the spike representations of SNNs with equivalent artificial neural network formulation and train SNNs by gradients from equivalent mappings to ensure descent directions. But they fail to achieve low latency and are also not online. In this work, we propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning by tracking presynaptic activities and leveraging instantaneous loss and gradients. Meanwhile, we theoretically analyze and prove that the gradients of OTTT can provide a similar descent direction for optimization as gradients from equivalent mapping between spike representations under both feedforward and recurrent conditions. OTTT only requires constant training memory costs agnostic to time steps, avoiding the significant memory costs of BPTT for GPU training. Furthermore, the update rule of OTTT is in the form of three-factor Hebbian learning, which could pave a path for online on-chip learning. With OTTT, it is the first time that the two mainstream supervised SNN training methods, BPTT with SG and spike representation-based training, are connected, and meanwhile it is in a biologically plausible form. Experiments on CIFAR-10, CIFAR-100, ImageNet, and CIFAR10-DVS demonstrate the superior performance of our method on large-scale static and neuromorphic datasets in a small number of time steps. Our code is available at https://github.com/pkuxmq/OTTT-SNN.

Remote sensing (RS) scene classification has attracted extensive attention due to its large number of applications. Recently, convolutional neural networks (CNNs) methods have shown impressive ability of feature learning in RS scene classification. However, the performance is still limited by large-scale variance and complex background. To address these problems, we present a multilayer feature fusion network with spatial attention and gated mechanism (MLF2Net_SAGM) for RS scene classification. At first, the backbone is employed to extract multilayer convolutional features. Then, a residual spatial attention module (RSAM) is proposed to enhance discriminative regions of the multilayer feature maps, and key areas can be harvested. Finally, the multilayer spatial calibration features are fused to form the final feature map, and a gated fusion module (GFM) is designed to eliminate feature redundancy and mutual exclusion (FRME). To verify the effectiveness of the proposed method, we conduct comparative experiments based on three widely used RS image scene classification benchmarks. The results show that the direct fusion of multilayer features via element-wise addition leads to FRME, whereas our method fuses multilayer features more effectively and improves the performance of scene classification.

Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. However, the supervised training of SNNs remains a hard problem due to the discontinuity of the spiking neuron model. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks, and use surrogate derivatives or compute gradients with respect to the spiking time to deal with the problem. These approaches either accumulate approximation errors or only propagate information limitedly through existing spikes, and usually require information propagation along time steps with large memory costs and biological implausibility. In this work, we consider feedback spiking neural networks, which are more brain-like, and propose a novel training method that does not rely on the exact reverse of the forward computation. First, we show that the average firing rates of SNNs with feedback connections would gradually evolve to an equilibrium state along time, which follows a fixed-point equation. Then by viewing the forward computation of feedback SNNs as a black-box solver for this equation, and leveraging the implicit differentiation on the equation, we can compute the gradient for parameters without considering the exact forward procedure. In this way, the forward and backward procedures are decoupled and therefore the problem of non-differentiable spiking functions is avoided. We also briefly discuss the biological plausibility of implicit differentiation, which only requires computing another equilibrium. Extensive experiments on MNIST, Fashion-MNIST, N-MNIST, CIFAR-10, and CIFAR-100 demonstrate the superior performance of our method for feedback models with fewer neurons and parameters in a small number of time steps. Our code is available at https://github.com/pkuxmq/IDE-FSNN.

Accurate delineation of individual tree crowns in human settlement is of vital importance to decision-making in environmental management. Increasing availability of LiDAR data and applications of mathematical morphology imply a paradigm shift in tree crown delineation. This paper introduces a new approach based on “mathematical morphology on grey-level images” that enables such delineation. We consider a LiDAR data set as a grey-level image in which the indexes are the (x,y) locations on the grid and in which each value is the corresponding height of the point acquired by using the LiDAR sensor (i.e. top of the tree at the (x,y) location). We have applied this approach to a large data set in the frame of a partnership with a Hungarian University, and the results we obtain are closely related to what can be seen on a 3D visualization of the LiDAR data set.

A potential application of LiDAR data in street-level greenery assessment has been explored in this study. Taking full advantage of the fine-scale tree structure modeling of LiDAR data, we propose a new green view index calculation method to assess street greenery visualization. Results indicate that LiDAR data is capable of differentiating the amount of greenness being perceived from different sites and the calculation method gives an objective measure of street-level greenery.

In this paper, STD (standard deviation) threshold method was proposed using to detect OLR (Outgoing Longwave Radiation) seismic anomalies. OLR data describe the radiation information from the top of the atmosphere, which be thought to reflect the energy changes of earth-atmosphere system prior to earthquakes. The method to identifying seismic anomaly and non-seismic anomaly has been proposed in this work. Based on this, Wenchuan Sichuan, May 12, 2008, Ms8.0 and Delingha, Qinghai, November 10, 2008, Ms6.3 have been studied respectively using STD threshold method. The results indicate that the seismic infrared radiation anomalies can be detected using STD threshold method. And these anomalies can reflect the process of earthquake preparation. At the same time, the spatial distributions of OLR anomalies have some indication effect on judge the seismogenic structure and the epicenter.