Satellite on-orbit remote sensing image intelligent interpretation relies on on-orbit devices with extremely limited computational resources, utilizing advanced neural networks designed through lightweight methodologies to achieve fast and accurate interpretation of on-orbit remote sensing images. Nevertheless, the current design of lightweight neural networks exhibits three salient issues: firstly, the prevailing “one-size-fits-all" network lightweight design pattern is notably inefficient; secondly, there is a deficiency in analysing the impact of lightweight operations on network performance; thirdly, the mutual influence among various lightweight operations is overlooked. To address these issues, firstly, we propose a neural network lightweightability measurement model and its computational method by investigating the effects of various lightweight operations; secondly, we propose a neural network general lightweight design framework (GLD) tailored for satellite on-orbit remote sensing images intelligent interpretation. Specifically, GLD, based on a meta-leaning approach, integrates knowledge distillation (KD), pruning and quantization, three general lightweight technologies, into a framework. It dynamically assesses the distillability, prunability and quantifiability of neural networks, and uses this assessment and uses this as supervision to dynamically jointly optimizes KD, pruning and quantization, making it applicable to various mainstream neural networks; furthermore, we explore the mutual influence among lightweight operations based on GLD; finally, through ablation experiments and comparative experiments, we further verify the effectiveness and superiority of GLD.

The role of on-orbit computing for satellites is transitioning from being a backup measure to becoming a primary key function. However, the limited computing resources available on satellites make it difficult to deploy advanced models with large parameters. Additionally, satellite on-orbit computing requires high speed and accuracy, posing significant challenges for developing suitable models. To overcome these challenges, we propose an efficient vision transformer, RepSViT, for satellite on-orbit computing. The RepSViT introduces Spiking neural networks (SNNs) with high biological plausibility, event-driven property and low power consumption into the field of remote sensing image processing and satellite on-orbit computing for the first time and incorporates structural reparameterization. Specifically, we design a dynamic dilated spiking convolution (D2SC) based on SNNs to improve the feature extraction capability and efficiency of RepSViT. We also develop a spiking guided attention module (SGAM) to make RepSViT pay more attention to object-related features with lower computational costs. Furthermore, we design an efficient coupled fine–coarse-grained block (ECFC) to enhance the model’s capability in extracting coarse and fine-grained features. To ensure effective feature extraction, inference speed and reduced computational costs, we design a reparameterized feed-forward network (RepFFN). RepSViT achieves an inference latency of 8.33 ms and a recognition accuracy of 95% on an embedded GPU, utilizing 3.77 million parameters and consuming 0.6 GFLOPs computational costs.

Extremely constrained storage and computational resources are one of the difficulties of satellite-on-orbit computing, which leads to over-parametric high-performance models not performing properly on-orbit. Knowledge distillation (KD) is an effective method for model compression; yet, there is a gap in the study of the limits and laws of KD-based model compression. To bridge this gap, we propose a novel KD framework, pyramid KD (PKD) and define a knowledge explosion and knowledge offset. Specifically, the pyramid distillation framework is built by stacking multiple sets of deep mutual learning (DML) models, with the smaller models on the top of the larger ones, and the overall structure is like a pyramid; hence, it is called PKD. To avoid knowledge explosion, we design a hybrid online–offline smooth distillation (HOSD) strategy by combining online distillation and offline distillation and reducing the difference between models. To avoid knowledge offset, we design an adaptive multiteacher distillation method to obtain multiteacher weighted knowledge by adaptively learning the weight of each teacher’s knowledge. We introduce an evolutionary algorithm to automatically find the optimal PKD configuration. We conduct ablation experiments and compare PKD with state-of-the-art distillation methods using ResNet series networks and VGG series networks as base models on Aircraft and FGSC-23 datasets, respectively. The experimental results show the effectiveness and advancement of PKD and reveal the law that the object recognition accuracy varies with the model compression rate.

In recent years, the performance of the deep learning based object detection models for remote sensing images improves with the increase of the hyperparameter scale. However, the state of the art of detection models are generally too cumbersome to adapt to the resource-constrained satellite platforms, and even general lightweight detection models cannot satisfy performance requirements. To solve those problems, a lightweight and accurate oriented object detection network for satellite on-orbit computing (SOCDet) is proposed from three levels. At the computing unit level, we propose an efficient computing unit Single-kernel Omni-dimensional Dynamic Convolution to make SOCDet feature extraction more efficient. At the network module level, we design a structural reparameterization block based on composite structure reparameterization to improve inference accuracy. A guided attention module for guiding SOCDet is designed to extract object features. We design a lightweight and concise one-stage detection architecture at the network architecture level to accommodate satellite platforms with extremely constrained computing and storage resources. We conduct ablation experiments on the ground server and compare experiments with the state-of-the-art methods on embedded GPU, using DOTA, HRSC2016 and FAIR1M. The experiment results show that SOCDet can achieve 2.36 times faster than the baseline in inference speed with 10.78 more mAP, 72.6% fewer Params and 92.56% fewer floating point operations. Compared with state-of-the-art methods, SOCDet improves the inference speed and derives competitive mAP results with affordable computing capability.

High-quality remote sensing images not only provide opportunities for deep learning-based image interpretation, but also challenge satellite edge devices for storage, processing and downlinking to the ground server. Satellite on-orbit object recognition is an effective measure to solve this challenge. However, there is an insurmountable gap between the extremely limited computing and storage resources of satellites and the demand for those of deep learning. To solve this problem, a fast and lightweight intelligent satellite-on-orbit computing network (SOCNet) is proposed. First, the overall network architecture based on the idea of flat multi-branch feature extraction is proposed to accelerate model inference and reduce the network depth. Second, we propose the idea of exchanging a larger receptive field for network depth and combine the idea of depthwise separable convolution to further reduce the amount of parameters. We design a feature extraction method of coupled fine-coarse-grained for efficient feature extraction. Finally, global average pooling is used for feature fusion to further reduce network parameters amount and computational complexity. We have performed multiple sets of ablation experiments and some state-of-the-art comparison experiments on the server and carried out experimental verification on the NVIDIA Jetson TX2 mobile device to mimic the resource-constrained environment on the satellite. SOCNet with parameters of 0.21 MB, model size of 1.22 MB and computation cost of 50.81 MB floating point operations has achieved 99% comprehensive accuracy, 2.45 ms latency and 408 images per second throughput on 30 types of aircraft datasets with the 224 × 224 remote sensing images.

The supervisory control and data acquisition (SCADA) systems are widely installed on wind turbines (WTs) in major wind farms, which produce a large amount of sensory data that can be used for fault diagnosis of WTs. However, these SCADA data are naturally multivariate time series, which represent complex temporal correlations within each sensor variable and spatial correlations between different sensor variables. To effectively capture spatio-temporal correlations in SCADA data, we propose a new spatio-temporal multiscale neural network (STMNN). The proposed STMNN model contains two parallel feature extraction modules: first, a multiscale deep echo state network module to extract temporal multiscale features; second, a multiscale residual network module to extract spatial multiscale features. Additionally, as the WTs are in a normal working state most of the time, there are a large amount of normal SCADA data and few failure SCADA data. To address the data imbalance problem of SCADA data and enhance the fault diagnosis performance, instead of cross-entropy loss, the STMNN model adopts focal loss as loss function. Our proposed STMNN method can provide an end-to-end fault diagnosis solution with imbalanced SCADA data, and is evaluated through experiments on an SCADA dataset from a real wind farm. The experimental results and comparative analysis have proved the effectiveness of our proposed STMNN model in practical applications.