Modality-agnostic Semantic Segmentation (MaSS) aims to achieve robust scene understanding across arbitrary combinations of input modality. Existing methods typically rely on explicit feature alignment to achieve modal homogenization, which dilutes the distinctive strengths of each modality and destroys their inherent complementarity. To achieve cooperative harmonization rather than homogenization, we propose CHARM, a novel complementary learning framework designed to implicitly align content while preserving modality-specific advantages through two components: (1) Mutual Perception Unit (MPU), enabling implicit alignment through window-based cross-modal interaction, where modalities serve as both queries and contexts for each other to discover modality-interactive correspondences; (2) A dual-path optimization strategy that decouples training into Collaborative Learning Strategy (CoL) for complementary fusion learning and Individual Enhancement Strategy (InE) for protected modality-specific optimization. Experiments across multiple datasets and backbones indicate that CHARM consistently outperform the baselines, with significant increment on the fragile modalities. This work shifts the focus from model homogenization to harmonization, enabling cross-modal complementarity for true harmony in diversity.

This article is concerned with the human behavior identification problem for linear human-in-the-loop (HiTL) systems in adversarial environments. By modeling the human as an optimal controller that minimizes his/her individual cost function and the adversarial environment as an opponent to maximize the cost function, the HiTL system is formulated as a linear-quadratic zero-sum differential game that consists of two players that are the human and adversarial environment. Then, the human behavior identification is transformed to an inverse reinforcement learning (IRL) problem. Accordingly, the main works carried out in this article can be summarized as follows: 1) an integral concurrent learning (ICL) law is proposed to estimate the feedback matrix of the human and 2) based on the estimated feedback matrix, the weighting matrices in human cost function are retrieved by minimizing a residual. The main focus of the developed human behavior identification method is to remove the persisting excitation constraint and the demand for measuring the control input of humans that are universally required in existing online learning approaches. Finally, the results of simulation and experiment on the lane keeping scenario of a vehicle verify the validity of the proposed adaptive-IRL-based human behavior identification strategy.

Band selection is an effective dimensionality reduction technique for hyperspectral image (HSI). In recent years, many unsupervised band selection methods have been proposed, but most of them measure the similarity between bands only from the spectral dimension, ignoring their high spatial correlation. In addition, these methods provide a less than ideal quantitative description of spectral information and neglect the redundancy within the selected bands. To address these issues, we propose a hyperspectral band selection via structural correlation and information measures (SCIMs), claiming the following contributions: 1) through introducing the structural similarity (SSIM) index to assess the correlation between bands, both spectral and spatial information are considered; 2) the HSI cube is partitioned into several groups by considering both intragroup and intergroup similarity measured by SSIM; and 3) in order to obtain a high-quality band subset, a representative band is selected in each group from the point of view of information as well as redundancy. The experimental results on three HSI datasets show that the proposed method has significant advantages compared with competitors.

Relative radiometric calibration (RRC) mainly focuses on color consistency and streak levels between multi-camera or multiple charge-coupled devices (CCDs), that’s to say, full field-of-view (FOV). But RRC may not be conducted completely or that useful due to some factors, such as data quality or quantity in lifetime image statistics, and ineffective side-slither RRC. Aimed to this, this paper proposes a novel approach to solve inner uneven illumination of each camera image and relative radiometric difference of multi-camera images. The highest layer of unidirectional pyramid (UDP) is decomposed into illumination and reflectance components. Uneven phenomenon in this scale is eliminated in illumination component with column-by-column compensation processing and different scales of non-uniformity are removed together with unidirectional pyramid reconstruction. Radiometric variation of multi-camera images is solved with iterative radiometric adjustment. Some typical data of HISEA-2 Multi-Spectral Scanner 1 (MSS-1) are used to validate the effectiveness of our method both in visual and quantitative terms.

To improve machines’ intelligence, it is necessary for the machines to learn human’s behavior. In this paper, we make a reasonable hypothesis that a human behaves like a linear quadratic regulator whose cost function is unknown to the machine when performing a task. In addition, the system dynamics in many real applications is completely unknown. Therefore, our purpose is to search for an equivalent cost function to the human only from control input and system state data for continuous-time linear human-in-the-loop (HiTL) systems with completely unknown dynamics. An adaptive inverse optimal control (IOC) method is proposed for this purpose, which can help the machine conduct a better understanding for the human behavior and makes it possible to reproduce a similar optimal controller in other environments. Noticing the difficulty of directly obtaining the weighting matrix, an adaptive integral concurrent learning (ICL) algorithm is developed to identify the system matrices and human feedback gain matrix online, which removes the persistent excitation (PE) conditions. Then, the weighting matrix is determined via solving a convex programming problem. Finally, simulation results on the lane-keeping assist system of an intelligent vehicle are presented to demonstrate the validity of the proposed adaptive IOC algorithm. Note to Practitioners—In practice, it is hoped that the machine can work like a human such that it can replace the human to complete certain tasks. However, it is not easy to design corresponding algorithms for the machine because many tests need to be carried out for selecting appropriate parameters. Instead, an effective method is to teach the machine learn the human’s demonstrated behavior. It is noteworthy that the environment (system dynamics) may be not prior knowledge and only system state and control input are measurable. To this end, an adaptive IOC method is developed for imitation learning the human’s behavior, which is implemented online but requires only limited data. The proposed approach can be used in autonomous driving vehicle, service robot, and medical rehabilitation, etc. In future research, we will extent the proposed method to more complex environment.

Relative radiometric correction of multiple remote sensing images is essential for remote sensing data processing and its product generation. Due to the limitation of RGB color space, many existing RGB-based methods may have local radiance anomalies and strong dependence on reference images when dealing with dramatic radiance differences. Therefore, a hue-distance constrained relative radiometric correction method considering the Helmholtz-Kohlrausch effect (H-K effect) is proposed, by which the radiance differences of images can be corrected without the reference image to obtain better radiance consistency results. For the radiance anomalies in the correction process, the method in this paper avoids the RGB color space limitation and minimizes the channel correlation in the correction process, and optimizes the correction results by using the hue-distance to impose adaptive constraints on the correction process. Considering the influence of the H-K effect, the subjective visual perception of the human eye is introduced into the radiance correction of remote sensing images for the first time and the global perceptual lightness mapping is performed on the images, which extends the perceptual lightness mapping from pixel level to image level. Experimental results show that the proposed method can effectively eliminate the huge radiance differences in multiple remote sensing images, suppress the possible radiance anomalies, and obtain the correction results in line with the subjective perception of the human eye, and it is of great help to the subsequent remote sensing processing.

Hyperspectral images (HSIs) have widespread applications in geoscience, environmental monitoring, and resource management. However, in practical engineering applications, random stripe noise in HSIs severely affects data quality and accuracy, impacting the subsequent use of HSIs. In this paper, we propose an HSI intensity adaptive destriping method based on reference image-guided pixel clustering. The proposed method removes stripe noise through three main stages. First, the HSI is analyzed to manually select a specific band image with minimal impact from random stripe noise. This image is designated the initial reference image, and its specific location is identified. Second, the proposed pixel threshold classification method, deep maximum inter-class variance, is used for pixel threshold classification for band images that are contaminated with random stripe noise and adjacent to the reference image. Finally, the resulting pixels are classified into two categories according to noise intensity: high intensity and low intensity. The proposed intensity adaptive grayscale value reconstruction algorithm is used to remove stripe noise from each pixel category, and the denoised image is updated as a new reference image. Starting from the initial reference image, these steps are repeated along the spectrum to achieve destriping of all bands. We compare the proposed method against traditional and deep learning methods using real HSIs. The experimental results show that after denoising of the image using our method, both visual quality and quantitative evaluation metrics are significantly improved, with particularly excellent stripe removal performance observed for HSIs with significant grayscale variation.

Remote sensing images are increasingly being used in military and civilian fields. However, due to the influence of factors such as detector movement and temperature changes during the acquisition of remote sensing images, these images are often contaminated by stripe noise, which has adverse effects on tasks such as inversion, target detection, and semantic segmentation. Therefore, we developed a method for completely removing stripe noise, without destroying necessary information in remote sensing images, based on global search optimization and dense Gabor filters. Unlike other algorithms that directly solve the underlying image, this method applies Gabor filters to estimate stripe noise in the noisy image, locate stripe or nonstripe noise, and then categorize the stripe noise as sparse or dense according to its distribution frequency. Finally, the directionality of the stripe noise, its smoothness along the noise direction, and the local continuity of the underlying image are fully determined. The noisy image is divided into blocks along the stripe noise direction, and the intensity range of the stripe noise is estimated at different positions in each subimage. The stripe noise is directly solved within the estimated intensity range using global search optimization to achieve stripe noise removal. The proposed method can select different solutions for processing according to the type of stripe noise. A large number of experiments were conducted using simulated and real data, and the results demonstrated that the proposed method qualitatively and quantitatively outperformed current state-of-the-art stripe noise removal methods.

Multimodal image-matching success rates (SRs) are often low due to nonlinear radiation differences. Furthermore, when geometric transformations such as rotation and resolution exist between images, the matching SR between multimodal images decreases even further. (It is worth noting that experiments have shown the impact of scale to be relatively small; therefore, this discussion focuses only on the influence of resolution differences on multimodal image matching.) To tackle these challenges, we have enhanced the feature point extraction, description, and association processes in image matching. This has resulted in a robust multimodal image-matching framework that is invariant to rotation and resolution, with a high SR. Specifically, inspired by the concept of image pyramids, we designed a strategy for extracting feature points in multiple resolution dimensions. This enables assigning resolution dimension information to feature points and expanding the set of points to be matched. Building upon feature point extraction, we enhanced the Log-Gabor filter and designed a novel feature descriptor. This descriptor can work robustly in scenarios with modal differences and rotational variances ranging from 0° to 360°. Applying this descriptor to the matching framework helps to eliminate the influence of angle differences on feature point associations between images. Furthermore, to further improve the matching SR, we adopted a resolution dimension traversal retrieval strategy for the association of feature points. Based on this strategy, the number of correct matches (NCM) can be increased under the condition of the same feature points, thereby increasing the inlier rate of the matching results and enhancing the SR of the matching results. To evaluate the performance of this matching framework, we created a testing dataset containing 42496 pairs of images using publicly available datasets. These images cover six categories including optical, synthetic aperture radar (SAR), digital elevation model (DEM), infrared, map, and nighttime light, with three types of transformations between images: translation, rotation, and scaling. We conducted comparative experiments using the multimodal image rotation and resolution invariant feature transformation (MIRRIFT) method against five advanced multimodal feature matching methods with publicly available source code, namely, radiation-invariant feature transform (RIFT), locally normalized image feature transform (LNIFT), histogram of absolute phase consistency gradients (HAPCG), histogram of the orientation of weighted phase (HOWP), and Log-Gabor histogram descriptor (LGHD). The results demonstrate that the MIRRIFT method proposed in this article exhibits robustness to rotational, resolution, and modal differences in images. Specifically, it achieved an average SR improvement of 59%, an increase in average correct matching points by 12%, and an average matching accuracy of 1.97. The executable program and sample data will be made available at:https://github.com/Geng-Zemin/MIRRIFT

The goal of image stitching is to generate high-quality panoramic images with minimal computational cost. However, variations in viewpoint or scene depth can cause parallax effects in UAV images, complicating precise alignment and leading to artifacts such as ghosting, blurring, and misalignment. While advanced seamline detection algorithms reduce ghosting and blurring, structural distortions and misalignments near the seamline often remain, negatively affecting stitching quality. Moreover, these algorithms typically face challenges in balancing computational efficiency with alignment accuracy. In this paper, we propose a robust and flexible UAV image stitching method based on global optimal seamline detection and local alignment with seamline constraint. Our approach ensures precise alignment while maintaining processing efficiency. First, a global transformation-based alignment algorithm is used to pre-align the images to a common coordinate system. Then, an efficient weighted fast sweeping (WFS) algorithm is proposed to detect the globally optimal seamline, minimizing artifacts in overlapping regions caused by alignment errors and dynamic objects. Finally, an optical flow-guided local alignment method with seamline constraint is developed to correct residual misalignments along the seamline, reducing global structural distortion. Extensive experiments on a range of challenging datasets demonstrate that the proposed method outperforms existing approaches, producing more natural-looking stitching results.

Optical remote sensing (ORS) multimodal spatial-spectral fusion (MSF) aims to obtain high-resolution images containing fine-grained spatial details and high-fidelity spectral information, which are crucial for downstream tasks and real-world applications. Existing methods can yield promising outcomes in specific fusion scenarios. However, due to the coarse representation of spatial details and the imprecise alignment of spatial-spectral features, the majority of methods encounter difficulties in balancing spatial and spectral preservation. This imbalance tends to cause distortion in the fused image, rendering these task-specific methods less adaptable and more challenging to apply simultaneously to different ORS-MSF tasks. To address this gap, this article introduces a generic framework that focuses on generalization and practical applicability, rather than solely optimizing the performance of models in a specific fusion task. By conducting a comprehensive analysis of theoretical models and network architectures, we systematically decompose the fusion process into two distinct phases, namely, detail reconstruction and feature alignment. Consequently, the proposed framework consists of two fundamental components: low-frequency-driven high-frequency salient detail reconstruction (LHSDR) and coordinate-modal-guided spatial-spectral feature progressive alignment (CSFPA). In LHSDR, the joint spatial degradation process in various frequency directions from diverse modal data is estimated and salient details are derived in a hierarchical integration, with low frequencies driving high ones. These coupled high-frequency details could lay the foundation for subsequent implementation of high-fidelity fusion. Furthermore, CSFPA estimates the joint spectral degradation process by establishing coordinate-mode relations between coupled high-frequency details and corresponding spectral information in the continuous domain. As a result, high spatial-spectral fidelity fused images are obtained through fine detail reconstruction and accurate feature alignment. Ten datasets derived from three different ORS-MSF tasks are utilized for an experiment, comprising eight simulated and five real test sets. Our proposed methodology demonstrates robust fusion performance and generalization capability on data with different spectral bands at various resolutions. All implementations will be published on our website.

Achieving color consistency is essential for stitching large-area optical satellite imagery. The narrow swath width of individual images, combined with varying acquisition conditions, inherently introduces color differences. These manifest as marked disparities in brightness, color tone, and local contrast, degrading overall regional consistency. Existing methods primarily focus on correcting color inconsistencies between adjacent images, while often overlooking intra-image illumination non-uniformity, thereby propagating radiometric errors into optimization frameworks. Whisk-broom sensors, which can acquire imagery over a much wider swath width along the parallels, frequently exhibit substantial intra-image brightness variations, particularly in high-latitude regions. Compounded by frequent cloud cover and rapid temporal changes of features, extracting reliable color correspondences for optimization becomes intractable. To address these challenges, we propose a novel color correction framework that simultaneously considers intra-image illumination non-uniformity and radiometric variations caused by cloud prevalence and dynamic surface changes. First, a solar elevation angle map is extracted for down-sampled source image based on their geographic metadata and sensor geometry. An inverse compensation based on the normalized sine value of the solar elevation angle is then applied to mitigate brightness disparities caused by varying incident radiance. Second, to address atmospheric effects that vary with wavelength, such as differential absorption and scattering that cause color casts especially in low-illumination regions, a reference spectral channel is selected to guide the correction. Finally, we introduce a hybrid strategy for selecting reliable color correspondences in overlapping regions, using both grayscale and texture similarity under complex coverage conditions. Residual radiometric information is incorporated into a cost function, which jointly considers original color control and overall color balance to enhance the global consistency of the corrected mosaic. Extensive experiments conducted on imagery from the Wide Swath Imager (WSI) of DaQi-1 (DQ-1) and the Chinese Ocean Color and Temperature Scanner (COCTS) of HaiYang-1E (HY-1E) demonstrate that the proposed method effectively removes uneven illumination and color discrepancies. Compared to three state-of-the-art methods, our approach achieves superior performance in both visual quality and quantitative metrics.

As the spatial resolution of satellite remote sensing increases, high-resolution remote sensing (HRRS) images provide richer surface structure details, also make noise and artifacts more obvious than before. Therefore, the building change detection (BCD) task is affected by irrelevant features caused by explicit noise and artifacts, resulting in incoherent semantics and rough boundaries of buildings, which leads to fragmented change detection results and blurred boundaries between background and changes. To address this, we propose the Progressive Temporal Difference Network (PTDNet). PTDNet employs an interleaved CNN-Transformer encoder to enhance semantic structural correlation, supplemented by a Global Information Supplement Module (GISM) for semantic alignment. The Progressive Temporal Difference Module (PTDM) then suppresses artifacts and reinforces change semantic coherence through multi-stage temporal difference fusion. Finally, a Change Guidance Module (CGM) with deep semantic-guided attention refines change boundary. During this process, multi-scale features are effectively aggregated layer by layer to produce the final binary prediction map. We conducted comparative experiments with other state-of-the-art (SOTA) methods on three public BCD datasets, namely WHU, LEVIR and SYSU, and the results show that our PTDNet achieves the highest F1-scores of 93.07%, 91.80% and 83.46% respectively. The code for this work is publicly available at https://github.com/Caijiaqi85/PTDNet-CD.

The widespread use of time delay and integration charge-coupled device (TDI CCD) technology in high-resolution spaceborne optical cameras has made high-frequency jitter effects a common issue, resulting in different levels of distortion in images. Current methods mostly concentrate on correction of obviously high levels of geometric distortion. Focusing on low levels of geometric distortion, which are more difficult to accurately detect, this paper proposes a progressive learning-based correction method for high-frequency jitter distortion in remote sensing images from spaceborne TDI CCD cameras, utilizing a Generative Adversarial Network (GAN). First, a distorted dataset with diverse jitter levels for progressive training is generated through jitter simulation model by adjusting the parameters. Then, a GAN model is employed for the correction task. The generator consists of the Distortion Net for geometric distortion correction and the Detail Enhancement Net for image detail restoration. Finally, a progressive learning strategy is used to gradually enhance the ability of network to correct minor geometric distortion. The proposed method is validated using simulated images and real-world satellite images. Experimental results demonstrate that the proposed method outperforms existing restoration methods both in simulated datasets and practical scenarios.

With the continuous development of imaging technology and the gradual expansion of the amount of image data, how to achieve high compression efficiency of high-resolution images is a challenge problem for storage and transmission. Image rescaling aims to reduce the original data amount through downscaling to facilitate data transmission and storage before encoding, and reconstruct the quality through upscaling after decoding, which is a key technology to assist in high-ratio image compression. However, existing rescaling approaches are more focused on reconstruction quality rather than image compressibility. In repetitive observation scenarios, multi-temporal images brought by periodic observations provide an opportunity to alleviate the conflict between reconstruction quality and compressibility, that is, the historical images as reference indicates what information can be dropped at downscaling to reduce the information content in downscaled image and provides the dropped information to improve the image restoration quality at upscaling. Based on this consideration, we propose a novel multi-temporal assisted reference-based image rescaling framework (RefScale). Specifically, a referencing network is proposed to calculate the similarity map to provide the referencing condition, which is then injected into the conditional invertible neural network to guide the information drop at the downscaling stage and information fusion at the upscaling stage. Additionally, a low-resolution guidance loss is proposed to further constrain the data amount of the downscaled image. Experiments conducted on both satellite imaging and autonomous driving show the superior performance of our approach over the state-of-the-art methods.

High-resolution remote sensing images (HRRSIs) contain abundant details and texture information. Existing lossy compression methods employ quantization to eliminate redundant information, but this leads to irreversible effects on the subtle details of HRRSIs. This paper proposes a quantization loss compensation network to address this issue. We use a trainable variational autoencoder to learn the details and texture information of HRRSIs from the error between pre-and post-quantized latent representations. During the encoding of HRRSIs, the quantization errors are inputted into the encoding module of the compensation network to generate bitstreams. When it comes to image decoding, using the decoding module of the compensation network to generate quantization loss compensation information, which, together with the quantization latent representations, contributes to the HRRSIs reconstruction. To validate the effectiveness of our approach in reconstructing details of HRRSIs, we conducted experiments on two remote sensing datasets. The experimental results also indicate that our method exhibits superior compression performance.

The on-board processing systems of high-resolution optical satellites performing hotspot observations must be of high efficiency and high precision. To meet these requirements, a real-time geometric correction (RGC) method was developed based on a target-driven virtual linear-array reimaging (VLR) model. First, the undistorted VLR was used instead of the original distorted physical linear array to achieve a relative orientation of the sub-images of the sensor, ensuring the relative geometric accuracy of the original multilinear array and multiband images. Next, the coordinate position of the region of interest (ROI) in the original image was accurately located in a step-by-step strategy. According to the object-space projection model (OPM) of the VLR and the physical strict model (PSM) of the original image, a coordinate mapping relationship could be established. Finally, the RGC of the ROI image was achieved through GPU-accelerated grayscale resampling. The method was then tested using panchromatic (PAN) and multispectral scanner (MSS) data of LuoJia3-02. The results showed that the processed images exhibited satisfactory band registration accuracy and maintained geometric consistency among various linear-array scanners. Furthermore, in terms of performance, the ROI processing speed was fully adapted to the imaging rate, which fulfilled the real-time on-board processing demands.

Thermal environment fluctuations introduce the attitude low-frequency error (ALFE) related to the orbital period. This causes the geometric positioning accuracy of visible and infrared multispectral sensor (VIMS) imagery of GaoFen-5B (GF-5B) satellite to vary from tens of meters to hundreds of meters. In this letter, a compensation method for ALFE is proposed based on the temporal and spatial characteristics. By considering the spatial characteristics of low-frequency error (LFE) in different regions, the models of internal and external LFE compensation were established using the segmented Fourier series model and the parameter of latitude. The sequential calibration and compensation method with different time intervals was proposed to address the temporal drift of LFE. Experimental results demonstrated that the internal and external LFE compensation improves geometric positioning accuracy of VIMS imagery from 6.927 to 2.428 pixels and from 3.045 to 1.438 pixels, respectively. The proposed method ensures the stability and consistency of geometric positioning accuracy of VIMS imagery across different time periods and regions.

Vehicle detection and tracking in satellite video is essential in remote sensing (RS) applications. However, upon the statistical analysis of existing datasets, we find that the dim vehicles with low radiation intensity and limited contrast against the background are rarely annotated, which leads to the poor effect of existing approaches in detecting moving vehicles under low radiation conditions. In this letter, we address the challenge by building a small and dim moving cars (SDM-Car) dataset with a multitude of annotations for dim vehicles in satellite videos, which is collected by the Luojia 3–01 satellite and comprises 99 high-quality videos. Furthermore, we propose a method based on image enhancement and attention mechanisms to improve the detection accuracy of dim vehicles, serving as a benchmark for evaluating the dataset. Finally, we assess the performance of several representative methods on SDM-Car and present insightful findings. The dataset is openly available athttps://github.com/TanedaM/SDM-Car.

To enhance the collaborative intelligence of a machine, it is important for the machine to understand what behavior a human may adopt to interact with the machine when performing a task in shared control. In this study, an online behavior learning method is proposed for continuous-time linear human-in-the-loop shared control systems by using the system state data only. A two-player nonzero-sum linear quadratic dynamic game paradigm is used for modeling the control interaction between a human operator and an automation that actively compensates for human control action. In this game model, the cost function representing the human behavior is assumed to have an unknown weighting matrix. Here, we want to learn the human behavior or retrieve the weighting matrix by using the system state data only. Accordingly, a new adaptive inverse differential game (IDG) method, which integrates concurrent learning (CL) and linear matrix inequality (LMI) optimization, is proposed. First, a CL-based adaptive law and an interactive controller of the automation are developed to estimate the feedback gain matrix of the human online, and second, an LMI optimization problem is solved to determine the weighting matrix of the human cost function. Finally, simulation results on a cooperative shared control driver assistance system are provided to elucidate the feasibility of the developed method.

In this paper, the issue of human behavior learning (HBL) is addressed for a class of nonlinear human-in-the-loop (HiTL) systems where the human operator is viewed as a nonlinear optimal controller. Owing to its outstanding interpretability and strong nonlinear representation capability, the Takagi-Sugeno (T-S) fuzzy model is employed to represent the nonlinear HiTL control system and approximate the unknown human control law based on the parallel distributed compensation (PDC) scheme. A quadratic-like cost function with fuzzy weighting matrices is built to depict the human behavior, which conforms to human thinking and is unknown to the machine. The aim of the HBL is to retrieve the fuzzy weighting matrices such that the human control law will be optimal in the sense of minimizing the retrieved cost function. In the proposed HBL scheme, the state-dependent Riccati equation (SDRE) based nonlinear optimal control technique plays an important role, which has a similar structure to the linear quadratic regulator (LQR) theory and thus is of low computational complexity. With the help of the PDC based fuzzy approximator for the unknown human control law, a two-step procedure is proposed for the HBL. First, a filter-based adaptive law is developed to learn the gain matrices of the fuzzy approximator using the system state data only. The convergence analysis of the adaptive estimator is also given. Then, a semidefinite programming (SDP) problem with the quadratic objective function can be set up for determining the fuzzy weighting matrices of the cost function. The simulation study on a steering control system of the intelligent vehicle is given to show the effectiveness and applicability of the developed approach.

Recently, many new results of sharpness assessment for digital images have been achieved, which help to select valuable images from massive images with ragged quality. However, remote sensing images encompass a wide range of scenes with diverse characteristics, and their acquisition is often influenced by blurs and noises. Many commonly used sharpness assessment methods based on uniform metrics face challenges in ensuring both subjective and objective impartiality when applied to remote sensing images. Therefore, a novel method for assessing the sharpness of remote sensing images based on a deep multi-branch network considering scene features is proposed. In the method, a multi-task module, comprising scene classification and sharpness assessment tasks, is proposed to comprehensively consider the potential impact of diverse scene characteristics on the assessment of sharpness. The accuracy of sharpness assessment is improved by sharing features that reflect the correlation between the scene and sharpness. To overcome the issue of imbalanced task predictions during the joint training of multiple tasks, a total loss function using gradient balance strategy is designed. In addition, the improved attention module and the feature fusion module are used to better utilize feature information at different scales. Experimental results obtained from datasets demonstrated that the proposed method can outperform the existing comparable methods and achieve satisfactory results, proving its feasibility and effectiveness.

To increase the observation performance of optical remote sensing satellites (ORSSs), onboard camera systems usually perform multislice and multiband imaging. The resulting multiband or multislice subimages require correction into an aligned and stitched complete image, to provide users with standard image products. In a linear array sensor, multiband and multislice sensor correction processing is achieved by mapping subimages on the same virtual linear array based on the determined geometric parameters and the principle of object space consistency. However, a planar array sensor has the same geometric parameters outside the camera; therefore, object space consistency is essentially equivalent to image space consistency, which simplifies coordinate mapping calculations. In this study, we developed a novel sensor correction method based on image space consistency for planar array sensors and combined it with a method to determine the relative geometric parameters between bands. The internal geometric parameters of multiple bands were determined under connection constraints between bands established by consistent image space pointing. Next, the coordinate mapping model between the virtual image point and physical subimage point was established, and the angular resolution was introduced for coordinate solving. Based on strict geometric correspondence between the subimages and virtual images, the whole image with interband registration and interslice stitching was generated from the subimages. We experimentally verified the accuracy and effectiveness of our method using real multiband and simulated multislice plane array sensor images from GF-4 satellites. The corrected whole image showed satisfactory interband registration accuracy and internal geometric accuracy.

Remote sensing image change detection is extensively utilized in various applications in the field of remote sensing, particularly in the realm of cropland conservation, where it plays a critical role in protecting the agro-ecosystem and ensuring global food security. However, the progressive improvement in resolution and size of remote sensing imagery has led to a ’scale gap’ challenge in the detection of small building changes in cropland areas. To address this challenge, an innovative multi-scale feature fusion change detection network (M-Swin) based on transformer using hierarchical windows is proposed. In order to obtain clearer edges and better separation of the change results, a novel saimese transformer encoder (MSW encoder) is proposed, which can better capture the change information in small building through hierarchical windows and fuse the multi-scale feature obtained from different windows. To effectively reduce missed and misdetected small-area of changing buildings, a novel bi-temporal image feature fusion module (BFFM) is proposed, which can enhance the features based on a priori guidance, thus improving the saliency of change regions. Additionally, a new remote sensing image change detection dataset for cropland, called LuojiaSET-CLCD, has been proposed. Experimentally demonstrates that M-Swin has good potential for highly accurate change detection of small buildings within cropland areas and outperforms several newly existing methods in three datasets (LEVIR, WHU-CD and LuojiaSET-CLCD). Our dataset will be publicly available at https://github.com/RSIIPAC/LuojiaSET-CLCD.

Time delay integration charge-coupled device (TDI CCD) is sensitive to the platform’s stability during push-broom imaging. Due to variations in total integration time, panchromatic and multispectral images suffer varying degrees of geometric distortion caused by satellite jitter with high frequency, which leads to different inner distortion in different band images and different band-to-band mismatching errors between different band combinations. To address this problem, this paper proposes a rigorous parallax observation model considering multi-stage integration time and presents a jitter distortion correction method for remote sensing panchromatic and multispectral images captured by TDI cameras based on it. First, the law of the amplitude attenuation and phase offset of platform jitter deviation on the image under different TDI stages is determined through simulation verification. Then, the rigorous parallax observation model is proposed to establish an accurate relationship between the relative jitter error of two multispectral images with multi-stage integration and the absolute single-stage integration jitter error by introducing the amplitude attenuation factor and phase offset. Finally, the jitter distortion curves of images with different integration stages and integration time can be reconstructed based on the estimated absolute jitter error and the imaging parameters. Subsequently, the jitter distortion can be further corrected by image resampling. The proposed method was verified through both simulation and real data experiments using GaoFen-9 satellite images. Experimental results show that the proposed method can effectively correct high-frequency jitter distortion in panchromatic and multispectral images, which cannot be corrected by traditional single-stage integration jitter detection model.

One goal of artificial intelligence (AI) research is to teach machines how to learn from humans, such that they can perform a certain task in a natural human-like way. In this article, an online adaptive inverse reinforcement learning (IRL) approach to human behavior modeling is proposed to enhance machine intelligence for a class of linear human-in-the-loop (HiTL) systems using the state data only, where the human behavior is described by a linear quadratic optimal control model with an unknown weighting matrix for the quadratic cost function. First, an integral concurrent adaptive law is developed to learn the human feedback gain matrix online using the demonstrated state data only, which removes the persistent excitation (PE) conditions required by traditional adaptive estimation approaches and thus is more in line with real applications. Then, with the learned feedback gain matrix, the IRL problem is formulated as a linear matrix inequality (LMI) optimization problem, which can be efficiently solved to retrieve the weighting matrix of the human cost function. Finally, a simulation example is provided to illustrate the effectiveness of the proposed approach.

The key to enhancing machine intelligence is to make the machine learn how human beings perform tasks. In this article, the issue of finite-horizon inverse linear quadratic (LQ) optimal control is investigated for human behavior learning in a class of human-in-the-loop (HiTL) systems. A novel finite-horizon inverse optimal control (FHIOC) approach is developed by integrating time-varying parameter identification and linear matrix inequality (LMI) optimization techniques. The proposed approach covers three steps: by only using the system state measurement, 1) an offline identification method is developed to provide a batch least-squares estimation for the human time-varying feedback gain matrix; 2) a recursive least-squares adaptive law is proposed to online learn the human time-varying feedback gain in real time; and 3) the weighting matrices of the human cost function are recovered via the time-convexity and LMI optimization techniques with the learned time-varying feedback gain. Finally, the validity of the proposed methods is supported by a supplementary steering system of an intelligent vehicle.

Relative radiometric calibration (RRC) is a crucial step in enhancing the quality of satellite images, and it serves as a fundamental technology to ensure the reliability of information extracted from these images. Traditional side-slither RRC methods typically use homogeneous scenes as imaging sites to ensure approximately identical input for detectors of multiple non-collinear time-delay integration-charge-coupled devices (TDI-CCDs). However, as spatial resolution improves, the demand for site uniformity is increasing, limiting the application of these traditional methods. Therefore, we propose a push-broom data-aided side-slither RRC method, which involves sequential calibration of a single TDI-CCD and an entire field of view (FOV). The first step is designed to eliminate response differences within each TDI-CCD, while the latter incorporates push-broom data to determine the adjacent radiometric relationship via adjustment with maximum standard deviation preservation (MSDP). Experiments show that our method has achieved better results in both visual effect and quantitative assessment, compared with the other two advanced RRC methods.

The main purpose of cloud detection is to estimate cloud coverage and thus determine whether to transmit remote sensing images to earth or execute subsequent tasks based on cloud coverage. Fast and accurate cloud coverage estimation is a necessary preprocessing step on board. Therefore, we propose a new approach for cloud coverage estimation using a regression network to directly predict the coverage. A cloud coverage estimation network, which is termed$\text{C}^{2}\text{E}$-Net, is proposed in this work. The proposed network consists of three modules, including an encoder for representation feature extraction, a coverage estimation for predicting the cover rate of clouds, and an auxiliary supervision module for improving the performance of the model. To verify the effectiveness of our method, experiments are performed on two open-source datasets (Landset 8 Biome dataset and GaoFen-1 WFV dataset). Our method effectively improves the efficiency of cloud detection by at least doubling, while keeping the estimation error low.

The accurate matching of multisource, multi-temporal remote sensing images is challenging because of significant nonlinear intensity differences (NIDs) and severe geometric distortions. To address these problems, we developed a robust image matching method: oriented filter-based matching (OFM). OFM is insensitive to NIDs, while exhibiting scale and rotational invariance. First, salient feature points with multiscale attributes were detected in the Gaussian-scale space of the input images. Then, the images were convoluted using multi-oriented filters, and unified feature maps were constructed by the extraction of orientation indices using effective data pooling operations. The constructed feature maps were highly resistant to NIDs. Five filters were integrated into the OFM framework to investigate their applicabilities in different application scenarios. Next, a novel rotation-invariant feature descriptor was constructed, using a dominant direction determination approach and a descriptor-grouping strategy. The dominant direction determination approach enables accurate dominant direction estimation, whereas the descriptor-grouping strategy improves the stability of the method under different rotational angles. Finally, brute-force matching was implemented to obtain initial matches; an improved mismatch elimination method was used to identify reliable putative matches. To evaluate the performance of OFM, we created a large dataset comprising 4,427 pairs of multitemporal optical–optical, optical–synthetic aperture radar (SAR), optical–infrared, and optical–depth images. OFM outperformed state-of-the-art methods in terms of number of correct matches, recall, inlier ratio, root mean square error and success rate. Our implement is publicly available1.

The traditional strategy of acquiring satellite images involves transmitting compressed satellite data to ground stations solely via the downlink, without utilizing the uplink. In this paper, we propose an enhanced remote sensing (RS) image compression approach that utilizes uplink assistance to improve compression efficiency. By leveraging the uplink, historical images from ground stations can serve as reference images for on-orbit compression, effectively eliminating spatio-temporal redundancy in RS images. However, due to radiation variations among RS images captured on different dates, pixel-wise referencing as employed in the prior codec paradigm is insufficient. To address this, we propose a novel dual-end referencing downsampling-based coding (RefDBC) framework. At the encoder, relevance embedding evaluates reconstructability and records information to restore texture details from the reference prior to downsampling. At the decoder, relevance-based super-resolution uses the identical reference and recorded relevance information to reconstruct the decoded low-resolution image. By incorporating relevance referencing, RefDBC effectively mitigates fake texture generation caused by downsampling and compression, achieving significant bitrate savings ranging from 35%-70% compared to standard, learning-based, and DBC compression baselines in experiments on Spot-5 and Luojia3 images. Code, data, and pretrained models are available online at https://github.com/WHW1233/RefDBC.

Cloud detection plays an important role in remote sensing image quality evaluation, information acquisition, and analysis. For most optical satellites, the acquired multispectral bands include only three visible bands and one near-infrared band. How to achieve fast and accurate cloud detection with this limited number of bands is a problem worth studying. In this paper, a cloud detection algorithm (SMFCD) based on a saliency-based mixed feature map of images is proposed. This algorithm first calculates the saliency characteristics from the band mean and haze optimized transformation (HOT) map. The transmission map obtained from dark channel prior theory is used to combine the mixed feature map for subsequent detection. The proposed algorithm generates the initial thick cloud mask using a segmentation method based on the Otsu algorithm later. Guided filtering is then used to refine the cloud mask. The final cloud detection result is acquired through post processing. The algorithm is tested on six different datasets. It is shown that the algorithm can obtain good detection results for most images, with overall accuracies from 0.860 to 0.969 on these datasets. The average time consumption on these datasets reaches 4.4~42.1 seconds, and a fast version of the proposed algorithm reduces these times by about 50%. The proposed algorithm can be used for remote sensing image quality evaluation and subsequent application preprocessing with greater adaptability and flexibility than existing algorithms, since it does not depend on specific satellite radiometric calibration coefficients.

Video frame interpolation has made great progress in estimating advanced optical flow and synthesizing in-between frames sequentially. However, frame interpolation involving various resolutions and motions remains challenging due to limited or fixed pre-trained networks. Inspired by the success of the coarse-to-fine scheme for video frame interpolation, i.e., gradually interpolating frames of different resolutions, we propose a progressive boosting network (ProBoost-Net) based on a multi-scale framework to achieve flexible recurrent scales and then gradually optimize optical flow estimation and frame interpolation. Specifically, we designed a dense motion boosting (DMB) module to transfer features close to real motion to the decoded features from the later scales, which provides complementary information to refine the motion further. Furthermore, to ensure the accuracy of the estimated motion features at each scale, we propose a motion adaptive fusion (MAF) module that adaptively deals with motions with different receptive fields according to the motion conditions. Thanks to the framework's flexible recurrent scales, we can customize the number of scales and make trade-offs between computation and quality depending on the application scenario. Extensive experiments with various datasets demonstrated the superiority of our proposed method over state-of-the-art approaches in various scenarios.

This article develops a novel state observer for delayed reaction–diffusion neural networks by utilizing incomplete measurements. To reduce the transmission cost efficiently, the space domain is divided into$L$parts and only partial information needs to be measured in every subdomain, such as a point in one-dimensional space, a line and a plane in two- and three-dimensional space, respectively. In addition, the time domain is divided: the measured output signals are transmitted intermittently. Then, new conditions that assure the asymptotic stability of observation error system are derived based on the Lyapunov direct method and several inequality techniques. Finally, the proposed approach’s effectiveness is demonstrated via three numerical examples.

Scene classification of aerial images is the basis of automatic recognition of complex scenes, and it is also a challenging computer vision task. In recent years, with the rapid development of deep learning, the semantic feature extraction method based on a convolutional neural network (CNN) has made great progress. Moreover, a recent study indicates that combining the semantic information of deep-layer features with the detailed texture information of shallow-layer features in CNN can further improve the performance of classification. In this letter, an end-to-end multilevel feature-based network named multilevel inheritance network (MINet) is proposed for aerial scene classification. First, the feature extraction module based on the feature pyramid network (FPN) is used to get multilevel feature maps. In the process of merging shallow features, high-level semantics of deep-layer are inherited. Then, an attention mechanism is added after the multilevel features to reduce the interference of redundant information and noise. Finally, we use a feature fusion module to automatically learn the weight of each feature layer and make a comprehensive decision. The effectiveness of the proposed method is verified in AID, WHU-RS19 and NWPU-RESISC45 datasets. Results show that the proposed method achieves competitive classification accuracy.

In recent years, with the development of high-resolution remote sensing (RS) images and deep learning technology, high-quality source data and state-of-the-art methods have become increasingly available, and great progress has been made in change detection (CD) in RS fields. However, existing methods still suffer from weak network feature representation and poor CD performance. To address these problems, we propose a novel CD network, called dual-pathway CD network (DP-CD-Net), which can help enhance feature representation and achieve a more accurate difference map. The proposed method contains a dual-pathway feature difference network (FDN), an adaptive fusion module (AFM), and an auxiliary supervision strategy. Dual-pathway FDNs can effectively enhance feature representation by supplementing the detailed information from the encoding layers. Then, we use the AFM method to fuse the difference maps. To solve the problem of training difficulty, we use the auxiliary supervision strategy to improve the performance of DP-CD-Net. We conduct extensive experiments to validate the performance of the proposed method on the LEVIR-CD dataset. The results demonstrate that the proposed method performs better than existing methods.

Semantic segmentation plays a vital role in the segmentation of remote sensing field for its wide range of applications. The major current method for segmentation of remotely sensed imagery is using multiple scales strategy to improve the performance of segmentation networks. However, the ground object with uncertain scale in high-resolution aerial imagery is difficult to be segmented with conventional models. To address this problem, an adaptive feature selection module is designed, in which attention module learns weight contributions of each feature blocks in different scales. We employ the pyramid scene parsing network (PSPNet), DeepLabV3, and U-Net with the proposed module to conduct experiments on two benchmarks (the Vaihingen set and the WHU Building data set). The experimental results and comprehensive analysis validate the efficiency and practicability of the proposed method in semantic segmentation of remote sensing images.

Interpolating video frames involving large motions remains an elusive challenge. In case that frames involve small and fast-moving objects, conventional feed-forward neural network-based approaches that estimate optical flow and synthesize in-between frames sequentially often result in loss of motion features and thus blurred boundaries. To address the problem, we propose a novel Recurrent Motion-Enhanced Interpolation Network (ReMEI-Net) by assigning attention to the motion features of small objects from both the intra-scale and inter-scale perspectives. Specifically, we add recurrent feedback blocks in the existing multi-scale autoencoder pipeline, aiming to iteratively enhance the motion information of small objects across different scales. Second, to further refine the motion features of the highly moving objects, we propose a Multi-Directional ConvLSTM (MD-ConvLSTM) block to capture the global spatial contextual information of motion from multiple directions. In this way, the coarse-scale features can be utilized to correct and enhance the fine-scale features through the feedback mechanism. Extensive experiments on various datasets demonstrate the superiority of our proposed method over state-of-the-art approaches in terms of clear locations and complete shape.

This article presents a novel reliable fuzzy output feedback controller for a class of semilinear parabolic partial differential equation systems with Markov jump actuator failures. First, the control strategy's novelties include the following aspects: 1) the considered system is represented by using a fuzzy modeling approach, based on which a new asynchronous fuzzy observer is constructed via utilizing a series of discrete output signals that are induced by samplers and quantizers; 2) a novel Markov jump input model, which is more fit for real applications, is introduced to depict various stochastically occurring actuator faults; and 3) inspired by the above discussion, a reliable mode-dependent fuzzy piecewise control strategy, which only needs limited actuators, is developed. Then, some new conditions, which can ensure that the closed-loop system is finite-time bounded, are established. Furthermore, some slave matrices are introduced to relax the strict constraints caused by asynchronous membership functions. Finally, two simulation examples are provided to support the validity of the proposed method.

This article investigates a spatial-$\mathcal {L}^{\infty }$-norm-based reliable bounded control problem for a class of nonlinear partial differential equation systems in a finite-time interval. The main novelties are reflected in the following aspects: 1) inspired by the sector-nonlinearity approach, the considered nonlinear system is reconstructed by a Takagi–Sugeno fuzzy model, which provides an effective method for control design. Besides, several actuator failures, such as stuck faulty, outage faulty, and bias faulty, are taken into account and modeled by a novel Markov process; 2) partial areas’ states are sampled and transmitted based on a new distributed event-triggered communication strategy, which reduces the cost of the system design and saves the limited network resources to some extent; and 3) on the basis of the first two works, a new piecewise fuzzy controller, which requires fewer actuators compared with the distributed control method, is constructed. Then, some sufficient conditions to guarantee the finite-time boundedness (in the sense of spatial$\mathcal {L}^{\infty }$norm) and mixed$\mathcal {L}_{2}-\mathcal {L}_{\infty }/\mathcal {H}_{\infty }$disturbance attenuation performance are established, and a new linear matrix inequality relax technique is introduced to deal with the strict constraint that is caused by the asynchronous phenomenon between plant and controller. Finally, two simulation studies are given to illustrate the effectiveness and advantages of the developed controller.

On-orbit geometric calibration (GC) is always performed to compensate for geometric distortion from the satellite’s camera. However, the traditional GC method is complex and difficult to apply broadly due to its reliance on a rigorous physical model (RPM), which involves not only complex processing of attitude, orbit, and time, but also the transformation among multiple coordinate systems. Additionally, the RPM is closely related to designs of satellites and cameras, which increases the complexity of the GC, and thus reduces generalizability. This paper proposes a practical and robust GC method to prevent camera distortion based on a standardized rational polynomial coefficient (RPC) model. Through a series of innovations, including stepwise optimization, a priori gross error elimination, the adjustment model with angular resolution, and the correction for the bias field-of-view (FOV) distortion, we were able to achieve robust GC for camera distortion, as well as accurate splicing and registration among segmented images. Method validation using data from the linear-array camera of the ZiYuan3-02 satellite, and the area-array camera of the GaoFen-4 satellite, produced satisfactory results, indicating that our method effectively compensates for systematic geometric distortion such that consistent GC and accuracy comparable with that of traditional RPM-based methods can be obtained.

Vehicle counting is important for smart city applications such as logistics management, traffic estimation, and financial analysis. To perform vehicle counting using aerial images, researchers have proposed many algorithms, including detection-based, regression-based and density-based methods. However, most of these algorithms are only applicable to high-resolution images, which require clear vehicle outlines. For the reasons of acquisition difficulty, frequency and cost, it is necessary to explore methods for vehicle counting using low-resolution or even very low-resolution images. We build a cross-resolution vehicle counting (CRVC) dataset, including 192 very low-resolution images and 8 high-resolution images of a port from 2016 to 2019. For this task, we propose a novel vehicle counting via cross-resolution spatial consistency and intra-resolution time continuity constraints. The segmentation map is first obtained by semantic segmentation with the prior information above. The vehicle coverage rate relative to the located parking lot is calculated and then converted to vehicle area. Finally, the relationship between the area and the number of vehicles is established by regression. Experiments show that the vehicle counting results obtained by our method are highly consistent with the annotations and outperform other state-of-the-art methods. Our method is also applicable for images with a lower resolution of 10m and other locations. Code, data and pre-trained models are available online at https://github.com/hbsszq/Vehicle-Counting-in-Very-Low-Resolution-Aerial-Images.

A novel state estimator is designed for genetic regulatory networks with reaction-diffusion terms in this study. First, the diffusion space (where mRNA and protein exist) is divided into several parts and only a point, a line, or a plane, etc., is measured in every subspace to reduce the measurement cost effectively. Then, samplers and network-induced time delay are considered to meet the network transmission requirement. A new criterion to ensure that the estimation error converges to zero is established by using the Lyapunov functional combined with Wirtinger's inequality, reciprocally convex approach, and Halanay's inequality; furthermore, the estimator's parameters are derived by solving linear matrix inequalities. Finally, two simulation examples (including one-dimensional and two-dimensional spaces) are presented to demonstrate the developed scheme's applicability.

Although time delay integration charge-coupled devices (TDI CCDs) have been widely used in high-resolution spaceborne optical cameras, they are sensitive to satellite jitter: the images obtained by them are affected by both distortion and blur. Therefore, according to the multistage integral imaging characteristics of TDI CCDs, this article not only proposes a continue dynamic shooting model (CDSM) to reflect the real push-broom mode of the satellite but also presents a method containing jitter detection and image restoration based on it. In the presented method, the CDSM subdivides the TDI CCD integration intervals. The subdivision number of CDSM is determined by the proposed integral transformation function (ITF). Then, it feeds back into the ITF and also contributes to the point spread function (PSF) estimation. Among the abovementioned, ITF defines the relationship between the parallax images and the jitter curve, and aims to improve the jitter detection performance. Finally, an adaptive image restoration based on context is conducted, which combines time, space, and spectrum information. Besides the simulated images, multispectral images of GaoFen-1 02 satellite were also adopted to validate the performance of the presented method. Experimental results indicate that the accuracy of the jitter detection is increased, and the geometric and radiometric qualities of restored images are also improved.

The log-ratio (LR) operator has been widely employed to generate the difference image for synthetic aperture radar (SAR) image change detection. However, the difference image generated by this pixelwise operator can be subject to SAR images speckle and unavoidable registration errors between bitemporal SAR images. In this letter, we proposed a spatial metric learning method to obtain a difference image that is more robust to the speckle by learning a metric from a set of constraint pairs. In the proposed method, the spatial context is considered in constructing constraint pairs, each of which consists of patches in the same location of bitemporal SAR images. Then, a semidefinite positive metric matrix M can be obtained by the optimization with the max-margin criterion. Finally, we verify our proposed method on four challenging data sets of bitemporal SAR images. Experimental results demonstrate that the difference map obtained by our proposed method outperforms than other state-of-the-art methods.

Owing to the effects of atmospheric refraction, the path of light propagation is bent, making the three-point collinear principle inapplicable and influencing the geometric accuracy of high-resolution optical satellite geometric positioning. This letter presents a novel geometric positioning method with atmospheric refraction calibration for optical remote sensing satellites. The atmospheric ellipsoid model is established using the measured atmospheric parameters and accurately describes the shape and characteristics of the real atmosphere. With iterative processing of geometric positioning and atmospheric refraction calibration, the path of light propagation in atmospheric ellipsoids is calibrated, and the real coordinates of the ground object are positioned accurately. With the advantages of simplicity and independence of sensors, the rational function model with atmospheric refraction calibration is proposed to achieve geometric positioning with higher geometric accuracy. Experimental results demonstrate that the proposed model can calibrate atmospheric refraction error and improve the geometric accuracy of the optical imagery with a large view angle. Furthermore, compared with the refraction index using the measurement data, it is proven that atmospheric refraction calibration should be implemented according to the measured atmospheric parameters during imaging instead of using the empirical model.

This article focuses on the asynchronous output-feedback control design for a class of nonlinear Markov jump distributed parameter systems based on a hidden Markov model. Initially, the considered systems are represented by a Takagi-Sugeno fuzzy model via a sector nonlinearity approach. Furthermore, asynchronous quantizers are introduced to save the limited communication resource in engineering applications. Then, based on the Lyapunov direct method and some inequality techniques, a series of novel stability criteria, which guarantee the finite-time boundedness and H∞disturbance attenuation performance of the target plants, is established in the form of spatial differential linear matrix inequalities. Finally, a simulation study is provided to verify the viability of the developed approach.

Object detection in high spatial resolution remote sensing images (HSRIs) is an important part of image information automatic extraction, analysis, and understanding. The region of interest (ROI) scale of object detection and the object feature representation are two vital factors in HSRI object detection. With respect to these two issues, this article presents a novel HSRI object detection method based on convolutional neural networks (CNNs) with suitable object scale features. First, the suitable ROI scale of object detection is obtained by compiling statistics for the scale range of objects in HSRIs. Then, a CNN framework for object detection in HSRIs is designed using a suitable ROI scale of object detection. The object features obtained using a CNN have good universality and robustness. Finally, a CNN framework with a suitable ROI scale of object detection is trained and tested. Using the WHU-RSONE data set, the proposed method is compared with the faster region-based CNN (Faster-RCNN) framework. The experimental results show that the proposed method outperforms the Faster-RCNN framework and provides good object detection results in HSRIs.

Change detection is a quite challenging task due to the imbalance between unchanged and changed class. In addition, the traditional difference map generated by log-ratio is subject to the speckle, which will reduce the accuracy. In this letter, an imbalanced learning-based change detection is proposed based on PCA network (PCA-Net), where a supervised PCA-Net is designed to obtain the robust features directly from given multitemporal synthetic aperture radar (SAR) images instead of a difference map. Furthermore, to tackle with the imbalance between changed and unchanged classes, we propose a morphologically supervised learning method, where the knowledge in the pixels near the boundary between two classes is exploited to guide network training. Finally, our proposed PCA-Net can be trained by the data sets with available reference maps and applied to a new data set, which is quite practical in change detection projects. Our proposed method is verified on five sets of multiple temporal SAR images. It is demonstrated from the experiment results that with the knowledge in training samples from the boundary, the learned features benefit change detection and make the proposed method outperform than supervised methods trained by randomly drawing samples.

GaoFen1 is the first high-resolution earth observation satellite built in China, and carries four wide field-of-view (WFV) cameras to achieve large-scale monitoring and mapping. However, the unstable attitude measurement accuracy of the satellite generally imparts low geopositioning accuracy and inconsistent geometric error in overlapping areas of WFV images. A feasible and effective large-scale planar block adjustment (PBA) method is presented that corrects the geometric errors of the vast WFV images integrally, further improving the geometric accuracy of these images. In addition, whether ground control points (GCPs) are needed, and the effect of different numbers of GCPs on PBA accuracy is also investigated. Two key technologies are used in this paper. First, a universal PBA error equation based on the virtual control points is presented to allow PBA with or without GCPs. Second, an adjustment method aided by a digital elevation model is adopted to overcome the weak convergence geometry among WFV images, further ensuring stable estimation of PBA. The effectiveness of the proposed method was verified by 664 WFV images covering most of mainland China. The satisfactory experimental results indicate that the method presented herein is reasonable and effective, but that a certain number of GCPs is needed to ensure the accuracy of large-scale PBA results for WFV images.

Due to the all-weather and all-day advantage and the high geometric accuracy, the spaceborne synthetic aperture radar (SAR) stereo pair has a wide application in digital elevation model (DEM) production and ground control points (GCPs) extraction around the world. The GaoFen3 (GF3) remote sensing satellite is the first C-band multipolarization SAR satellite with a resolution of 1 m in China and plays an important role in commercial exploitation and scientific research. This letter performs a comprehensive analysis of the geometric accuracy for GF3 stereo pair orientation. By analyzing the orientation error sources, it is proven that the affine transformation model in image space can effectively compensate the unmodeled errors in GF3 imagery. Based on the rational polynomial coefficient model, the integrated orientation model for the GF3 stereo pair is presented here. Moreover, the model parameters and weight determination are further studied to analyze the influence of weight matrixes on geometric accuracy. Experimental results demonstrate that the proposed integrated orientation model can be effectively used for the GF3 stereo pair. The GCPs, convergent angle, and weight setting are the key factors influencing the geometric accuracy.

Image segmentation is the foundation of object-based image analysis, and many researchers have sought optimal segmentation results. The initial image oversegmentation and the optimal segmentation scale are two vital factors in high spatial resolution remote sensing image segmentation. With respect to these two issues, a novel image segmentation method combining superpixels with a minimum spanning tree is proposed in this paper. First, the image is oversegmented using a simple linear iterative clustering algorithm to obtain superpixels. Then, the superpixels are clustered by regionalization with a dynamically constrained agglomerative clustering and partitioning (REDCAP) algorithm using the initial number of segments, and the local variance (LV) and the rate of LV change (ROC-LV) indicator diagrams corresponding to the number of segments are obtained. The suitable number of image segments is determined according to the LV and ROC-LV indicator diagrams corresponding to the number of segments. Finally, the superpixels are reclustered using the REDCAP algorithm based on the suitable number of image segments to obtain the image segmentation result. Through two sets of experiments, the proposed method is compared with two other segmentation algorithms. The experimental results show that the proposed method outperforms the others and obtains good image segmentation results.

Patch imaging is an important capability of GaoFen4, which is the first Chinese high-resolution planar array satellite in geosynchronous orbit. When the satellite collects images in the patch-imaging mode, overlapping images can be successively obtained. As these images are captured at different times and from a very high orbit, the initial geometric accuracy of the overlapping area between images is inconsistent, which directly affects image mosaicking. In this paper, a novel bundle block adjustment method based on the rational function model without ground control points (GCPs) is proposed to eliminate the relative geometric error among the images and to generate refined rational polynomial coefficients (RPCs). In this method, the average elevation is used instead of the true elevation of the ground points to solve the problem of weak convergence of corresponding bundles, which would result in unstable calculation in the height direction. Virtual control points (VCPs) generated from the original RPCs are used to restrain the freedom of the entire block in the horizontal direction, thereby ensuring stable calculation without the use of GCPs. A successive RPC regeneration method based on VCPs is also presented. To verify the effectiveness of the proposed method, four experiments were performed using real data to assess the geometric accuracy of the method, and the satisfactory experimental results indicate that the presented method is both practical and effective.

Due to the limitations in the geometric accuracy of reference data and matching accuracy between images from different sensors, the conventional method of calibration may not perform well in terms of accuracy for an agile optical satellite (AOS) with submeter resolution. Due to the high mobility of an AOS, a cross-image pair (CIP) including a push-broom image and a swing scanning image for the same area could be collected. This letter developed a novel calibration method for an AOS, which is characterized using the constraints from the CIP instead of reference images covering a calibration site. In this method, the viewing angle of a charge-coupled device detector fit by two polynomials is introduced into a rigorous imaging model to establish the calibration model, and a bundle adjustment under the constraint of an aided digital surface model (DSM) is applied to solve the unstable calculation stemming from the strong correlation between the parameters of the coupled images. To verify the effectiveness of the method, experiments were conducted using the CIP simulated according to the rigorous imaging process of an AOS. In tests, the method achieved a high theoretical accuracy of better than 0.1 pixels.

The launch of the ZY-3 surveying and mapping satellite (ZSMS) by China has resulted in a significant increase in the volume of image data collected for subsequent processing. In this letter, we present our research on the message passing interface (MPI), open multiprocessing (OpenMP), and compute unified device architecture (CUDA)-based (MOC-based) preprocessing of ZSMS images in a system that consists of multiple central processing units (CPUs) and graphics processing units (GPUs). First, CPUs and GPUs in the system are organized into atomic computing resources (ACRs) by means of an MPI. Then, three cooperative methods are proposed for potential performance improvement of the processors with OpenMP and CUDA. The input/output (I/O) overhead is also addressed in this letter. The experimental results show that the total execution time of the 12 ZSMS nadir images with four ACRs is reduced to 86.10 s, which could provide near-real-time response for the time-critical applications that follow.

This paper analyzes the characteristics of the ground remote sensing satellite data pre-processing system and proposes an approach to encapsulate the processing steps into plug-ins with the idea of separating the satellite data processing business with the underlying algorithms. Then the data processing chain is organized to construct the ground multi-satellite/sensor data pre-processing system. This technology has been used by ground pre-processing systems of China resources satellites (ZY1-02C, ZY3).

The volume of the image data collected by optical satellites is increasing tremendously nowadays. In this paper, we present our recent research on the multi-graphics processing unit (multi-GPU) based near real-time preprocessing and releasing system (RPRS) of optical satellite images. RPRS consists of two sub-systems, namely, the image preprocessing sub-system and the image releasing sub-system. The image preprocessing sub-system ports the four image processors (i.e. relative radiometric correction, median filtering, modulation transfer function compensation and geometric correction) to the GPUs for execution with the compute unified device architecture (CUDA). Both the basic GPU implementation and three optimization measures are presented. To validate our strategy, an entire ZY-3 nadir image strip consisting of 80 images is used for experiment on the multi-GPU system that consists of four NVIDIA Tesla M2050 GPUs. The results show that the efficiency of the processors is substantially accelerated by the use of GPU. The speedup ratios are between 10.94 and 44.89. Moreover, when using all the four GPUs, the processing of the entire image strip only costs 19 min 45 s, which could meet the near realtime preprocessing requirement. The image releasing system affords the direct visualization of the georeferenced images on the geographic base map of GeoGlobe, a Google Earth like 3D spatial information releasing platform developed by State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing, China. It is very convenient for the follow-up relevant applications.

One of the most important tasks in remote sensing data processing is the production of orthorectified images. Such tasks are computationally intensive and can become a bottleneck for remote sensing image processing, particularly in high-throughput environments, such as large satellite imagery processing centers. This letter explores the use of massive parallel processing graphical processing unit (GPU) in a clustered network environment to speed up image processing tasks, such as orthorectification. Our parallelization method is based on inverse sensor model and the stream model for image processing, which allow the flexibility of placing computational units on proper computation units, such as GPU, CPU cores, or nodes in a cluster. In our experiments on images of two satellites, more than 198 times and 50.3 times speedup over one and multiple thread CPU versions have been achieved, respectively.

This letter presents a method of seamline determination based on segmentation for orthoimage mosaicking in an urban area. Image segmentation is used to achieve regions of objects. First, preferred regions through which seamlines are inclined to be passed are determined by spans of segmented regions. Second, pixel-level optimization is carried out using Dijkstra's algorithm based on differential cost to find the optimized seamline. The experimental results on digital aerial orthoimages in the urban area prove that the new method is promising for the seamline determination in mosaicking.

The area Voronoi diagrams with overlap (AVDO) method was recently presented and has been used to generate a seamline network for the mosaicking of orthoimages. The method shows considerable potential advantages for seamless mosaics covering a large geographic region. In this paper, the method is further improved, and a seamline network refinement approach based on AVDO is presented. The improvements of the presented approach include the detection of valid regions for orthoimages, a more general algorithm for the generation of bisectors, and the refinement of the seamline network combining the bottleneck model and the Dijkstra's algorithm. Finally, a number of digital aerial orthoimages are employed to test the proposed approach, and the experimental results demonstrate its efficacy.

The building detection is one of the most challenging issues in remote sensing image processing. In this paper, a novel approach for building detection using corner detection, segmentation and adaptive windowed Hough Transform is presented. In the first, the Mean shift segmentation is used to split the image into a numbers of classes. In the second step, the scale invariant feature transform (SIFT) is used to extract the corners in the original image. In the third step the corners are used as one of the evidences to verify the presence of buildings. In the Mean shift segmentation result image, around the corners detected by SIFT algorithm, the approximate boundary of the buildings is extracted. With the help of approximate boundary of the buildings, the size of the building can be estimate. Finally, in order to extract the precise building roof boundary, the adaptive windowed Hough Transform is used to extract the straight line of the building boundary. Preliminary experimental results indicate that the proposed method produced promising results.

Remote sensing image enhancement occupies a peculiar position in remote sensing image processing and is an important preprocessing step for subsequent analysis. Numerous image enhancement techniques are available for remote sensing image enhancement. In this paper, the color constancy technique is introduced, and a novel color constancy remote sensing images enhancement algorithm is proposed. This algorithm can not only restore more details in the dark area of the image, but also self-adaptive to the luminance conditions. Based on the linear transform, the proposed algorithm contains two parts: (1) the scale parameter is calculated by the adaptive quadratic function with gamma correction to enhance the luminance; (2) the shifting parameter is used to restore the edge details. The experiments are conducted using images downloaded from NASA's website. Experimental results indicated that the proposed algorithm performs much better in preserving the hue and saturation and avoiding color distortion, especially in the dark area.

High precision MTF measurement is the basis of high quality image restoration. Since the presence of noise in images, traditional MTF measurement based on Target image will produce biased result, and the biased result will introduce new noise after image restoration. In this paper, based on analysis of characteristics and limitation of traditional image restoration methods, we propose an image restoration approach based on Kalman filter, this approach firstly uses Gaussian fitting to obtain theoretical value of line spread function, then it uses KALMAN filter to obtain the true value of line spread function from theoretical value and measured value. Experiments on TDI-CCD images show that the approach proposed in this paper make better performance.

Considering band misalignment caused by attitude jittering or other factors, band registration becomes the most critical pre-processing step for multispectral imagery as the registration result will directly influence the following applications. So band registration accuracy evaluation is necessary before registered imagery going through the next processing step. This paper proposes an automatic approach to evaluate the band registration accuracy for multi-spectral imagery. The proposed method is based on the theory of image matching, which includes three main steps: 1) feature points detection, and then 2) corresponding points matching, and 3) accuracy evaluation. Experiments are designed for the validation of the proposed approach with RGB image of Toronto in Canada captured by the Microsoft Vexcel's UltraCam-D (UCD) camera, in which quantitative analyses are applied to assess the accuracy and reliability of the method. And evaluation result for multi-spectral images of satellite, i.e., ZiYuan-3, by this method were presented. The result shows that the proposed method can automatic evaluate band registration accuracy for multi-spectral imagery accurately, efficiently and objectively.

In the distributed cross-organization business process, each organization only knows its private orchestration and partial public choreography, It doesn't have any information about the global process. If one organization has to change, it is difficult to compute the impact imposed on other organizations. In this paper, we propose a method to get the global services dependency matrix automatically through analysing the orchestrations and choreography. Then we introduces the service impact analysis model based dependency matrix and intraservice and inter-service change propagation, through which we can ascertained the impact caused by the changes.

Digital aerial orthoimages have been widely used in surveying, mapping, geographic information systems, visualization, and other applications. However, when producing digital aerial orthoimages, radiometric equalization over large areas is often a most time-consuming and costly process and has become a bottleneck. This letter presents a network-based radiometric equalization approach to eliminate the radiometric differences between images. The network is constructed using the area Voronoi diagrams with overlap and is based on the topological relationship of the constructed network; transferring paths between images are determined, and a global-to-local strategy is used to improve the algorithm, both in its global and local performance. Digital aerial orthoimages from both film-based and digital cameras are used to evaluate the performance of the presented algorithm.

The mosaicking of orthoimages has been used to cover a large geographic region for various applications ranging from environmental monitoring to disaster management. However, existing mosaicking methods mainly focus on the generation of seamlines between two adjacent orthoimages. In this paper, we present a novel approach based on the use of a seamline network formed by a novel area Voronoi diagrams with overlap and the use of effective mosaic polygons (EMPs) to define the pixels of each orthoimage for the final mosaic. The generated seamline network is global based and is also optimized after refinement. It gives an effective partitioning for the regions of all orthoimages to form EMPs. The partitioning is unique, seamless, and has no redundancy. The algorithm is parallel, and the EMP of each orthoimage only has relation to orthoimages which have overlaps with it. It can ensure the flexibility and efficiency of mosaicking, without an intermediate process and independent of the sequence of the image composite. The experimental results obtained from the mosaicking of 40 color orthoimages demonstrate considerable potential for generating a seamline network automatically and effectively. This is extremely useful when a seamless mosaic is required to cover a large geographic region.