Sentinel-2 data is highly valuable in remote sensing applications owing to its open accessibility and comprehensive spatial-temporal coverage. However, it poses a unique challenge due to its varying spatial resolutions across its different spectral bands (ranging from 10-m to 60-m). High-resolution data offers finer details and significantly enhances the accuracy of analyses, benefiting a wide range of fields. The majority of current methods for enhancing Sentinel-2 image resolution do not address the enhancement of all bands through a unified network. To address this issue, we propose a novel deep learning-based solution named SenGLEAN, for enhancing multi-resolution bands (specifically, 10-m and 20-m Ground Sampling Distance - GSD) to a unified 5-m GSD. SenGLEAN leverages the concept of Generative LatEnt bANks (GLEAN) and employs a multi-resolution encoder-bank-decoder architecture to achieve high-resolution remote sensing imagery. Notably, our model incorporates channel-attention (CA) and pixel-attention modules (PA) within its design to enhance the spatial quality of results. Through quantitative comparison, we demonstrate that our network shows significant improvements, by increasing the PSNR by 0.28 dB for 10-m bands and 2.92 dB for 20-m bands while reducing the RMSE by 3.11 for 10-m bands and 52.26 for 20-m bands. Furthermore, we introduce a lightweight variant, LightSenGLEAN, retaining critical components while reducing total parameters by 81.89%, which still offers competitive performance. In summary, our proposed model provides an efficient solution to enhance both 10-m and 20-m Sentinel-2 bands to 5-m resolution using a single deep learning framework, facilitating precise image analysis and geoscience applications.

In recent years, deep learning methods have been widely applied to hyperspectral image (HSI) classification. Besides convolutional neural network (CNN)-based deep learning, deep random forest (RF)-based method, such as densely connected deep RF (DCDRF), was also developed for HSI classification which utilized the spectral–spatial information to improve the classification accuracy. In DCDRF, evenly distributed image patches with a fixed patch size are utilized to extract the spatial information of ground objects. However, the spatial information in each patch is not always correct, especially when the patch center is close to the edge of ground objects. In this letter, we propose a new classification method called spectral–spatial deep RF (SSDRF) which can fully utilize the spatial information existing in HSIs to further improve the classification accuracy. The joint region that combines both the fixed-size patch and shape-adaptive superpixel is proposed to exploit more accurate spatial information. The RF used in the classification model is replaced by extremely random forest (EF) to avoid overfitting. Moreover, the majority voting is conducted within superpixels and among different scales of superpixels to optimize the classification. The experimental results on three HSIs demonstrate that the proposed SSDRF can achieve satisfactory classification results and outperforms patched-based DCDRF.

The rapid development of remote sensing sensors has made it possible to collect airborne hyperspectral data with high spectral and spatial resolution. Such data can provide valuable information to identify tree species in the forest. However, it is a challenge to efficiently utilize the abundant spectral information and complex spatial information within the data. In this article, a Spectral-Spatial and Cascaded Multilayer Random Forests (SSCMRF) method is proposed to classify tree species in the high spatial resolution hyperspectral image. The SSCMRF adopts two classification stages to fully exploit the spatial information within shape-adaptive superpixels and shape-fixed patches. Two different kinds of spatial information are integrated by concatenating the output of the superpixel-based classification and the spectral features as the input of the patch-based classification. To demonstrate the superiority of the proposed SSCMRF, experiments are conducted with an airborne hyperspectral data set of a forest area with the spatial resolution of 1 m. Training with 2.5% randomly selected ground truth samples, the proposed SSCMRF achieves a classification accuracy of 97.50% within 6 minutes. In addition, the experiment results demonstrate that the proposed SSCMRF outperforms some state-of-art spectral-spatial classification models in terms of quantitative metrics and visual quality on the classification map.

Semantic segmentation is one of the most important tasks in remote sensing. However, as spatial resolution increases, distinguishing the homogeneity of each land class and the heterogeneity between different land classes are challenging. The Markov random field model (MRF) is a widely used method for semantic segmentation due to its effective spatial context description. To improve segmentation accuracy, some MRF-based methods extract more image information by constructing the probability graph with pixel or object granularity units, and some other methods interpret the image from different semantic perspectives by building multilayer semantic classes. However, these MRF-based methods fail to capture the relationship between different granularity features extracted from the image and hierarchical semantic classes that need to be interpreted. In this article, a new MRF-based method is proposed to incorporate the multigranularity information and the multilayer semantic classes together for semantic segmentation of remote sensing images. The proposed method develops a framework that builds a hybrid probability graph on both pixel and object granularities and defines a multiclass-layer label field with hierarchical semantic over the hybrid probability graph. A generative alternating granularity inference is suggested to provide the result by iteratively passing and updating information between different granularities and hierarchical semantics. The proposed method is tested on texture images, different remote sensing images obtained by the SPOT5, Gaofen-2, GeoEye, and aerial sensors, and Pavia University hyperspectral image. Experiments demonstrate that the proposed method shows a better segmentation performance than other state-of-the-art methods.

Currently, most individual tree crowns delineation methods are proposed for aerial images and Lidar data. In this paper, a scheme is proposed to delineate tree crowns from the high spatial resolution WorldView-3 satellite imagery. We select a multiscale fitting scheme for the delineation and propose two modifications to optimize the result generated from this scheme. First, the object-based classification is applied to detect the shadows between tree crowns. Second, we modified the multiscale fitting process to make the selection of optimal scale fairer. The experiment results of a subset in a WorldView-3 satellite image demonstrate that the proposed modifications can improve the delineation accuracy.

Vehicle detection is a significant and challenging task in aerial remote sensing applications. Most existing methods detect vehicles with regular rectangle boxes and fail to offer the orientation of vehicles. However, the orientation information is crucial for several practical applications, such as the trajectory and motion estimation of vehicles. In this paper, we propose a novel deep network, called a rotatable region-based residual network (R3-Net), to detect multioriented vehicles in aerial images and videos. More specially, R3-Net is utilized to generate rotatable rectangular target boxes in a half coordinate system. First, we use a rotatable region proposal network (R-RPN) to generate rotatable region of interests (R-RoIs) from feature maps produced by a deep convolutional neural network. Here, a proposed batch averaging rotatable anchor strategy is applied to initialize the shape of vehicle candidates. Next, we propose a rotatable detection network (R-DN) for the final classification and regression of the R-RoIs. In R-DN, a novel rotatable position-sensitive pooling is designed to keep the position and orientation information simultaneously while downsampling the feature maps of R-RoIs. In our model, R-RPN and R-DN can be trained jointly. We test our network on two open vehicle detection image data sets, namely, DLR 3K Munich Data set and VEDAI Data set, demonstrating the high precision and robustness of our method. In addition, further experiments on aerial videos show the good generalization capability of the proposed method and its potential for vehicle tracking in aerial videos. The demo video is available athttps://youtu.be/xCYD-tYudN0.

The synergistic use of spatial features with spectral properties of satellite images enhances thematic land cover information. This study aims to address the lack of high-level features by proposing a classification framework based on convolutional neural network (CNN) to learn deep spatial features for wetland. In particular, a CNN model was used for classification of remote sensing imagery with limited number of training data by fine-tuning of a preexisting CNN (AlexNet). The classification results obtained by the deep CNN were compared with those based on well-known ensemble classifiers, namely Random Forest (RF), to evaluate the efficiency of CNN Experimental results demonstrated that CNN was superior to RF for complex wetland mapping even by incorporating the small number of input features (i.e., 3 features) for CNN compared to RF. The proposed classification scheme serves as a baseline framework to facilitate further scientific research using the latest state-of-art machine learning tools for processing remote sensing data.

The Markov random field (MRF) model has attracted great attention in the field of image segmentation. However, most MRF-based methods fail to resolve segmentation misclassification problems for high spatial resolution remote sensing images due to insufficiently using the hierarchical semantic information. In order to solve such a problem, this paper proposes an object-based MRF model with auxiliary label fields that can capture more macro and detailed information and apply it to the semantic segmentation of high spatial resolution remote sensing images. Specifically, apart from the label field, two auxiliary label fields are first introduced into the proposed model for interpreting remote sensing images from different perspectives, which are implemented by setting a different number of auxiliary classes. Then, the multilevel logistic model is used to describe the interactions within each label field, and a conditional probability distribution is developed to model the interactions between label fields. A net context structure is established among them to model the interactions of classes within and between label fields. A principled probabilistic inference is suggested to solve the proposed model by iteratively renewing the label field and auxiliary label fields, in which different information of auxiliary label fields can be integrated into the label field during iterations. Experiments on different remote sensing images demonstrate that our model produces more accurate segmentation than the state-of-the-art MRF-based methods. If some prior information is added, the proposed model can produce accurate results even in complex areas.

Multi-view satellite images can form multiple stereo image pairs (MSIPs). Each pair allows generating two types of surface information: surface disparity maps (SDMs) and digital elevation models (DEMs). Both types are important for 3D assisted information extraction applications. Since multi-view images can be used to generate more accurate stereo-based DEMs than using one stereo-pair, it is more computationally cost-effective to generate enhanced SDMs from MSIPs. However, the disparity values from different stereo-pairs are unequal. Hence, redundant data cannot be generated to enhance the SDMs. Therefore, our research focuses on the generation of inter-proportional disparity maps from multi-view very high resolution satellite images. The method uses the sensor model information of the stereo images to construct multiple epipolar images in the object-space. After implementation, the disparity values of the constructed images were found to be proportional. This property allows the generation of SDMs that may replace the DEMs to some extent.

The Markov random field (MRF) model serves as one of the most powerful tools to improve the robustness of fuzzy c-means (FCM) clustering. However, the use of a pixel-level MRF makes the clustering deficient to deal with images with macro texture patterns. In order to overcome such a problem, this letter presents a novel method that segments images by combining FCM with the region-level MRF (RMRF) model. In this method, a fuzzy novel energy function is established for the RMRF model and utilized in the process of fuzzy clustering, which plays an important role in describing large-range variations of macro textures. Considering the complexity of image textures, a region-level mean template is also established to enhance the relationships between neighboring regions in terms of spectral and structural information. Experiments are conducted using high-resolution remote sensing images, which demonstrate that the proposed method can improve the segmentation accuracy compared with four state-of-the-art competitors.

The synthesis of low-resolution panchromatic (Pan) image is a critical step of ratio enhancement (RE) and component substitution (CS) pansharpening methods. The two types of methods assume a linear relation between Pan and multispectral (MS) images. However, due to the nonlinear spectral response of satellite sensors, the qualified low-resolution Pan image cannot be well approximated by a weighted summation of MS bands. Therefore, in some local areas, significant gray value difference exists between a synthetic Pan image and a high-resolution Pan image. To tackle this problem, the pixels of Pan and MS images are divided into several classes by$k$-means algorithm, and then multiple regression is used to calculate summation weights on each group of pixels. Experimental results demonstrate that the proposed technique can provide significant improvements on reducing color distortion.

Fuzzy c-means (FCM) clustering with spatial constraints has attracted great attention in the field of image segmentation. However, most of the popular techniques fail to resolve misclassification problems due to the inaccuracy of their spatial models. This paper presents a new unsupervised FCM-based image segmentation method by paying closer attention to the selection of local information. In this method, region-level local information is incorporated into the fuzzy clustering procedure to adaptively control the range and strength of interactive pixels. First, a novel dissimilarity function is established by combining region-based and pixel-based distance functions together, in order to enhance the relationship between pixels which have similar local characteristics. Second, a novel prior probability function is developed by integrating the differences between neighboring regions into the mean template of the fuzzy membership function, which adaptively selects local spatial constraints by a tradeoff weight depending upon whether a pixel belongs to a homogeneous region or not. Through incorporating region-based information into the spatial constraints, the proposed method strengthens the interactions between pixels within the same region and prevents over smoothing across region boundaries. Experimental results over synthetic noise images, natural color images, and synthetic aperture radar images show that the proposed method achieves more accurate segmentation results, compared with five state-of-the-art image segmentation methods.

An improved mean shift segmentation method featuring adaptive parameter selection is presented in this paper. We associate the bandwidths and weight for each point in a spatial-range feature space with boundary information in an image plane. Varying weight and bandwidth for each pixel are assigned according to a boundary map, which is obtained by learning multiple edge cues. We consider two groups of edge cues and two regressing modules, approach the cue combination as a supervised learning problem from the ground truth data (manually sketched boundary maps). From our preliminary results, the provided method can combine the top-down information got from regression models with the mean shift process and constrain over-clustering of pixels belonging different land objects.

Automatic detection of buildings out of urban objects is not a straightforward task due to the existing spectral and textural similarities. The problem gets even worse in buildings with pitched roofs. Pitched roof buildings receive dissimilar amount of solar radiation on their different faces causing different brightness values for a single roof. Thus, in object based classification methods, each side will probably be assigned to different segments preventing proper building boundary detection. In this study, in order to detect the proper building boundaries through image segmentation, IHS (Intensity, Hue, and Saturation) color system is used. Then, to detect buildings out of the segmented image, elevation information extracted from stereo satellite imagery is benefited. The presented method was tested on GeoEye stereo imagery and 92% of the image buildings were detected precisely.

Integration or registration of optical images with elevation data is important for many applications such as building detection. Despite the fact that VHR (very high resolution) satellite images are acquired mostly off-nadir, the use of such images in the previous research works is less common than the nadir ones. This is due to the high relief distortion of above-ground objects, such as buildings, in these off-nadir images. For the applications of information extraction, which prefer the integration of the optical images and elevation data, this relief distortion makes the integration between the perspective off-nadir VHR optical images and orthographic digital surface models (DSMs) almost impossible unless a true orthorectification process is implemented. However, true orthoimages are expensive, time consuming and mostly difficult to achieve. This paper proposes an integration method of the DSMs elevations with off-nadir VHR optical imagery by projecting the ground elevations back to the image space using the relevant sensor model. Elevation data is derived using off-nadir VHR stereo images which one of them is then used for the integration. The proposed method is found to be efficient in terms of sub-pixel accuracy and ease of implementation. Additionally, it preserves the original image information which is essential for some applications such as image classification.

Due to the difference of “mean information” between substitution component and substituted component, spectral distortion often occurs in component substitution (CS) pansharpening. In this paper, a data fitting scheme is adopted to improve spectral quality in image fusion based on well-established CS approach. A generalized CS framework that is capable of modeling any CS image fusion method is also presented. In this framework, instead of injecting detail information of panchromatic (Pan) image into substituted component, the data fitting strategy is designed to adjust the mean information of Pan image in the construction of substitution component. The data fitting scheme involves two matrix subtractions and one matrix convolution. It is fast in implementation and is effective to avoid the spectral distortion problem. Experimental results on a large number of Pan and multispectral images show that the improved CS methods have good performance on the spatial and spectral fidelity. Moreover, experiments carried out on large-size images also show an excellent running time performance of the proposed methods.

High resolution panchromatic (Pan) and multispectral (MS) images from modern satellites have been increasingly used in remote sensing applications. Many pan-sharpening algorithms have been adopted by commercial remote sensing and GIS software packages. However, discrepant pan-sharpening results have been reported by different papers using different fusion techniques and images. There is a lack of common understanding of the performance of the techniques. The paper provides an overview and the results of a comprehensive test and evaluation of commercial techniques for pan-sharpening IKONOS, QuickBird, GeoEye-1, and WorldView-2 images.

A rational polynomial camera (RPC) model is a kind of generic sensor model that can be used in different remote sensing systems to model the relationship between object space and image space and transform image data to conform to a map projection. Unlike traditional physical camera models, an RPC model has many coefficients (a total of 80), and these coefficients do not have a physical interpretation. This represents a difficult challenge for the mapping community. For RPC refinement, many solutions, including direct and indirect methods, have been developed. One of them, the recent developed generic method has been shown to be a robust method. Because the generic method can simulate the camera's exterior parameters, it can be used in any geometric situation. Even so, the performance of bundle adjustment with the generic method is still unknown. In this paper, through experiments with a stereo pair and a stereo triplet, the capability of high-accuracy geopositioning based on the generic method is demonstrated. We first give a brief review of previous bundle adjustment methods based on RPC. Then, the bundle adjustment algorithm based on the generic method is introduced in detail. We finally present the experiments with both IKONOS and QuickBird imageries. The experiments show that the bundle adjustment based on the generic method can reach subpixel accuracy in image space and submeter accuracy in object space.

Interest-point matching is a key technique for image registration. It is widely used for 3-D shape reconstruction, change detection, medical image processing, computerized visioning systems, and pattern recognition. Although numerous algorithms have been developed for different applications, processing local distortion inherent in images that are captured from different viewpoints remains problematic. High-resolution satellite images are normally acquired at widely spaced intervals and typically contain local distortion due to ground relief variation. Interest-point-matching algorithms can be grouped into two broad categories: area based and feature based. Although each type has its own particular advantages in specific applications, they all face the common problem of dealing with ambiguity in smooth (low-texture) areas, such as grass, water, highway surfaces, building roofs, etc. In this paper, a new algorithm for interest-point matching of high-resolution satellite images is proposed. The conceptual basis of this algorithm is the detection of “super points,” those points which have the greatest interest strength (i.e., which represent the most prominent features) and the subsequent construction of a control network. Sufficient spatial information is then available to reduce the ambiguity and avoid false matches. We commence this paper with a brief review of current research on interest-point matching. We then introduce the proposed algorithm in detail and describe experiments with three sets of high-resolution satellite images. The experiment results show that the proposed algorithm can successfully process local distortion in high-resolution satellite images and can avoid ambiguity in matching the smooth areas. It is simple, fast, and accurate.

Image registration is the process of geometrically aligning one image to another image of the same scene taken from different viewpoints or by different sensors. High resolution remote sensing images have made it more convenient for people to study the earth; however, they also create challenges for traditional research methods. In terms of image registration, there are a number of problems with using current image registration techniques for high resolution images. This study proposes a new image registration technique, which is based on the combination of feature-based matching (FBM) and area-based matching (ABM). A wavelet-based feature extraction technique, normalized cross-correlation matching and relaxation-based image matching techniques are employed in this new method. Two pairs of data sets, panchromatic images of IKONOS and a panchromatic image of IKONOS with a multispectral image of Quickbird, are used to evaluate the proposed image registration algorithm. The experiment results show that the proposed algorithm can select enough control points to reduce the local distortions caused by terrain relief.