Federated learning is a decentralized framework that enables client participation in collaborative learning without centralized data collection. However, the framework is susceptible to suboptimal model convergence induced by heterogeneity among the client datasets. These discrepancies, including label imbalance, dissimilarity in data distributions, and uneven data volumes between clients, may cause disagreements among local client updates, affecting the ability of the global model to converge effectively during aggregation. We suggest that one potential solution to this problem lies in weighting the model aggregation by client importance and client-to-client relationships. Based on this idea, we propose HyperGenFL (HG-FL), a hypernetwork that generates aggregation weights from learnable client embeddings without requiring any training or benchmarking data. HG-FL utilizes the attention mechanism to capture inter-client relationships based on learnable client-specific embeddings in order to generate model aggregation weights dynamically during federated learning. By guiding the aggregation process with these learnable relationships between local models, HG-FL reduces update conflicts and improves global model performance. We assess HG-FL under various data-heterogeneous environments based on different benchmark datasets including Fashion-MNIST, CIFAR10, CIFAR100 and Tiny-ImageNet. Experimental results demonstrate that HG-FL can achieve superior performance over a range of existing baseline methods under challenging cases with various heterogeneous environments, large models and a large number of clients.

Federated learning is a decentralized approach to training machine learning models while preserving data privacy. To accommodate data heterogeneity among clients, a longstanding issue in Federated Learning, many Personalized Federated Learning (PFL) strategies decompose each client model into global modules, which are collaboratively learned by all clients and the server, and local modules, which are only trained locally on private data. While these strategies require every client to participate in training, in reality, many client devices lack sufficient data or computing resources to perform meaningful local training, making it difficult to achieve personalization for every client. In this paper, we present HyperFLoRA, a PFL framework that leverages knowledge learned from training-capable clients to enable the immediate creation of personalized models for training-incapable or new clients. HyperFLoRA uses adapters for personalization to minimize communication costs and client training workload while employing a trainable hypernetwork to generate personalized adapter weights for each client using minimal client statistical information. From experiments conducted on both convolutional and Transformer neural networks, HyperFLoRA can achieve superior model personalization performance for new clients that did not participate in training than conventional PFL methods, while significantly reducing training-related communication costs and client workload.

Unmanned aerial vehicle (UAV)-borne hyperspectral imagery has been applied in precision agriculture, owing to its high spatial and spectral resolution. Specifically, the high spatial resolution is conducive to revealing the textural characteristics of crops, while the high spectral resolution can depict detailed spectral differences among crops. In this study, we explored the potential of extended attribute profiles (EAP) in modeling the spectral-spatial characteristics of UAV-borne hyperspectral imagery for the precise crop classification. Specifically, two dimensionality reduction approaches, namely, principle component analysis (PCA) and independent component analysis (ICA), were performed on the hyperspectral image to extract components, based on which a series of EAP that measure different image characteristics are generated. To exploit the complementary information of different attributes, the extracted EAP were fused for the classification of crops using feature stacking (FS) and decision fusion (DF) strategies. Meanwhile, random forest (RF), support vector machine (SVM), and deep neural networks (DNN) were used as classifier for the precise classification of crops. Experiments conducted on the WHU-Hi dataset demonstrated that EAP exploited the spectral-spatial information of UAV-borne hyperspectral imagery and obtained satisfactory crop classification performance.

This letter proposes a novel active learning (AL) framework that utilizes the information derived from multiscale superpixel maps for the classification of hyperspectral image. Considering that the nearby pixels with similar spectral properties tend to belong to the same class, we introduce the multiscale superpixel maps for the automatic labeling of the selected informative samples. Moreover, to exploit the multiscale characteristics of objects in the image, a hierarchical fusion approach is developed to integrate the spatial information provided by the superpixel maps into the classification result. To illustrate the effectiveness of the proposed AL framework, experiments on a series of hyperspectral images are conducted and analyzed. The results confirm the superiority of the proposed method compared to the other algorithms.

Query feeds recommendation is a new recommended paradigm in mobile search applications, where a stream of queries need to be recommended to improve user engagement. It requires a great quantity of attractive queries for recommendation. A conventional solution is to retrieve queries from a collection of past queries recorded in user search logs. However, these queries usually have poor readability and limited coverage of article content, and are thus not suitable for the query feeds recommendation scenario. Furthermore, to deploy the generated queries for recommendation, human validation, which is costly in practice, is required to filter unsuitable queries. In this paper, we propose TitIE, a query mining system to generate valuable queries using the titles of documents. We employ both an extractive text generator and an abstractive text generator to generate queries from titles. To improve the acceptance rate during human validation, we further propose a model-based scoring strategy to pre-select the queries that are more likely to be accepted during human validation. Finally, we propose a novel dual learning approach to jointly learn the generation model and the selection model by making full use of the unlabeled corpora under a semi-supervised scheme, thereby simultaneously improving the performance of both models. Results from both offline and online evaluations demonstrate the superiority of our approach.

Ammonia nitrogen (NH4-N) can cause water eutrophication and is the main oxygen-consuming pollutant in water bodies. Remote sensing methods are more macroscopic than traditional measurement methods. However, due to the weak optical characteristics of NH4-N, traditional remote sensing data cannot meet the needs of NH4-N monitoring. In response to this problem, this paper attempts to use unmanned aerial vehicles (UAV) hyperspectral imagery combined with extreme gradient boosting (XGBoost)regression algorithm to quantitatively retrieve NH4-N in urban rivers. The results show that compared with the traditional empirical semi-empirical model, the accuracy of using the XGBoost algorithm to estimate the NH4-N in the water body is significantly improved, and is consistent with the field measurement.

UAV hyperspectral imagery has been widely applied in the fine classification of crops because of its high spectral resolution and high spatial resolution. As the crops in hyperspectral image show complicated characteristics, only the spectral information is insufficient to distinguish them. Therefore, we use multiple feature fusion method for fine classification of crops in UAV hyperspectral imagery. In our work, the GLCM texture, morphological profile, and endmember abundance feature, are extracted. Meanwhile, three fusion strategies, namely decision fusion, probability fusion, and stacking fusion, are employed to obtain the classification results. The experimental results illustrate the superiority of the multiple fusion approaches in the crop fine classification with hyperspectral imagery.

Object detection in aerial images is an active yet challenging task in computer vision because of the bird’s-eye view perspective, the highly complex backgrounds, and the variant appearances of objects. Especially when detecting densely packed objects in aerial images, methods relying on horizontal proposals for common object detection often introduce mismatches between the Region of Interests (RoIs) and objects. This leads to the common misalignment between the final object classification confidence and localization accuracy. In this paper, we propose a RoI Transformer to address these problems. The core idea of RoI Transformer is to apply spatial transformations on RoIs and learn the transformation parameters under the supervision of oriented bounding box (OBB) annotations. RoI Transformer is with lightweight and can be easily embedded into detectors for oriented object detection. Simply apply the RoI Transformer to light head RCNN has achieved state-of-the-art performances on two common and challenging aerial datasets, i.e., DOTA and HRSC2016, with a neglectable reduction to detection speed. Our RoI Transformer exceeds the deformable Position Sensitive RoI pooling when oriented bounding-box annotations are available. Extensive experiments have also validated the flexibility and effectiveness of our RoI Transformer.

Aerial image scene classification is a fundamental problem for understanding high-resolution remote sensing images and has become an active research task in the field of remote sensing due to its important role in a wide range of applications. However, the limitations of existing datasets for scene classification, such as the small scale and low-diversity, severely hamper the potential usage of the new generation deep convolutional neural networks (CNNs). Although huge efforts have been made in building large-scale datasets very recently, e.g., the Aerial Image Dataset (AID) which contains 10,000 image samples, they are still far from sufficient to fully train a high-capacity deep CNN model. To this end, we present a larger-scale dataset in this paper, named as AID++, for aerial scene classification based on the AID dataset. The proposed AID++ consists of more than 400,000 image samples that are semi-automatically annotated by using the existing the geographical data. We evaluate several prevalent CNN models on the proposed dataset, and the results show that our dataset can be used as a promising benchmark for scene classification.

Many significant applications need land cover information of remote sensing images that are acquired from different areas and times, such as change detection and disaster monitoring. However, it is difficult to find a generic land cover classification scheme for different remote sensing images due to the spectral shift caused by diverse acquisition condition. In this paper, we develop a novel land cover classification method that can deal with large-scale data captured from widely distributed areas and different times. Additionally, we establish a large-scale land cover classification dataset consisting of 150 Gaofen-2 imageries as data support for model training and performance evaluation. Our experiments achieve outstanding classification accuracy compared with traditional methods.

Notwithstanding aerial image retrieval is an important and obligatory task, existing retrieval systems lose their efficiency when there is no available aerial image used as the exemplar query. In this paper, we take free-hand sketches into consideration and address the problem of sketch-based aerial image retrieval. This is an extremely challenging task due to the complex surface structures and huge variations of resolutions of aerial images, and few works have been devoted to it. For the first time to our knowledge, we propose a framework to bridge the gap between sketches and aerial images. Specifically, an aerial sketch-image dataset is first collected. Sketches and aerial images are augmented to varied levels of details and used to train a multi-scale deep hierarchical model. The fully-connected layers of the deep model are used as cross-domain features, and the similarity between aerial images and sketches is measured by the Euclidean distance. Experiments on several public aerial image datasets demonstrate the efficiency and superiority of the proposed method.

Prior information about classes plays an important role in the high-resolution image classification. Produced by volunteers with GPS tracking practice and local knowledge, the crowdsourced OpenStreetMap (OSM) data have shown potential as a time-saving and cost-effective way to provide prior information for image classification. In this letter, we develop a high-resolution remote-sensing image classification method using OSM information. OSM objects of classes of interest except roads are extracted to construct the training set for classification. To decrease the misleading errors and redundancy in OSM, a series of approaches is employed successively to refine the training set. Furthermore, OSM road information is directly superimposed on the learned classification result owing to its good quality and completeness. The main contributions of this letter are: 1) the refinement of OSM-derived training samples and 2) the utilization of OSM road superimposition strategy. The high-resolution GF-2 image over the Guangzhou peri-urban area as well as the corresponding OSM data is employed in the experiments. The results illustrate the effectiveness of the proposed method.

The spatial information has been proved to be effective in improving the performance of spectral-based classification. However, it is difficult to describe different image scenes by using monofeature owing to complexity of the geospatial scenes. In this letter, a novel framework is developed to combine the multiple spectral and spatial features based on the Markov random field (MRF). Specifically, the pixels in an image are separated into reliable and unreliable ones according to the decision of multifeature classifications. The labels of the reliable pixels can be conveniently determined, but the unreliable pixels are then classified by fusing the multifeature classification results and reducing the classification uncertainties based on the MRF optimization. Experiments are conducted on three multispectral high-resolution images to verify the effectiveness of the proposed method. Several state-of-the-art multifeature classification methods are also achieved for the purpose of comparison. Moreover, three classifiers (i.e., multinomial logistic regression, support vector machines, and random forest) are used to test the performance of the proposed framework. It is shown that the proposed method can effectively integrate multiple features, yield promising results, and outperform other approaches compared.

In this paper, we proposed a relearning method for classification post-processing (CPP). CPP can be viewed as a label refinement method to improve the classification accuracy. The proposed approach considers the frequency and spatial arrangements of the labels to enhance the classification performance by iteratively learning the classification map. Experiments conducted on a series of images obtained by different sensors show that, the proposed relearning approach present promising performances compared to the state-of-the-art techniques such as filtering, Markov random field (MRF) and object-based voting.

In this paper, a multi-index learning (MIL) approach is proposed to represent and classify the complex urban scenes using a set of low-dimension information indices instead of the traditional high-dimensional spatial features. Specifically, two categories of indices are proposed: 1) Primitive indices (PI), involving a series of basic urban primitives, e.g., buildings, shadow, vegetation; and 2) Variation indices (VI), describing the spectral and spatial variation of the urban scenes. Experiments conducted on a large-scale image (260 km2) captured by the ZY3 satellite (the first Chinese civilian high-resolution satellite) show that the proposed MIL approach can provide promising accuracies even though the complicated urban landscape is represented via low-dimensional feature space. The satisfactory results achieved by the MIL can be attributed to the low-dimensional but high-level semantic information considered.

This paper develops several new strategies for remote sensing image classification postprocessing (CPP) and conducts a systematic study in this area. CPP is defined as a refinement of the labeling in a classified image in order to enhance its original classification accuracy. The current mainstream classification methods (preprocessing) extract additional spatial features in order to complement spectral information and enhance classification using spectral responses alone. On the other hand, however, the CPP methods, providing a new solution to improve classification accuracy by refining the initial result, have not received sufficient attention. They have potential for achieving comparable accuracy to the preprocessing methods but in a more direct and succinct way. In this paper, we consider four groups of CPP strategies: filtering; random field; object-based voting; and relearning. In addition to the state-of-the-art CPP algorithms, we also propose a series of new ones, e.g., anisotropic probability diffusion and primitive cooccurrence matrix. In experiments, a number of multisource remote sensing data sets are used for evaluation of the considered CPP algorithms. It is shown that all the CPP strategies are capable of providing more accurate results than the raw classification. Among them, the relearning approaches achieve the best results. In addition, our relearning algorithms are compared with the state-of-the-art spectral-spatial classification. The results obtained further verify the effectiveness of CPP in different remote sensing applications.