Graph representation learning methods are highly effective in handling complex non-Euclidean data by capturing intricate relationships and features within graph structures. However, traditional methods face challenges when dealing with heterogeneous graphs that contain various types of nodes and edges due to the diverse sources and complex nature of the data. Existing heterogeneous graph neural networks (HGNNs) have shown promising results but require prior knowledge of node and edge types and unified node feature formats, which limits their applicability. Recent advancements in graph representation learning using large language models (LLMs) offer new solutions by integrating LLMs' data processing capabilities, enabling the alignment of various graph representations. Nevertheless, these methods often overlook heterogeneous graph data and require extensive preprocessing. To address these limitations, we propose an LLM-enhanced Heterogeneous Graph Neural Network (LHGNN). LHGNN leverages the strengths of both LLM and GNN, allowing for the processing of graph data with any format and type of nodes and edges without the need for type information or special preprocessing. LHGNN employs LLM to automatically summarize and classify different data formats and types, aligns node features, and uses a specialized GNN for targeted learning, thus obtaining effective graph representations for downstream tasks. Theoretical analysis and experimental validation have demonstrated the effectiveness of our method.

The use of large language models (LLMs) as feature enhancers to optimize node representations, which are then used as inputs for graph neural networks (GNNs), has shown significant potential in graph representation learning. However, the fundamental properties of this approach remain underexplored. To address this issue, we propose conducting a more in-depth analysis of this issue based on the interchange intervention method. First, we construct a synthetic graph dataset with controllable causal relationships, enabling precise manipulation of semantic relationships and causal modeling to provide data for analysis. Using this dataset, we conduct interchange interventions to examine the deeper properties of LLM enhancers and GNNs, uncovering their underlying logic and internal mechanisms. Building on the analytical results, we design a plug-and-play optimization module to improve the information transfer between LLM enhancers and GNNs. Experiments across multiple datasets and models validate the proposed module.

Graph Neural Networks (GNNs) excel in homophilic graphs but struggle with heterophilic graphs where connected nodes exhibit divergent characteristics. Existing methods underutilize semantic information in node attributes, a key factor for interpreting heterophilic interactions. To bridge this gap, we propose LLM-Enhanced Graph Neural Network for Heterophily (LEGNNH), a novel framework that synergizes large language model (LLM) with spectral graph convolutions for node classification in heterophilic graphs. LEGNNH operates in three stages: (1) Task-aware semantic embedding using LLMs with instruction-based prompting to encode raw node text; (2) Multi-channel spectral filtering dynamically aggregating local patterns via adaptive low-pass, high-pass, and identity filters; (3) A hierarchical attention module selectively integrates higher-order neighborhood semantics while suppressing noise propagation from label-discordant connections. Addressing the scarcity of heterophilic benchmarks, we contribute two text-attributed graph datasets (YelpNYC and Amazon-Review) with explicit feature-label heterophily. Extensive experiments on seven benchmarks show LEGNNH achieves state-of-the-art performance (average rank 1.86 across 10 baselines), outperforming the best heterophily-specific model by$\mathbf{1. 3 1 - 5. 3 3 \%}$accuracy on key datasets.

Large Language Models (LLMs) have achieved remarkable success across various domains. However, they still face significant challenges, including high computational costs for training and limitations in solving complex reasoning problems. Although existing methods have extended the reasoning capabilities of LLMs through structured paradigms, these approaches often rely on task-specific prompts and predefined reasoning processes, which constrain their flexibility and generalizability. To address these limitations, we propose a novel framework that leverages graph learning to enable more flexible and adaptive reasoning capabilities for LLMs. Specifically, this approach models the reasoning process of a problem as a graph and employs LLM-based graph learning to guide the adaptive generation of each reasoning step. To further enhance the adaptability of the model, we introduce a Graph Neural Network (GNN) module to perform representation learning on the generated reasoning process, enabling real-time adjustments to both the model and the prompt. Experimental results demonstrate that this method significantly improves reasoning performance across multiple tasks without requiring additional training or task-specific prompt design. Code can be found in https://github.com/zch65458525/L2T.

Modeling the structure and events of the physical world constitutes a fundamental objective of neural networks. Graph Network Simulators (GNS) have become the leading approach due to their computational efficiency and accuracy. However, the scarcity of comprehensive datasets limits their full potential, especially for multibody dynamics, crucial in real-world applications. In response to this, a high-quality physical simulation dataset has been constructed, encompassing 1D, 2D, and 3D scenes, along with more trajectories and time-steps compared to existing datasets. Furthermore, this work distinguishes itself by developing eight complete scenes, significantly enhancing the dataset’s comprehensiveness. A key feature of our dataset is the inclusion of precise multibody dynamics, facilitating a more realistic simulation of the physical world. Using this high-quality dataset, a systematic evaluation of various existing GNS methods has been conducted.

Modeling the 3D dynamics of relational systems is a key challenge in the field of natural sciences, especially in molecular simulations and particle mechanics. However, existing graph neural network methods overlook two critical practical issues: (1) Error accumulation in multi-step reasoning: They are designed for single-step predictions, which leads to severe error accumulation during multi-step inference. (2) Overly strict equivariance constraints: In scientific and engineering applications, dynamics often exhibits symmetry breaking or relaxation due to boundary conditions. In this work, we propose Group Point Multi-Step Dynamics - Graph Neural Networks (GPMS-GNN), which consists of two main modules: (1) the neural operator, which approximates dynamics as a time-evolving function and enables multi-step prediction in a single inference; (2) the relaxed equivariant graph neural network, which relaxes continuous equivariant inductive biases into discrete forms to accommodate scenarios involving symmetry breaking. Extensive experiments on particle simulations, human motion capture, and molecular dynamics demonstrate that GPMS outperforms state-of-the-art methods in both accuracy and data efficiency.

Diffusion Models are a potent class of generative models capable of producing high-quality images. However, they often inadvertently favor certain data attributes, undermining the diversity of generated images. This issue is starkly apparent in skewed datasets like CelebA, where the initial dataset disproportionately favors females over males by 57.9%, this bias amplified in generated data where female representation outstrips males by 148%. In response, we propose a plug-and-play method named Manifold Guidance Sampling, which is also the first unsupervised method to mitigate bias issue in DDPMs. Leveraging the inherent structure of the data manifold, this method steers the sampling process towards a more uniform distribution, effectively dispersing the clustering of biased data. Without the need for modifying the existing model or additional training, it significantly mitigates data bias and enhances the quality and unbiasedness of the generated images.

Filter pruning is a widely adopted technique for compressing convolutional neural networks (CNNs). However, common filter pruning algorithms often overlook varying importance and feature distributions across different layers, leading to suboptimal results. Additionally, they frequently require intricate fine-tuning processes and lack fine-grained control over intragroup coefficients, potentially failing to fully exploit correlations among individual features within a group. To overcome these limitations, we introduce a sparse penalty based on Sparse Group Lasso, encouraging the model to engage in more stringent feature selection within groups, thus flexibly adapting to intra-group sparsity. We propose an adaptive sparse weights scheme, dynamically allocating sparsity rates for each layer in the network based on evaluations of feature distribution and importance. Upon completion of training, the algorithm directly eliminates redundant parameters, eliminating the need for a fine-tuning process. The efficacy of the proposed methodology has been substantiated through both theoretical derivations and experimental validations by the research team.

In this paper, we introduce an Feature Aggregate Side Network (FASN), a simple, efficient, and easy-to-train method for open-vocabulary semantic segmentation. Building upon existing models based on the CLIP-Side Network framework, we address the issue of CLIP-generated features lacking pixel-level recognition capability by layering a novel fast graph representation learning layer between the CLIP and side networks. This integration introduces an inductive bias for the aggregation of local features, thereby better addressing the challenges in semantic segmentation. Through validation on five distinct datasets and extensive ablation studies, we have demonstrated the effectiveness of our modifications. Our findings indicate that with a slight increase in the number of parameters, there is a significant enhancement in performance.

While generative adversarial networks have made significant strides in natural image synthesis, their performance in specialized remote sensing (RS) imagery, particularly in capturing fine details of small objects like airplanes and ships, needs enhancement. This deficiency frequently results in shape distortion within the generated images. This challenge, compounded by the prohibitive costs of annotating RS images, motivates the development of an efficient unsupervised approach. In response, this paper introduces Object-Aided Generative Adversarial Network (OAGAN), an innovative model for unsupervised RS image generation. Initially, it employs an object-centric learning mechanism to extract structural semantic maps of foreground objects, without the need for labels. Subsequently, this paper proposes the Shape Encoding Layer (SEL) to encode the structural semantics of objects, which is seamlessly integrated into the intermediate layers of the generative model. This integration enables the model to prioritize the structural information of foreground objects. Additionally, to enhance the diversity of generated images, this paper designs a novel style regularization term. Comprehensive experiments are conducted on three distinct RS image datasets. Experiment results demonstrate that the proposed method surpasses state-of-the-art models in terms of the quality of generated images.

Node classification is a crucial area in graph representation learning, with significant applications in real-world network scenarios. However, due to the complexity of the relationships among nodes in the topological graph, precise data labeling is often challenging, leading to a significant amount of label noise. Partial Label Learning (PLL) is a weakly supervised learning problem designed to accommodate label noise. Therefore, we introduce PLL to address the issue of label noise. Currently, Graph Neural Networks (GNNs) are the primary method for addressing node classification problems. However, existing research has demonstrated that GNNs tend to amplify the similarity between node features, posing challenges for label disambiguation in partial label scenarios. To address this issue, We conduct an analysis of the composition of node features and propose a novel method, which aims to enhance feature quality and reduce node feature similarity in partial label scenarios of node classification. Extensive experiments on challenging homogeneous graph datasets indicate that PLNR achieves state-of-the-art performance and demonstrates comparable results to fully supervised learning.

Convolutional Neural Networks (CNNs) have exhibited remarkable success in various vision tasks. The allocation of channels within each layer significantly influences CNN performance. Despite the acknowledged importance of channel configurations, achieving an optimal distribution that balances computational efficiency and model accuracy remains challenging. In this paper, we initially transform the CNN training problem into an analogous knapsack optimization problem, incorporating the loss sensitivity criteria. Subsequently, we introduce the concept of value density, employing greedy algorithms, to accurately quantify the improvement in accuracy achievable by increasing the unit FLOPs on every layer. This fosters a proficient exploration and optimization of channel configurations within convolutional layers. Additionally, the efficacy of the proposed methodology has been substantiated through both theoretical derivations and experimental validations by the research team, outperforming popular pruning methods under equal-scale FLOPs conditions. We hope that the integration of value density into channel search algorithms will contribute to the development of more powerful CNNs.

Accurately simulating physical processes is an extremely challenging task, but the rapid development of machine learning and the availability of large datasets have made Graph Neural Networks (GNNs) a powerful tool for effectively simulating physical systems. Currently, GNNs-based methods are primarily used in simple scenarios such as the free fall and collision of objects, fluid flow, and gravitational interactions among atoms. However, in complex industrial environments, there are always intricate interference factors such as friction, bearing connections, and torque affecting the motion between objects. Consequently, GNNs-based methods largely fail to solve practical physical problems related to complex multi-body dynamics. In this paper, to address the current lack of multi-body dynamics datasets in this field, we first introduce a multi-body dynamics dataset comprising eight different scenarios, each embodying distinct physical principles. Furthermore, we explore Graph Neural Simulators (GNSs) structure and physical priors and propose an efficient novel model, the Graph Neural Network with Restrgints (GOAT), that can directly learn the relationships between systems from multi-body trajectories, thereby enhancing performance. Our results have shown significant improvements compared to other state-of-the-art baselines, demonstrating strong generalization capabilities and data efficiency.

Generative adversarial networks (GANs) have shown notable accomplishments in remote sensing (RS) domain. However, this article reveals that their performance on RS images falls short when compared to their impressive results with natural images. This study identifies a previously overlooked issue: GANs exhibit a heightened susceptibility to overfitting on RS images. To address this challenge, this article analyzes the characteristics of RS images and proposes manifold constraint regularization (MCR), a novel approach that tackles overfitting of GANs on RS images for the first time. Our method includes a new measure for evaluating the structure of the data manifold. Leveraging this measure, we propose the MCR term, which not only alleviates the overfitting problem, but also promotes alignment between the generated and real data manifolds, leading to enhanced quality in the generated images. The effectiveness and versatility of this method have been corroborated through extensive validation on various RS datasets and GAN models. The proposed method not only enhances the quality of the generated images, reflected in a 3.13% improvement in Fréchet inception distance (FID) score, but also boosts the performance of the GANs on downstream tasks, evidenced by a 3.76% increase in classification accuracy. The source code is available athttps://github.com/rootSue/Manifold-RSGAN.

Recent studies have shown that machine models do not coincide with the human perception of image quality, making mainstream image enhancement methods not always compatible with downstream tasks. To ameliorate this issue, this paper targets homography estimation, which is a fundamental step in image interpretation, to explore a hybrid-degraded image enhancement approach. Specifically, we build a reinforcement learning image processing framework with multiple lightweight tools, and develop a two-policy agent to progressively handle hybrid-degraded images. To match homography estimators with different properties, we further propose an environmental epistemic model (EEM) to build task-specific prior knowledge of uncertain environments. During the training process, the EEM is updated online and used to guide the agent’s exploration and exploitation. Comprehensive experiments are conducted on the aerial dataset where images are degraded from four visual perspectives: brightness, contrast, sharpness and stimulus. Results show that our agent can be trained to generate high-quality images for both learning-based and traditional homography estimators.

By utilizing previously offline data, offline reinforcement learning (offline RL) can develop effective policies for the environment with complex online interaction. However, due to the incomplete coverage of offline datasets, the estimation errors of Q-functions caused by out-of-distribution (OOD) actions may cause current off-policy methods to fail. Offline RL uses policy constraints, value function regularization, or uncertainty estimation to drive the learned policy from resembling behavioral policy. Unfortunately, the policy constraints approach restricts the learned policy to a region near the sub-optimal behavioral policy. In addition, the value function regularization approach does not accurately assess OOD actions, which can cause it to be too conservative in estimating the Q-value of actions within the proximity distribution. Finally, the uncertainty estimation is biased due to the complex environment or inaccurate valuation early in training. We suggest Density-UCR as a solution to the aforementioned issues. Density-UCR makes the Q-function estimate have a lower-confidence bound (LCB) and penalizes the OOD actions by using the estimation error of the ensemble Q-functions as a penalty value. Additionally, Density-UCR models the offline data's distribution using a density estimator to derive more accurate uncertainty weights for the penalty value. Density-UCR employs uncertainty estimates as the weight of the priority replay buffer to increase the stability of online fine-tuning and prevent performance degradation caused by the distribution shift of offline samples over online samples. Our experiments on the D4RL benchmark show that Density-UCR significantly outperforms the policy constraints approach with the value function regularization approach. Furthermore, Density-UCR also offers excellent fine-tuning performance.

Deep reinforcement learning (DRL) provides a new way to generate robot control policy. However, the process of training control policy requires lengthy exploration, resulting in a low sample efficiency of reinforcement learning (RL) in real-world tasks. Both imitation learning (IL) and learning from demonstrations (LfD) improve the training process by using expert demonstrations, but imperfect expert demonstrations can mislead policy improvement. Offline to Online reinforcement learning requires a lot of offline data to initialize the policy, and distribution shift can easily lead to performance degradation during online fine-tuning. To solve the above problems, we propose a learning from demonstrations method named Accelerating Self-Imitation Learning from Demonstrations (A-SILfD), which treats expert demonstrations as the agent's successful experiences and uses experiences to constrain policy improvement. Furthermore, we prevent performance degradation due to large estimation errors in the Q-function by the ensemble Q-functions. Our experiments show that A-SILfD can significantly improve sample efficiency using a small number of different quality expert demonstrations. In four Mujoco continuous control tasks, A-SILfD can significantly outperform baseline methods after 150,000 steps of online training and is not misled by imperfect expert demonstrations during training. In addition, our ablation experiments demonstrate the effectiveness of each part of the method.

Abstract Compared with natural images, geospatial images cover larger area and have more complex image contents. There are few algorithms for generating controllable geospatial images, and their results are of low quality. In response to this problem, this paper proposes Geospatial Style Generative Adversarial Network to generate controllable and high‐quality geospatial images. Current conditional generators suffer the mode collapse problem in geospatial field. The problem is addressed via a modified mode seeking regularization term with contrastive learning theory. Besides, the discriminator network architecture is modified to process global feature information and texture information of geospatial images. Feature loss in the generator is introduced to stabilize the training process and improve generated image quality. Comprehensive experiments are conducted on UC Merced Land Use Dataset, NWPU‐RESISC45 Dataset, and AID Dataset to evaluate all compared methods. Experiment results show our method outperforms state‐of‐the‐art models. Our method not only generates high‐quality and controllable geospatial images, but also enhances the discriminator to learn better representations.

Nowadays, robots have gradually replaced humans to perform tasks in challenging environments, but visually satisfactory images may not always contain high-quality features required by downstream algorithms. In this paper, we reveal that in the homography estimation, which is the essential task of robot vision, algorithms show different sensitivities to aerial images and sometimes perform better in over/under-exposed scenes. To this point, we utilize the gamma correction theory to design an interpretable exposure adjustment function, and train a personalized exposure selection network ESH-Net by contrastive learning. After adjusting the exposure of original input by our method, algorithms can obtain better homography results. Comprehensive experiments are conducted on samples generated from the aerial DOTA dataset. Results show that our method can produce high-quality images for both learning-based and traditional homography estimators, and outperforms other exposure enhancement networks.

Recently, although advanced satellite can collect massive image data, the efficiency of information extraction is still lacking due to low-quality and redundancy of data, and limited transmission bandwidth worsens this situation. As the result, researchers start to focus on software-defined satellite. Through directly deploying image processing applications on a space-based computing platform, satellite's capabilities can be flexibly changed, expanded or enhanced. Under such a trend, this paper proposes a global motion estimation system which attempts to reduce data redundancy in satellite videos. Our work consists of two stages: first, adaptively feature matching and optical flow are combined to estimate global motion. Following this, we give the transform formula between homography matrices in different resolution images, thus algorithm can be faster executed at a smaller search space. The proposed system runs on the Android platform, which is easy to be programmed, ported and updated. It is validated on the newest software-defined satellite TianZhi-1, and shows outstanding results under real experimental environment.

Near real-time moving object detection in Wide Area Motion Imagery (WAMI) can support applications in many fields. However, since the targets’ search space are extremely large, current state-of-the-art methods usually suffers high computational cost. It’s crucial to recommend candidate regions for detection first. Different from most research that extracts candidate blobs as proposals, this paper attempts to offer high-quality patches, which can provide background info and be taken as many algorithms’ input (convolutional neural network, optical flow, block matching, etc.). First by the idea that local errors are tolerable, neighborhood frame differencing with local computation is applied to roughly obtain irregular blobs. After that, an adaptive spatial clustering algorithm which utilizes grid and density reachable, is proposed to generate multi-size motion patches quickly. Compared with traditional clustering, advantages of this algorithm include parameter-free, saving of time and widely applications. Experimental results show that the proposed method is competitive especially in dense traffic regions.

Automatic detection on orbit is an efficient way to filter useless data downloaded to the ground. However, detection on orbit is a challenging task due to limited computational resources on the satellite. In this paper, a context-aware and depthwise-based detection framework for remote sensing images is proposed which can be used on orbit. In the result of limited computational resources on the satellite, on-orbit object detection should detect with low memory cost and fast speed while ensuring the accuracy. To address the problem of small model in the process of feature extracting, a depthwise convolution is applied instead of typical convolution. In this light, a small deep neural network is built to run on orbit, using Single Shot Multibox Detector (SSD) as basic detection module. Motivated by its weak performance on remote sensing image owing to few pixel about target object, context information about target object is added to improve performance. To further investigate the context information influence, we add a balance factor to balance the context information and background noise it brings. Then an experiment on real remote sensing image dataset is conducted comparing our extended model with other current state-of-the-art detection models. Results show our extended model outperforms other models in accuracy and speed. Deploying the pretrained model on the Android Platform with only 60M memory cost confirms the feasibility to detect on orbit. This detection system is to be verified on the TZ-1 satellite which will be launched in the year of 2018.

In some missions of NanoSats, we find images captured are disturbed by motion blur which caused under the situation that NanoSats work in low-earth orbit at high speeds. In this paper, we address the problem of deblurring images degraded due to space-based imaging system shaking or movements of observing targets. We propose a motion deblurring strategy via using Generative Adversarial Networks(GAN) to realize an end-to-end image processing without kernel estimation in orbit. We combine Wasserstein GAN(WGAN) and loss function based on adversarial loss and perceptual loss to optimize the result of deblurred image. The experimental results on the two different datasets prove the feasibility and effectiveness of the proposed strategy which outperforms the state-of-the-art blind deblurring algorithms using for remote sensing images both quantitatively and qualitatively.

Data acquisition from satellite is a challenging task due to the limitation of ground station resource and data transmission capacity. Considering that most of the raw data downloaded to the ground are useless, it is worthy to directly get the results by automatic detection on orbit and only transfer the images that include the target objects, which can filter the useless data efficiently. On orbit automatic detection, satellite computing resources need to be considered, so a smaller and faster model needs to be built. Though enormous object detection methods have been proposed and several application have emerged, a detailed survey on different models about detection accuracy and detection speed as well as memory cost is still lacking. This paper aims to provide a survey on the recent object detection researches and make a strategy to detect on orbit. To further compare the performance among different methods, we conduct an experiment in the same real dataset and compare them from accuracy, speed and memory cost. Following the experiment result, a feasible strategy of object detection for the TZ-1 satellite on-orbit which has a low memory dependency, fast speed and comparable accuracy adapt to its computing resources is proposed.

Blind image deblurring is an important topic which is widely used in many research fields such as photography, optics, astronomy, medical images, monitoring, military and so on. Although many algorithms have been proposed to improve the deblurring result in the past years, most of them cannot perform perfectly in some challenging cases. This study presents a novel blind deblurring method based on an adaptive weighted total variation (TV) algorithm. The blur kernel estimation is based on the image structure, the sparsity and continuity prior of point spread function is also taken into account. To get better effect of removing the ringing artefacts, adaptive weight calculated according to the property of the higher‐order partial derivatives in the local image is proposed in TV algorithm to alleviate the ill‐posed inverse problem and stabilise the solution for latent image restoration. The experimental results prove that the proposed algorithm can suppress the ringing artefacts to a great extent in the latent image, and can get much better effect in both vision and theoretical results than traditional algorithms.