Hyperspectral images (HSIs) are vital for scene analysis, as they capture detailed spatial and spectral information to characterize surface materials. However, accurate HSI classification is challenged by significant intra-class spectral variability and spatial complexity. To address this, we leverage the fact that pixels of the same class typically form irregular local regions. We propose a structure-adaptive graph transformer (SAGT) that dynamically captures irregular spatial topologies and homogeneous spectral information to achieve adaptive HSI representation and precise classification. Specifically, a structure-aware self-attention (SASA) module is developed to embed graph structures into the self-attention mechanism as a robust positional indicator, which can be extended easily and effectively. SASA comprehensively accounts for the spatial structures and spectral autocorrelation of ground objects, facilitating the aggregation of homogeneous spectral information for noise-robust spectral representations. Additionally, a structure-adaptive pooling (SAP) module is designed to dynamically adjust graph structures by discarding irrelevant edges, thus better indicating spatial relationships. By coupling the SASA and SAP modules, our proposed SAGT model significantly alleviates spectral variability and tolerates prior noise. Furthermore, data augmentation techniques of random discard and random offset are built, which randomly drop and shift graph nodes to generate more diverse samples during preprocessing. In postprocessing, multiview decision-making integrates results from multiple contextual views to provide more robust predictions. Experimental results on three benchmark datasets consistently demonstrate that SAGT is more effective and reliable than other state-of-the-art methods. To facilitate reproduction, we will release the source code for SAGT at https://github.com/ShuGuoJ/SAGT.git.

Unmanned aerial vehicle (UAV) hyperspectral imaging offers an efficient and cost-effective way to map tree species at the individual tree levels. Conventional methods mostly rely on large samples of natural RGB images of tree crowns, lacking the ability to distinguish species, particularly for trees with overlapping crowns. This study proposed a novel scale pyramid graph network (SPGN) for instance segmentation that can simultaneously apply pixel-level (node) classification for discriminating species and edge prediction for delineating individual trees. Based on a graph-in-graph (GiG) convolution, we built a scale pyramid module (SPM) that extracts multiscale features at pixels, superpixels, and subgraph levels to aggregate the over-segmented superpixels into the same species and the same tree. We also proposed an innovative concept of subgraph positional encoding (SPE) to represent the natural spatial relationship of graph-structured data. The SPGN method was evaluated in a case study involving eleven subtropical broadleaf species under an urban environment in south China. The accuracy of species classification achieved 93%, and the area under the curve (AUC) of individual tree segmentation reached 0.96. Compared with state-of-the-art methods such as DeepForest, Detectree2, and segment anything model (SAM), SPGN presented fewer errors in tree detection and outperformed in instances of crown overlaps. Ablation studies proved the effectiveness of SPM and SPE modules, which improved segmentation by 10% and classification by 7% in accuracy, respectively. The findings confirm the benefits of incorporating spatial context, such as crown textures and tree positional relationships, for species differentiation; in return, accurate species identification combined with spectral information assists the individual tree segmentation. This effective strategy can be potentially extended to a broader range of regions and forest types.

With the rapid development of autonomous driving and SLAM technology, the perception system of a vehicle heavily relies on laser and image sensors to capture the real-world scenario and avoid obstacles autonomously. To achieve accurate and robust multi-sensor fusion computation, high-precision extrinsic calibration of camera and laser scanner is a necessary requirement. Traditional multi-sensor calibration methods based on manual features rely on specific scenarios and may not provide feature information over long distances. In this paper, we present a novel approach for robustly calibrating the extrinsic parameters of a solid-state(SS) lidar-camera system in a natural environment. Our proposed method begins with obtaining robust line feature information. we first innovatively employ a super-voxel clustering method to extract global 3D line features from the complete point cloud and then back-project these 3D line features into 2D space. Afterward, a transformer-based edge detection network, EDTER, is used to detect the edge features and estimate the probability pixel-by-pixel. To consider the uncertainty of two-dimensional line features and the inconsistency of residuals at different distances, we construct a line feature weight model for line feature residual calculation. Finally, we minimize the residual errors using least squares optimization to recover the relative pose of the camera and the lidar sensor. We conducted a performance study to compare our proposed method against existing targetless calibration methods on various natural scenarios. The experimental results demonstrate that our proposed method achieves higher robustness, accuracy, and consistency, making it suitable for real-world applications.

When taking a picture, any camera shake during the shutter time can result in a blurred image. Recovering a sharp image from the one blurred by camera shake is a challenging yet important problem. Most existing deep learning methods use supervised learning to train a deep neural network (DNN) on a dataset of many pairs of blurred/latent images. In contrast, this paper presents a dataset-free deep learning method for removing uniform and non-uniform blur effects from images of static scenes. Our method involves a DNN-based re-parametrization of the latent image, and we propose a Monte Carlo Expectation Maximization (MCEM) approach to train the DNN without requiring any latent images. The Monte Carlo simulation is implemented via Langevin dynamics. Experiments showed that the proposed method outperforms existing methods significantly in removing motion blur from images of static scenes.

Abstract. Deep learning has proved to be a powerful tool for solving inverse problems in imaging, and most of the related work is based on supervised learning. In many applications, collecting truth images is a challenging and costly task, and the prerequisite of having a training dataset of truth images limits its applicability. This paper proposes a self-supervised deep learning method for solving inverse imaging problems that does not require any training samples. The proposed approach is built on a reparametrization of latent images using a convolutional neural network, and the reconstruction is motivated by approximating the minimum mean square error estimate of the latent image using a Langevin dynamics–based Monte Carlo (MC) method. To efficiently sample the network weights in the context of image reconstruction, we propose a Langevin MC scheme called Adam-LD, inspired by the well-known optimizer in deep learning, Adam. The proposed method is applied to solve linear and nonlinear inverse problems, specifically, sparse-view computed tomography image reconstruction and phase retrieval. Our experiments demonstrate that the proposed method outperforms existing unsupervised or self-supervised solutions in terms of reconstruction quality.

With the rapid development of deep convolutional neural networks (CNNs), super-resolution (SR) in hyperspectral image (HSI) has achieved good results. Current methods generally use 2-D convolution for feature extraction, but they cannot effectively extract spectral information. Although 3-D convolution can better characterize feature structure of HSI, it will lead to parameter redundancy, model complexity, and severe memory shortage. To address the above problems, we propose a new HSI SR method, named diffused CNN (DCNN). Specifically, spectral convolutions have been added into the enhanced convolutional neural (ECN) block, and a series of spectral convolutions are introduced in the residual network to learn features in the channel direction of different depths. Furthermore, histogram of oriented gradient (HOG) and local binary pattern (LBP) are used to retain the shape and texture information of the image, respectively, which can well represent the spatial structure of the object. To effectively make use of the extracted shallow and deep features, a feature fusion strategy is used to reinforce the reconstruction efficiency. Besides, an image enhancement module has been developed to diffuse the SR image into the image space. Extensive evaluations and comparisons show that our DCNN approach can not only recover the HSI data with richer details but also achieve superiority over several state-of-the-art methods.

While supervised deep learning has been a prominent tool for solving many image restoration problems, there is an increasing interest on studying self-supervised or un-supervised methods to address the challenges and costs of collecting truth images. Based on the neuralization of a Bayesian estimator of the problem, this paper presents a self-supervised deep learning approach to general image restoration problems. The key ingredient of the neuralized estimator is an adaptive stochastic gradient Langevin dy-namics algorithm for efficiently sampling the posterior distri-bution of network weights. The proposed method is applied on two image restoration problems: compressed sensing and phase retrieval. The experiments on these applications showed that the proposed method not only outperformed existing non-learning and unsupervised solutions in terms of image restoration quality, but also is more computationally efficient.

Blind deconvolution is about estimating both the convolution kernel and the latent signal from their convolution. Many blind deconvolution problems have a short-and-sparse (SaS) structure; i.e., the signal (or its gradient) is sparse and the kernel size is much smaller than the signal size. While $\ell_1$-norm relating regularizations have been widely used for solving SaS blind deconvolution problems, the so-called region/edge selection technique brings great empirical improvement to such $\ell_1$-norm relating regularizations in image deblurring. The essence of region/edge selection is during an alternative iterative scheme of SaS blind deconvolution: one estimates the kernel on an estimate of the latent image with well-separated image edges instead of the one with the least fitting error. In this paper, we first examine the validity and soundness of $\ell_1$-norm relating regularization in the setting of 1D SaS blind deconvolution. The analysis reveals the importance of the separation of nonzero signal entries toward the soundness of such a regularization. The studies laid out the foundation of region selection technique; i.e., during the iteration, an estimate of the latent image with well-separated edges is a better candidate for estimating the kernel than the one with the least fitting error. Based on the studies conducted in this paper, an alternating iterative scheme with region selection model is developed for SaS blind deconvolution, which is then applied to blind motion deblurring. The experiments show its effectiveness over many existing $\ell_1$-norm relating approaches.

To explore the airspace structure and safety performance of unmanned aerial vehicle (UAV) system based on spatial digital twins (DTs), the study introduces DTs technology, and combines convolutional neural network (CNN) algorithm with UAV autonomous network. The DTs system of UAV is constructed by using wireless communication technology, and its security performance is simulated. The results show that in the analysis of the system packet loss rate, it is found that with the increase of the acquisition points, the amount of transmitted data only increases slightly, but the packet loss rate does not change significantly. In the analysis of the network performance of the unmanned aircraft system, it is found that the node energy-based weighted clustering algorithm (EWCA) can be used to increase the life of the overall network and enhance its availability by rationally controlling the number of nodes and the number of switching between clusters. As the number of nodes increases, the minimum survival time of each clustering algorithm decreases linearly. When the number of nodes is less than 600, the growth rate of cluster head is higher; When the number of nodes is more than 600, the curve growth is relatively smooth. In the analysis of the probability of network safety interruption, it is found that using the model constructed, when the energy acquisition coefficient is close to 0.5, the energy conversion efficiency is higher, the signal-to-noise ratio is larger. Also, when the number of intermediate nodes is increased to 10, the UAV has the best network safety performance. Therefore, through the research, it is found that the UAV DTs system constructed can significantly improve the safety performance of the UAV during its airspace flight. It can provide experimental references for the widespread application of the UAV in the later period.

Transmembrane proteins (TMPs) serve as drug targets for more than half of the drugs currently available in the market. However, it had not been clearly explained how they realize their drug effects through multiple complex molecules bindings actions. Research into TMPs bindings and corresponding structural basis will provide key information for drug research and new drug development. In this study, we defined the binding domain of TMPs according to the binding region investigation of multiple conjugate types. A 3D deep learning model was architected to discover the structural universality inside those domains. The experimental results proved such binding domains existing on the surface of TMPs, and they are structural specific distinguishing to the surface regions without any binding activities. This work provides a new theoretical basis for TMPs binding research and can greatly boost the development of the drug industry.

With the widely deployed wireless access points (APs) and the worldwide popularization of smartphones, WiFi-based indoor positioning has attracted great attention to both industry and academia. Locating and tracking objects within an indoor environment plays an important role in Internet of Things application and service. However, it is a challenging problem to achieve high accuracy using WiFi positioning technique due to the high instability in received signal strength from AP. Thus, it is desirable to select APs by considering both signal strength and connection quality. In this article, an AP selection algorithm based on multiobjective optimization is proposed to improve indoor WiFi positioning accuracy. The self-adaptive AP selection algorithm can be easily applied to various real scenarios and the performance of the new method is considerably better than classical algorithms. Learning algorithm is exploited to obtain the optimal solution for the self-adaptive AP selection algorithm. Experiments are conducted and the proposed algorithm is compared with classical algorithms. The experimental results demonstrate that the performance of the self-adaptive AP selection algorithm is at least a few decimeters better than classical algorithms in terms of RMSE of position estimation. Meanwhile, the new method is robust to the random generation of initial particles and normalizing factor as their effect on the positional accuracy is less than 1 decimeter.

With the fast development of consumer-level RGB-D cameras, real-world indoor three-dimensional (3D) scene modeling and robotic applications are gaining more attention. However, indoor 3D scene modeling is still challenging because the structure of interior objects may be complex and the RGB-D data acquired by consumer-level sensors may have poor quality. There is a lot of research in this area. In this survey, we provide an overview of recent advances in indoor scene modeling methods, public indoor datasets and libraries which can facilitate experiments and evaluations, and some typical applications using RGB-D devices including indoor localization and emergency evacuation.

In this paper, we extend the previously proposed Tomo-PSInSAR method to detect homogeneous objects in the urban environment. Tomo-PSInSAR integrates conventional persistent scatterer (PS) interferometry and multidimensional SAR tomography to monitor complex built environments [1]. It can jointly detect single and overlaid PSs by constructing a two-tier hierarchical network. Robust estimators (M-estimator and ridge estimator) are introduced to improve the robustness of estimation. To monitor the semi-artificial regions (e.g, pavements and small grassed lands) that are normally distributed scatterers (DSs) in SAR images [2], we analyze homogeneous pixels on the basis of Tomo-PSInSAR. Before estimating the geophysical parameters, we perform a two-sample Anderson-Darling test for the identification of statistically homogeneous pixels at the stage of interferometry. In the first-tier network, the most reliable PSs are identified and they will be used as reference points in the second-tier network. In the second-tier network, the geophysical parameters (e.g., height, deformation velocity) of overlaid PSs are estimated using tomographic imaging [3] and the geophysical parameters of DSs are estimated using the Capon-Beamforming algorithm [4]. The removal of atmospheric delay in the second-tier network is accomplished by subtracting the phase of adjacent PSs that are detected in the first-tier network. In this sense, the proposed integrated method as shown in Fig. 1 can jointly monitor single PSs, overlaid PSs, and DSs according to specific cases. TerraSAR-X/TanDEM-X images are used to validate this method. The results are shown in Fig. 2-4.

This paper takes Shenzhen Fustian comprehensive transportation junction as the case, and makes use of continuous multiple real-time dynamic traffic information to carry out monitoring and analysis on spatial and temporal distribution of passenger flow under different means of transportation and service capacity of junction from multi-dimensional space-time perspectives such as different period and special period. Virtual reality geographic information system is employed to present the forecasting result.