Event cameras capture visual information as asynchronous pixel change streams, excelling in challenging lighting and high-dynamic scenarios. Existing multimodal large language models (MLLMs) concentrate on natural RGB images, failing in scenarios where event data fits better. In this paper, we introduce EventGPT, the first MLLM for event stream understanding, pioneering the integration of large language models (LLMs) with event-based vision. To bridge the huge domain gap, we propose a three-stage optimization paradigm to progressively equip a pre-trained LLM with event understanding. Our EventGPT consists of an event encoder, a spatio-temporal aggregator, a linear projector, an event-language adapter, and an LLM. Firstly, GPT-generated RGB image-text pairs warm up the linear projector, following LLaVA, as the gap between natural images and language is smaller. Secondly, we construct N-ImageNet-Chat, a large synthetic dataset of event data and corresponding texts to enable the use of the spatio-temporal aggregator and to train the event-language adapter, thereby aligning event features more closely with the language space. Finally, we gather an instruction dataset, EventChat, which contains extensive real-world data to fine-tune the entire model, further enhancing its generalization ability. We construct a comprehensive benchmark, and experiments show that EventGPT surpasses previous state-of-the-art MLLMs in generation quality, descriptive accuracy, and reasoning capability. Code: EventGPT

Speckle noise significantly affects synthetic aperture radar (SAR) imaging systems, causing difficulties in the postprocessing of SAR images. To strike a balance between denoising performance and detail retention, this study proposes an innovative deep learning despeckling network based on feature supervision. The network aims to enhance the noise reduction capability by applying regularized supervision to shallow features within the network. With this enhancement, a robust feature extraction module (RFEM) helps further improve the feature-capture capability by decomposing the image into different frequency components. Subsequently, by integrating complementary prior information captured from different network architectures, we introduced a deep cross-denoising module (DCDM) to enrich the feature details. Furthermore, the incorporation of an attention mechanism enables the network to concentrate more on the information-rich regions of interest, thereby further refining and enhancing the detailed features of the image. The experimental results demonstrate the superiority of the proposed network in effective SAR image despeckling while preserving the image details.

Randomized stepped frequency radar (RSFR) is suitable for handling tasks in complex electromagnetic environments. Due to the fact that the target typically occupies a series of range cells in the high-resolution range profiles synthesized by RSFR, it is referred to as an extended target. However, current joint reconstruction methods based on sparsity theory are still limited by the length of the extended target, resulting in block mismatch and high computational complexity. How to adaptively determine the size of the extended target remains a challenge. In this paper, a novel deep unfolding network is proposed for reconstruction of extended target with block-sparse property, called Sparse Bayesian Learning (SBL)-Damped Generalized Approximate Message Passing (DGAMP)-Net. The network proposed in the paper can learn block sparsity information from data without block partition information. Particularly, in each layer of SBL-DGAMP-Net, we replace the M-step of pattern-coupled sparse Bayesian learning (PC-SBL) with a convolutional neural network, thus overcoming the fragility of PC-SBL parameter selection. The E-step of the network is an unfolding of DGAMP, with the damping factor optimized by deep learning. Furthermore, the architecture of SBL-DGAMP-Net can accept measurement matrices as inputs to the network, thus avoiding the need for retraining. The simulation results indicate that this method exhibits superior reconstruction accuracy and computational efficiency compared to existing high-resolution range-velocity joint reconstruction algorithms for RSFR.

Synthetic aperture radar (SAR) generally suffers from enormous strains from large quantities of sampling data and serious interferences from the speckle noise. This letter proposes a novel deep network to address these problems. By utilizing the prior knowledge in a more reasonable way, the proposed network could realize SAR imaging and despeckling with down-sampled data simultaneously. Specifically, we decompose the SAR image in the SAR imaging-despeckling observation model into a sparse matrix and a low-rank matrix, and then establish an optimization problem with the corresponding sparse and low-rank priors. Moreover, the deep convolutional neural networks (CNN) denoiser prior is also introduced to further improve the speckle reduction capability. Then, we devise a deep network called SLRCP-Net to solve this problem. Experiments conducted on real Radarsat-1 down-sampled data demonstrate the validity of SLRCP-Net in SAR imaging and speckle suppression.

Large quantities of sampling data and speckle noise are two serious problems existing in synthetic aperture radar (SAR). The former puts enormous strain on data measurement, transmission, and storage. The latter deteriorates imaging quality, disturbing the subsequent processing in SAR systems. This study proposes a real-time deep network to address these issues. The proposed network is able to concurrently achieve SAR imaging and despeckling with down-sampled data. Specifically, to fit more diverse imaging regions, we suppose that the noise in the SAR imaging-despeckling observation model follows a universal complex generalized Gaussian distribution. Based on this assumption, an optimization problem with a convex$L_{q}$-norm ($q >$1) fidelity term is constructed through the maximum$a$posteriori(MAP) estimation. We employ an$L_{1}$-norm sparse constraint and a convolutional neural networks (CNNs) projection-based detail preservation constraint to further promote the imaging and speckle reduction capabilities. Then, the complex-valued split Bregman method (CV-SBM) is applied to convert the proposed problem into an equivalent sequence of sub-problems. We devise a computationally efficient solution for the fidelity term-related sub-problem, due to the specific down-sampled strategy in SAR. A substitutive cost function and a CNN structure are introduced to solve the projection-related sub-problem. Finally, the iterative steps of CV-SBM are cast into a deep network-dubbed$L_{q}$-split Bregman (SPB)-Net to yield a desirable imaging and despeckling result within a small number of iterations. Numerical experiments based on the real Radarsat-1 data validate the efficiency and feasibility of the proposed$L_{q}$-SPB-Net in real-time imaging and despeckling with down-sampled data.

Synthetic aperture radar (SAR) typically faces both large-scale data and speckle noise problems, which, respectively, induce enormous strains on transmission and storage and interfere with the analysis and interpretation of SAR images. To tackle these, we present a real-time processing deep network, called SPB-Net, to implement imaging and speckle suppression simultaneously. First, a novel imaging-despeckling observation model with the nonlogarithmic additive speckle noise is established. Subsequently, guided by the statistical properties of noise and sparse and detail-preserved requirements in SAR imaging and despeckling, we formulate an$L_{2}$along with two$L_{1}$regularizations as the fidelity, sparse, and image detail-preserved constraints, respectively. Convolution layers are employed to improve the feature representation capability in the latter$L_{1}$term as well. Based on this, we construct a corresponding convex optimization problem. Then, the complex-valued split Bregman method, focusing on the complex-variable convex problem, is unfolded into a parameter-learnable and architecture-fixed SPB-Net to solve the proposed problem effectively and efficiently. Experimental results with the downsampled Radarsat-1 raw data demonstrate the validity in imaging and speckle suppression and the real-time processing capability of the proposed SPB-Net.

The visualization of synthetic aperture radar (SAR) images plays a critical role in remote sensing applications. To effectively obtain the image suitable for human observation, this paper introduces a new SAR image visualization algorithm to map the high dynamic range SAR amplitude values to low dynamic range displays via reflectivity distortion preserved entropy maximization. Its designed objective is to present the maximal amount of information content in the displayed image, and being optimal in an information theoretical sense, as well as restricting the upper bound of the reflection distortion caused by tone mapping. The resulting optimization problem can be graph theoretically modeled as a K-edges maximum weight path problem in a directed acyclic graph, and it can be solved efficiently by dynamic programming in real time. Empirical evidences are provided to demonstrate the superior visual quality obtained by our new visualization technique.

Graphic processing units (GPUs) is widely used to accelerate the processing speed of the radar detection procedure, including the range compression, coherent integration and constant false alarm rate. Specifically, detailed parallel design of the radar algorithm and the thread programming are shown. The experimental results show that, by engaging the parallel technology into the radar processing procedure, much high speedup ratio can be obtained. Furthermore, precise target detection can be guaranteed.

As a new enchanting theory, compressed sensing (CS) demonstrates that a sparse signal can be recovered through a surprisingly small number of linear measurements by solving a problem of ℓ 1 norm minimisation (which can be thought as a special case of the signomial diversity measures). However, the traditional CS model with ℓ 1 norm minimisation can not fully exploit the sparsity especially when the degree of sparsity increases or the measurements number reduces. In this study, the Cauchy diversity measures is incorporated into the proposed model to deal with the above difficulties. The simulation results demonstrate that under the same condition, this new model offers a superior reconstruction precision compared with the common used signomial diversity measures.