Modeling complex rigid motion across large spatiotemporal spans remains an unresolved challenge in dynamic reconstruction. Existing paradigms are mainly confined to short-term, small-scale deformation and offer limited consideration for physical consistency. This study proposes PMGS, focusing on reconstructing Projectile Motion via 3D Gaussian Splatting. The workflow comprises two stages: 1) Target Modeling: achieving object-centralized reconstruction through dynamic scene decomposition and an improved point density control; 2) Motion Recovery: restoring full motion sequences by learning per-frame SE(3) poses. We introduce an acceleration consistency constraint to bridge Newtonian mechanics and pose estimation, and design a dynamic simulated annealing strategy that adaptively schedules learning rates based on motion states. Futhermore, we devise a Kalman fusion scheme to optimize error accumulation from multi-source observations to mitigate disturbances. Experiments show PMGS’s superior performance in reconstructing high-speed nonlinear rigid motion compared to mainstream dynamic methods.

Super-Resolution from a Blurry low-resolution image (SRB) constitutes a severely ill-posed inverse problem. Current learning-based SRB approaches primarily rely on synthetic, well-labeled paired datasets to regularize solution spaces, yet they exhibit limited generalizability in practical applications due to significant domain discrepancies between simulated degradations and real-world imaging conditions. To bridge this synthetic-to-real gap, we propose a novel Self-supervised Event-based SRB (SE-SRB) framework that leverages neuromorphic event streams as physical priors and adopts a lightweight neural architecture tailored for effective domain adaptation. Specifically, the proposed SE-SRB introduces a self-supervised learning paradigm based on asymmetric integral driven consistency, which enforces temporal coherence between predictions derived from RGB and asynchronous event streams at different time points. Extensive experiments validate that SE-SRB consistently outperforms state-of-the-art methods on both synthetic and real-world datasets. Built upon a lightweight parallel two-stream architecture, SE-SRB achieves high computational efficiency, featuring reduced parameter count, lower FLOPs, and real-time inference capability (40 FPS).

This paper explores scene affinity (AIScene), namely intra-scene consistency and inter-scene correlation, for semi-supervised LiDAR semantic segmentation in driving scenes. Adopting teacher-student training, AIScene employs a teacher network to generate pseudo-labeled scenes from unlabeled data, which then supervise the student network’s learning. Unlike most methods that include all points in pseudo-labeled scenes for forward propagation but only pseudo-labeled points for backpropagation, AIScene removes points without pseudo-labels, ensuring consistency in both forward and backward propagation within the scene. This simple point erasure strategy effectively prevents unsupervised, semantically ambiguous points (excluded in backpropagation) from affecting the learning of pseudo-labeled points. Moreover, AIScene incorporates patch-based data augmentation, mixing multiple scenes at both scene and instance levels. Compared to existing augmentation techniques that typically perform scene-level mixing between two scenes, our method enhances the semantic diversity of labeled (or pseudo-labeled) scenes, thereby improving the semi-supervised performance of segmentation models. Experiments show that AIScene outperforms previous methods on two popular benchmarks across four settings, achieving notable improvements of 1.9% and 2.1% in the most challenging 1% labeled data. The code will be released at https://github.com/azhuantou/AIScene.

Recent advances in 3D Gaussian Splatting have shown remarkable potential for novel view synthesis. However, most existing large-scale scene reconstruction methods rely on the divide-and-conquer paradigm, which often leads to the loss of global scene information and requires complex parameter tuning due to scene partitioning and local optimization. To address these limitations, we propose MixGS, a novel holistic optimization framework for large-scale 3D scene reconstruction. MixGS models the entire scene holistically by integrating camera pose and Gaussian attributes into a view-aware representation, which is decoded into fine-detailed Gaussians. Furthermore, a novel mixing operation combines decoded and original Gaussians to jointly preserve global coherence and local fidelity. Extensive experiments on large-scale scenes demonstrate that MixGS achieves state-of-the-art rendering quality and competitive speed, while significantly reducing computational requirements, enabling large-scale scene reconstruction training on a single 24GB VRAM GPU.

By leveraging the blur-noise trade-off, imaging with non-uniform exposures largely extends the image acquisition flexibility in harsh environments. However, the limitation of conventional cameras in perceiving intra-frame dynamic information prevents existing methods from being implemented in the real-world frame acquisition for real-time adaptive camera shutter control. To address this challenge, we propose a novel Neuromorphic Shutter Control (NSC) system to avoid motion blur and alleviate instant noise, where the extremely low latency of events is leveraged to monitor the real-time motion and facilitate the scene-adaptive exposure. Furthermore, to stabilize the inconsistent Signal-to-Noise Ratio (SNR) caused by the non-uniform exposure times, we propose an event-based image denoising network within a self-supervised learning paradigm, i.e., SEID, exploring the statistics of image noise and inter-frame motion information of events to obtain artificial supervision signals for high-quality imaging in real-world scenes. To illustrate the effectiveness of the proposed NSC, we implement it in hardware by building a hybrid-camera imaging prototype system, with which we collect a real-world dataset containing well-synchronized frames and events in diverse scenarios with different target scenes and motion patterns. Experiments on the synthetic and real-world datasets demonstrate the superiority of our method over state-of-the-art approaches.

Object detection is critical in autonomous driving, and it is more practical yet challenging to localize objects of unknown categories: an endeavour known as Class-Agnostic Object Detection (CAOD). Existing studies on CAOD predominantly rely on RGB cameras, but these frame-based sensors usually have high latency and limited dynamic range, leading to safety risks under extreme conditions like fast-moving objects, overexposure, and darkness. In this study, we turn to the event-based vision, featured by its sub-millisecond latency and high dynamic range, for robust CAOD. We propose Detecting Every Object in Events (DEOE), an approach aimed at achieving high-speed, class-agnostic object detection in event-based vision. Built upon the fast event-based backbone: recurrent vision transformer, we jointly consider the spatial and temporal consistencies to identify potential objects. The discovered potential objects are assimilated as soft positive samples to avoid being suppressed as backgrounds. Moreover, we introduce a disentangled objectness head to separate the foreground-background classification and novel object discovery tasks, enhancing the model's generalization in localizing novel objects while maintaining a strong ability to filter out the background. Extensive experiments confirm the superiority of our proposed DEOE in both open-set and closed-set settings, outperforming strong baseline methods.

Due to the extremely low latency, events have recently been utilized to complement lost information in motion deblurring. Existing approaches largely rely on the perfect pixel-wise alignment between intensity images and events, which usually conflicts with the real world. To tackle this problem, we propose a novel coarse-to-fine framework, named network of event-based motion deblurring with stereo event and intensity cameras (St-EDNet), to recover high-quality images directly from the misaligned inputs that contain both blurry images and the concurrent event stream. Specifically, the coarse spatial alignment of the blurry image and the event stream is first implemented with a cross-modal stereo-matching module without the need for ground-truth depths. Then, a dual-feature embedding architecture is proposed to gradually build the fine bidirectional association of the coarsely aligned data and reconstruct the sequence of the latent sharp images. Furthermore, we build a new dataset with stereo event and intensity cameras (StEIC), containing real-world events, intensity images, and dense disparity maps. Experiments on real-world datasets demonstrate the superiority of the proposed network over state-of-the-art methods. The code and dataset are available at https://mingyuan-lin.github.io/St-ED_web/.

Event-based Synthetic Aperture Imaging (E-SAI) extends the SAI technique to observe targets behind extremely dense occlusions. Existing approaches remain confined to the de-occlusion of a specific depth plane, i.e., single depth in focus, unable to be applied to observe occluded targets with varying depths due to the decreased focus range. To achieve All-in-Focus E-SAI, i.e., recovering the occlusion-free image of all depth planes, the depth information behind the occlusions should be given to ensure accurate event refocusing. In this paper, we first prove the feasibility of predicting the depth map from captured events in the presence of dense occlusions. Then, we propose the ESAI-AF network, which consists of a Depth Estimation Module (DEM) designed to estimate the depth information from multi-view events and an Image Enhancement Module (IEM) designed to reconstruct high-quality occlusion-free images from the refocused events. We employ only multi-view occlusion-free images as supervised signals for end-to-end training of the above modules. Extensive experiments have shown that the proposed method can effectively perform All-in-Focus image reconstruction of occluded multi-depth targets and achieves superior performance to existing methods.

Keypoint detection and tracking in traditional image frames are often compromised by image quality issues such as motion blur and extreme lighting conditions. Event cameras offer potential solutions to these challenges by virtue of their high temporal resolution and high dynamic range. However, they have limited performance in practical applications due to their inherent noise in event data. This paper advocates fusing the complementary information from image frames and event streams to achieve more robust keypoint detection and tracking. Specifically, we propose a novel keypoint detection network that fuses the textural and structural information from image frames with the high-temporal-resolution motion information from event streams, namely FE-DeTr. The network leverages a temporal response consistency for supervision, ensuring stable and efficient keypoint detection. Moreover, we use a spatio-temporal nearest-neighbor search strategy for robust keypoint tracking. Extensive experiments are conducted on a new dataset featuring both image frames and event data captured under extreme conditions. The experimental results confirm the superior performance of our method over both existing frame-based and event-based methods. Our code, pre-trained models, and dataset are available at https://github.com/yuyangpoi/FE-DeTr.

Most artificial lights exhibit subtle fluctuations in intensity and frequency in response to the influence of the grid's alternating current, providing the potential to estimate the Electric Network Frequency (ENF) from conventional frame-based videos. Nevertheless, the performance of Video-based ENF (V-ENF) estimation largely relies on the imaging quality and thus may suffer from significant interference caused by non-ideal sampling, scene diversity, motion interference, and extreme lighting conditions. In this paper, we show that the ENF can be extracted without the above limitations from a new modality provided by the so-called event camera, a neuromorphic sensor that encodes the light intensity variations and asynchronously emits events with extremely high temporal resolution and high dynamic range. Specifically, we formulate and validate the physical mechanism for the ENF captured in events and then propose a simple yet robust Event-based ENF (E-ENF) estimation method through mode filtering and harmonic enhancement. To validate the effectiveness, we build the first Event-Video ENF Dataset (EV-ENFD) and its extension EV-ENFD+ with diverse scenarios, including static, dynamic, and extreme lighting scenes. Comprehensive experiments have been conducted on our proposed datasets, showcasing that our proposed E-ENF significantly outperforms the V-ENF in extracting accurate ENF traces, especially in challenging environments.

Making line segment detectors more reliable under motion blurs is one of the most important challenges for practical applications, such as visual SLAM and 3D line mapping. Existing line segment detection methods face severe performance degradation for accurately detecting and locating line segments when motion blur occurs. While event data shows strong complementary characteristics to images for minimal blur and edge awareness at high-temporal resolution, potentially beneficial for reliable line segment recognition. To robustly detect line segments over motion blurs, we propose to leverage the complementary information of images and events. Specifically, we first design a general frame-event feature fusion network to extract and fuse the detailed image textures and low-latency event edges, which consists of a channel-attention-based shallow fusion module and a self-attention-based dual hourglass module. We then utilize the state-of-the-art wireframe parsing networks to detect line segments on the fused feature map. Moreover, due to the lack of line segment detection datasets with pairwise motion-blurred images and events, we contribute two datasets, i.e., synthetic FE-Wireframe and realistic FE-Blurframe, for network training and evaluation. Extensive analyses on the component configurations demonstrate the design effectiveness of our fusion network. When compared to the state-of-the-arts, the proposed approach achieves the highest detection accuracy while maintaining comparable real-time performance. In addition to being robust to motion blur, our method also exhibits superior performance for line detection under high dynamic range scenes.

Even though the collaboration between traditional and neuromorphic event cameras brings prosperity to frame-event based vision applications, the performance is still confined by the resolution gap crossing two modalities in both spatial and temporal domains. This paper is devoted to bridging the gap by increasing the temporal resolution for images, i.e., motion deblurring, and the spatial resolution for events, i.e., event super-resolving, respectively. To this end, we introduce CrossZoom, a novel unified neural Network (CZ-Net) to jointly recover sharp latent sequences within the exposure period of a blurry input and the corresponding High-Resolution (HR) events. Specifically, we present a multi-scale blur-event fusion architecture that leverages the scale-variant properties and effectively fuses cross-modal information to achieve cross-enhancement. Attention-based adaptive enhancement and cross-interaction prediction modules are devised to alleviate the distortions inherent in Low-Resolution (LR) events and enhance the final results through the prior blur-event complementary information. Furthermore, we propose a new dataset containing HR sharp-blurry images and the corresponding HR-LR event streams to facilitate future research. Extensive qualitative and quantitative experiments on synthetic and real-world datasets demonstrate the effectiveness and robustness of the proposed method.

The Rolling Shutter (RS) effect and motion blur are common challenges in images captured by CMOS cameras during dynamic scenes. Inspired by biological vision principles, event cameras capture intensity changes asynchronously with low latency, providing valuable insights into image degradation during exposure. This study addresses the dual challenges of rolling shutter correction and deblurring using event data, merging them into a unified one-stage network. This streamlined approach reduces cumulative errors and inference time compared to traditional two-stage methods. To achieve this, we introduce an Event Representation for Rolling Shutter Deblurring, which explicitly models the conversion relationship between the input RS blurry frame and the latent image using events. To enhance the fusion of image and event information, we present a Time-guided Cross-Modal Attention module. Furthermore, we improve performance by incorporating a Multi-Scale Context-Aware Transformer Block, effectively addressing varying degrees of distortion and blurriness using a multi-scale attention mechanism. Extensive experiments validate that our method outperforms existing state-of-the-art approaches.

Bimodal objects, such as the checkerboard pattern used in camera calibration, markers for object tracking, and text on road signs, to name a few, are prevalent in our daily lives and serve as a visual form to embed information that can be easily recognized by vision systems. While binarization from intensity images is crucial for extracting the embedded information in the bimodal objects, few previous works consider the task of binarization of blurry images due to the relative motion between the vision sensor and the environment. The blurry images can result in a loss in the binarization quality and thus degrade the downstream applications where the vision system is in motion. Recently, neuromorphic cameras offer new capabilities for alleviating motion blur, but it is non-trivial to first deblur and then binarize the images in a real-time manner. In this work, we propose an event-based binary reconstruction method that leverages the prior knowledge of the bimodal target's properties to perform inference independently in both event space and image space and merge the results from both domains to generate a sharp binary image. We also develop an efficient integration method to propagate this binary image to high frame rate binary video. Finally, we develop a novel method to naturally fuse events and images for unsupervised threshold identification. The proposed method is evaluated in publicly available and our collected data sequence, and shows the proposed method can outperform the SOTA methods to generate high frame rate binary video in real-time on CPU-only devices.

The goal of blurry image deblurring and unfolding task is to recover a single sharp frame or a sequence from a blurry one. Recently, its performance is greatly improved with introduction of a bio-inspired visual sensor, event camera. Most existing event-assisted deblurring methods focus on the design of powerful network architectures and effective training strategy, while ignoring the role of blur modeling in removing various blur in dynamic scenes. In this work, we propose to implicitly model blur in an image by computing blurriness representation with an event-assisted blurriness encoder. The learning of blurriness representation is formulated as a ranking problem based on specially synthesized pairs. Blurriness-aware image unfolding is achieved by integrating blur relevant information contained in the representation into a base unfolding network. The integration is mainly realized by the proposed blurriness-guided modulation and multi-scale aggregation modules. Experiments on GOPRO and HQF datasets show favorable performance of the proposed method against state-of-the-art approaches. More results on real-world data validate its effectiveness in recovering a sequence of latent sharp frames from a blurry image.

Conventional cameras face challenges when capturing motion information during the exposure due to their physical design, rendering the motion deblurring task ill-posed. To this end, we propose a Two-stage Residual-based Motion Deblurring (TRMD) framework for an event camera, which converts a blurry image into a sequence of sharp images, leveraging the abundant motion features encoded in events. In the first stage, a residual estimation network is trained to estimate the residual sequence, which measures the intensity difference between the intermediate frame and other frames sampled during the exposure. In the subsequent stage, the previously estimated residuals are combined with the blurry image to reconstruct the deblurred sequence based on the physical model of motion blur. To facilitate the efficient integration of image and event modalities for residual estimation, we propose a cross-modal fusion module based on spatial-channel attention, aiming to fuse the complementary spatial-temporal features of two modalities. Extensive experiments demonstrate that our method outperforms current state-of-the-art approaches on the synthetic dataset GOPRO and produces superior visualization with less noise and artifacts on the real blur event dataset REBlur.

The stereo event-intensity camera setup is widely applied to leverage the advantages of both event cameras with low latency and intensity cameras that capture accurate brightness and texture information. However, such a setup commonly encounters cross-modality parallax that is difficult to be eliminated solely with stereo rectification especially for real-world scenes with complex motions and varying depths, posing artifacts and distortion for existing Event-based Video Frame Interpolation (E-VFI) approaches. To tackle this problem, we propose a novel Stereo Event-based VFI (SE-VFI) network (SEVFI-Net) to generate high-quality intermediate frames and corresponding disparities from misaligned inputs consisting of two consecutive keyframes and event streams emitted between them. Specifically, we propose a Feature Aggregation Module (FAM) to alleviate the parallax and achieve spatial alignment in the feature domain. We then exploit the fused features accomplishing accurate optical flow and disparity estimation, and achieving better interpolated results through flow-based and synthesis-based ways. We also build a stereo visual acquisition system composed of an event camera and an RGB-D camera to collect a new Stereo Event-Intensity Dataset (SEID) containing diverse scenes with complex motions and varying depths. Experiments on public real-world stereo datasets, i.e., DSEC and MVSEC, and our SEID dataset, demonstrate that our proposed SEVFI-Net outperforms state-of-the-art methods by a large margin. The code and dataset are available athttps://dingchao1214.github.io/web_sevfi/.

Most of the artificial lights fluctuate in response to the grid's alternating current and exhibit subtle variations in terms of both intensity and spectrum, providing the potential to estimate the Electric Network Frequency (ENF)from conventional frame-based videos. Nevertheless, the performance of Video-based ENF (V-ENF) estimation largely re-lies on the imaging quality and thus may suffer from significant interference caused by non-ideal sampling, motion, and extreme lighting conditions. In this paper, we show that the ENF can be extracted without the above limitations from a new modality provided by the so-called event camera, a neuromorphic sensor that encodes the light intensity variations and asynchronously emits events with extremely high temporal resolution and high dynamic range. Specifically, we first formulate and validate the physical mechanism for the ENF captured in events, and then propose a simple yet robust Event-based ENF (E-ENF) estimation method through mode filtering and harmonic enhancement. Furthermore, we build an Event-Video ENF Dataset (EV-ENFD) that records both events and videos in diverse scenes. Extensive experiments on EV-ENFD demonstrate that our proposed E-ENF method can extract more accurate ENF traces, outperforming the conventional V-ENF by a large margin, especially in challenging environments with object motions and extreme lighting conditions. The code and dataset are available at https://github.com/x1x-creater/E-ENF.

Detecting arbitrarily oriented tiny objects poses intense challenges to existing detectors, especially for label assignment. Despite the exploration of adaptive label assignment in recent oriented object detectors, the extreme geometry shape and limited feature of oriented tiny objects still induce severe mismatch and imbalance issues. Specifically, the position prior, positive sample feature, and instance are mismatched, and the learning of extreme-shaped objects is biased and unbalanced due to little proper feature supervision. To tackle these issues, we propose a dynamic prior along with the coarse-to-fine assigner, dubbed DCFL. For one thing, we model the prior, label assignment, and object representation all in a dynamic manner to alleviate the mismatch issue. For another, we leverage the coarse prior matching and finer posterior constraint to dynamically assign labels, providing appropriate and relatively balanced supervision for diverse instances. Extensive experiments on six datasets show substantial improvements to the baseline. Notably, we obtain the state-of-the-art performance for one-stage detectors on the DOTA-v1.5, DOTA-v2.0, and DIOR-R datasets under single-scale training and testing. Codes are available at https://github.com/Chasel-Tsui/mmrotate-dcfl.

Event-based motion deblurring has shown promising results by exploiting low-latency events. However, current approaches are limited in their practical usage, as they assume the same spatial resolution of inputs and specific blurriness distributions. This work addresses these limitations and aims to generalize the performance of event-based de-blurring in real-world scenarios. We propose a scale-aware network that allows flexible input spatial scales and enables learning from different temporal scales of motion blur. A two-stage self-supervised learning scheme is then developed to fit real-world data distribution. By utilizing the relativity of blurriness, our approach efficiently ensures the restored brightness and structure of latent images and further generalizes deblurring performance to handle varying spatial and temporal scales of motion blur in a self-distillation manner. Our method is extensively evaluated, demonstrating remarkable performance, and we also introduce a real-world dataset consisting of multi-scale blurry frames and events to facilitate research in event-based deblurring.

Extracting building footprints from aerial images is essential for precise urban mapping with photogrammetric computer vision technologies. Existing approaches mainly assume that the roof and footprint of a building are well overlapped, which may not hold in off-nadir aerial images as there is often a big offset between them. In this paper, we propose an offset vector learning scheme, which turns the building footprint extraction problem in off-nadir images into an instance-level joint prediction problem of the building roof and its corresponding “roof to footprint” offset vector. Thus the footprint can be estimated by translating the predicted roof mask according to the predicted offset vector. We further propose a simple but effective feature-level offset augmentation module, which can significantly refine the offset vector prediction by introducing little extra cost. Moreover, a new dataset, Buildings in Off-Nadir Aerial Images (BONAI), is created and released in this paper. It contains 268,958 building instances across 3,300 aerial images with fully annotated instance-level roof, footprint, and corresponding offset vector for each building. Experiments on the BONAI dataset demonstrate that our method achieves the state-of-the-art, outperforming other competitors by 3.37 to 7.39 points in F1-score. The codes, datasets, and trained models are available athttps://github.com/jwwangchn/BONAI.git.

Super-Resolution from a single motion Blurred image (SRB) is a severely ill-posed problem due to the joint degradation of motion blurs and low spatial resolution. In this article, we employ events to alleviate the burden of SRB and propose an Event-enhanced SRB (E-SRB) algorithm, which can generate a sequence of sharp and clear images with High Resolution (HR) from a single blurry image with Low Resolution (LR). To achieve this end, we formulate an event-enhanced degeneration model to consider the low spatial resolution, motion blurs, and event noises simultaneously. We then build an event-enhanced Sparse Learning Network (eSL-Net++) upon a dual sparse learning scheme where both events and intensity frames are modeled with sparse representations. Furthermore, we propose an event shuffle-and-merge scheme to extend the single-frame SRB to the sequence-frame SRB without any additional training process. Experimental results on synthetic and real-world datasets show that the proposed eSL-Net++ outperforms state-of-the-art methods by a large margin. Datasets, codes, and more results are available at https://github.com/ShinyWang33/eSL-Net-Plusplus.

Although synthetic aperture imaging (SAI) can achieve the seeing-through effect by blurring out off-focus foreground occlusions while recovering in-focus occluded scenes from multi-view images, its performance is often deteriorated by dense occlusions and extreme lighting conditions. To address the problem, this paper presents an Event-based SAI (E-SAI) method by relying on the asynchronous events with extremely low latency and high dynamic range acquired by an event camera. Specifically, the collected events are first refocused by a Refocus-Net module to align in-focus events while scattering out off-focus ones. Following that, a hybrid network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs) is proposed to encode the spatio-temporal information from the refocused events and reconstruct a visual image of the occluded targets. Extensive experiments demonstrate that our proposed E-SAI method can achieve remarkable performance in dealing with very dense occlusions and extreme lighting conditions and produce high-quality images from pure events. Codes and datasets are available at https://dvs-whu.cn/projects/esai/.

Single image deraining (SIDR) often suffers from over/under deraining due to the nonuniformity of rain densities and the variety of raindrop scales. In this paper, we propose acontinuousdensity-guided network (CODE-Net) for SIDR. Particularly, it is composed of a rain streak extractor and a denoiser, where the convolutional sparse coding (CSC) is exploited to filter out noises from the extracted rain streaks. Inspired by the reweighted iterative soft-threshold (ISTA) for CSC, we address the problem of continuous rain density estimation by learning the weights with channel attention blocks from sparse codes. We further develop a multiscale strategy to depict rain streaks appearing at different scales. Experiments on synthetic and real-world data demonstrate the superiority of our methods over recent state-of-the-arts, in terms of both quantitative and qualitative results. Additionally, instead of quantizing rain density with several levels, our CODE-Net can provide continuous-valued estimations of rain densities, which is more desirable in real applications.

The Event-based Synthetic Aperture Imaging (E-SAI) has recently been proposed to see through extremely dense occlusions. However, the performance of E-SAI is not consistent under sparse occlusions due to the dramatic de-crease of signal events. This paper addresses this problem by leveraging the merits of both events and frames, leading to a fusion-based SAl (EF-SAI) that performs consistently under the different densities of occlusions. In particular, we first extract the feature from events and frames via multi-modal feature encoders and then apply a multi-stage fusion network for cross-modal enhancement and density-aware feature selection. Finally, a CNN decoder is employed to generate occlusion-free visual images from selected features. Extensive experiments show that our method effectively tackles varying densities of occlusions and achieves superior performance to the state-of-the-art SAl methods. Codes and datasets are available at https://github.com/smjsc/EF-SAI

Focus control (FC) is crucial for cameras to capture sharp images in challenging real-world scenarios. The autofocus (AF) facilitates the FC by automatically adjusting the focus settings. However, due to the lack of effective AF methods for the recently introduced event cameras, their FC still relies on naive AF like manual focus adjustments, leading to poor adaptation in challenging real-world conditions. In particular, the inherent differences between event and frame data in terms of sensing modality, noise, temporal resolutions, etc., bring many challenges in designing an effective AF method for event cameras. To address these challenges, we develop a novel event-based autofocus framework consisting of an event-specific focus measure called event rate (ER) and a robust search strategy called event-based golden search (EGS). To verify the performance of our method, we have collected an event-based autofocus dataset (EAD) containing well-synchronized frames, events, and focal positions in a wide variety of challenging scenes with severe lighting and motion conditions. The experiments on this dataset and additional real-world scenarios demonstrated the superiority of our method over state-of-the-art approaches in terms of efficiency and accuracy.

Slow shutter speed and long exposure time of frame-based cameras often cause visual blur and loss of inter-frame information, degenerating the overall quality of captured videos. To this end, we present a unified framework of event-based motion deblurring and frame interpolation for blurry video enhancement, where the extremely low latency of events is leveraged to alleviate motion blur and facilitate intermediate frame prediction. Specifically, the mapping relation between blurry frames and sharp latent images is first predicted by a learnable double integral network, and a fusion network is then proposed to refine the coarse results via utilizing the information from consecutive blurry inputs and the concurrent events. By exploring the mutual constraints among blurry frames, latent images, and event streams, we further propose a self-supervised learning framework to enable network training with real-world blurry videos and events. Extensive experiments demonstrate that our method compares favorably against the state-of-the-art approaches and achieves remarkable performance on both synthetic and real-world datasets. Codes are available at https://github.com/XiangZ-0/EVDI.

Compressed sensing based algorithms are utilized to obtain high-resolution time-frequency distribution (TFD) with negligible cross-terms (CTs), however, performance deteriorates when the signal is composed of closely-located or overlapped components. Moreover, there is still an impressive resolution gap between obtained and ideal TFDs. Aiming at eliminating CTs meanwhile preserving resolution as high as possible in various hard cases, we propose a new U-Net aided iterative shrinkage-thresholding algorithm (U-ISTA), where unfolded ISTA with structure-aware thresholds is exploited to reconstruct near-ideal TFD. Specifically, we regard the U-Net as an adaptive threshold block, and structured sparsity of TFD is learned from numerous training data, thus underlying dependencies among neighboring time-frequency coefficients are incorporated into reconstruction. Experimental results over synthetic and real-life signals demonstrate that the proposed U-ISTA achieves superior performance compared with state-of-the-art algorithms.

The design of high-resolution and cross-term (CT) free time-frequency distributions (TFDs) has been an open problem. Classical kernel based methods are limited by the trade-off between resolution and CT suppression, even under optimally derived parameters. To break the current limitation, we propose a data-driven model directly based on Wigner-Ville distribution (WVD). The proposed data-driven high-resolution TFD (DH-TFD) includes several stacked multi-channel convolutional kernels. Specifically, convolutional layers with skipping operators are utilized to learn coarse features, while a weighted block is employed to refine these features independently in both channel and spatial dimensions. By doing so, CTs can be effectively eliminated while maintaining a high resolution. Numerical experiments on both synthetic and real-world data confirm the superiority of the proposed DH-TFD in simultaneously extracting and representing a target signal over state-of-the-art methods.

Detecting airplanes from high-resolution remote sensing images has a variety of applications. The characteristics of clear details, rich spatial and texture information of objects in high-resolution remote sensing images make it possible to identify different types of airplanes from backgrounds. However, airplanes usually exhibit slight inter-class discrepancy and unbalanced class distribution, which pose significant challenges to fine-grained detection of airplanes. In this paper, we propose the ISCL, an Instance Switching-based Contrastive Learning method for fine-grained airplane detection. Specifically, we introduce a Contrastive Learning-based Module (CLM) to widen the inter-class distance while narrowing the intra-class distance by optimizing feature space distribution with the InfoNCE+loss, which is built on a serial head in a cascaded way. Then, we design a Refined Instance Switching (ReIS) module to alleviate the class imbalance problem. To take full advantage of the CLM and ReIS, we further introduce an optimization strategy which is an organic combination of the two modules to widen the distances of different airplane categories that are easily confused. In addition, we contribute a fine-grained attribute-assisted dataset, dubbed GF-RarePlanes Dataset (GRD), to help the detectors better learn the subtle differences between the airplanes. Extensive experiments on two datasets (i.e., GF and FAIR1M) demonstrate that our proposed method can significantly improve the accuracy of fine-grained airplane detection under both HBB and OBB scenarios. Dataset and codes will be available at https://lanxin1011.github.io/ISCL/.

Synthetic aperture imaging (SAI) is able to achieve the see through effect by blurring out the off-focus foreground occlusions and reconstructing the in-focus occluded targets from multi-view images. However, very dense occlusions and extreme lighting conditions may bring significant disturbances to the SAI based on conventional frame-based cameras, leading to performance degeneration. To address these problems, we propose a novel SAI system based on the event camera which can produce asynchronous events with extremely low latency and high dynamic range. Thus, it can eliminate the interference of dense occlusions by measuring with almost continuous views, and simultaneously tackle the over/under exposure problems. To reconstruct the occluded targets, we propose a hybrid encoder-decoder network composed of spiking neural networks (SNNs) and convolutional neural networks (CNNs). In the hybrid network, the spatio-temporal information of the collected events is first encoded by SNN layers, and then transformed to the visual image of the occluded targets by a style-transfer CNN decoder. Through experiments, the proposed method shows remarkable performance in dealing with very dense occlusions and extreme lighting conditions, and high quality visual images can be reconstructed using pure event data.

In this paper, we propose an end-to-end learning framework for event-based motion deblurring in a self-supervised manner, where real-world events are exploited to alleviate the performance degradation caused by data inconsistency. To achieve this end, optical flows are predicted from events, with which the blurry consistency and photometric consistency are exploited to enable self-supervision on the deblurring network with real-world data. Furthermore, a piecewise linear motion model is proposed to take into account motion non-linearities and thus leads to an accurate model for the physical formation of motion blurs in the real-world scenario. Extensive evaluation on both synthetic and real motion blur datasets demonstrates that the proposed algorithm bridges the gap between simulated and real-world motion blurs and shows remarkable performance for eventbased motion deblurring in real-world scenarios.

With the continuous improvement of attack methods, there are more and more distributed, complex, targeted attacks in which the attackers use combined attack methods to achieve the purpose. Advanced cyber attacks include multiple stages to achieve the ultimate goal. Traditional intrusion detection systems such as endpoint security management tools, firewalls, and other monitoring tools generate a large number of alerts during the attack. These alerts include attack clues, as well as many false positives unrelated to attacks. Security analysts need to analyze a large number of alerts and find useful clues from them and reconstruct attack scenarios. However, most traditional security monitoring tools cannot correlate alerts from different sources, so many multi-step attacks are still completely unnoticed, requiring manual analysis by security analysts like finding a needle in a haystack. We propose MAAC, a multi-step attack alert correlation system, which reduces repeated alerts and combines multi-step attack paths based on alert semantics and attack stages. The evaluation results of the real-world datasets show that MAAC can effectively reduce the alerts by 90% and find attack paths from a large number of alerts.

Over the past few decades, image de-raining has witnessed substantial progress due to the development of priors and deep learning based methods. However, few studies combine the merits of both. In this paper, we argue that domain expertise of conventional convolutional sparse coding (CSC) is still valuable, and it can be combined with the key ingredients of deep learning to achieve further improved results. Specifically, motivated by the success of reweighting algorithms, we propose solving the CSC model by learning weighted iterative soft thresholding algorithm (LwISTA) in a convolutional manner where the reweighted ℓ1-norm is introduced. Based on this, we present a novel framework for single image de-raining, in which the channel attention is employed to learn the weight. Extensive experiments demonstrate the superiority of our method over recent state-of-the-art image de-raining methods, in terms of both quantitative and qualitative results.

Visible and near-infrared (VIS-NIR) face recognition remains a challenging task due to distinctions between spectral components of two modalities. Inspired by the CycleGAN, this paper presents a method aiming to translate between VIS and NIR face images. To achieve this, we propose a new facial feature embedded CycleGAN. Firstly, to learn the particular feature while preserving common facial representation between VIS and NIR domains, we employ a general facial feature extractor (FFE) to extract effective features. Herein the MobileFaceNet is pre-trained on a VIS face database and serves as the FFE. Secondly, the domain-invariant feature learning is enhanced by proposing a new pixel consistency loss. Lastly, we establish a new WHU VIS-NIR database including varies in face rotation and expressions to enrich the training data. Experimental results on the Oulu-CASIA and our WHU VIS-NIR databases show that the proposed FFE-based CycleGAN (FFE-CycleGAN) outperforms some state-of-the-art methods and achieves 96.5% accuracy.

The event camera is a novel sensor that records brightness change in the form of asynchronous events with high temporal resolution, and simultaneously outputs intensity images with a lower frame rate. Events recorded by sensors have a lot of noise and the intensity images captured often suffer from motion blur and noise effects. Therefore, to reconstruct high quality images is of great significance for the application of event camera in computer vision. However, the existing reconstruction methods only addressed the motion blur issue without considering the influence of noise. In this paper, we propose a variational model by using spatial smooth constraint regularization to recover clean image frames from blurry and noisy camera images and events at any frame rate. We present experimental results on synthetic dataset as well as real dataset with high speed and high dynamic range to demonstrate that the proposed algorithm is superior to the other reconstruction algorithms.

Researching networks' theoretical properties and behavior have drawn considerable attention from the perspective of ordinary differential equation (ODE). For solving ODE, explicit and implicit Euler schemes are the most common methods. Some works utilize explicit Euler theory to analyze and design networks. However, focusing on parameters convergence and system stability, implicit Euler has been proved to be better than explicit one. It motivates us to explore implicit Euler's potential in neural networks. In this paper, we establish connections between implicit Euler and networks, which also effectively explain some existing networks such as LISTA and DRRN. In addition, by re-deriving implicit Euler, we propose an adaptive implicit network (AIM-NET) which allows model to have a flexible convergence interval to ensure parameters convergence as well as model performance. Particularly, we obtain AIM-LISTA and AIM-DRRN by applying AIM-NET on LISTA and DRRN respectively. Finally, we perform experiments on both synthetic data and real images and the experimental results show that adaptive implicit structure is able to significantly improve performance.

sEvent-to-image translation is a popular problem where the goal is to obtain a mapping from an input event stream to an output intensity image using a set of aligned image pairs for training. However, due to the high temporal resolution of the event camera, the alignment of the ground truth to the events is difficult to acquire. In this paper, we firstly propose an enhanced Cycle-Consistency Generative Adversarial Networks (enhanced Cycle-GAN), called Cycle-Event Network, where paired data is not required for the training phase. Besides, noises from event cameras can severely contaminate the data quality and makes the reconstruction an ill-posed problem. In order to generate high frame-rate video from events with less noisy background and richer texture details, a novel attention mechanism (Residual Channel-wise Attention Gate) is then proposed to reweight the feature of the generator in Cycle-Event Network. The qualitative results are presented on several datasets, and the quantitative comparisons clearly demonstrate the effectiveness of our proposed Cycle-Event Networks.

Comparing to the low level local features, Transform Invariant Low-Rank Textures (TILT) can in some sense globally rectify a large class of low-rank textures in 2D images, and thus more accurate and robust. However, TILT is still rather rudimentary, and have some limitations in applications. In this paper, we proposed a novel algorithm for better solving TILT. Our method is based on the application of Bayesian framework in robust principal component analysis (RPCA), besides less local minima, nonparametric Bayesian method introduces the uncertainty in the parameters, which make our new algorithm can handle more complex situations. Experimental results on both synthetic and real data indicate that our new algorithm outperforms the existing algorithm especially for the case with corruptions and occlusions.

Compressive sensing (CS) provides a new paradigm of sub‐Nyquist sampling which can be considered as an alternative to Nyquist sampling theorem. In particular, providing that signals are with sparse representations in some domain, information can be perfectly preserved even with small amount of measurements captured by random projections. Besides sparsity prior of signals, the inherent structure property behind some specific signals is often exploited to enhance the reconstruction accuracy. In this study, the authors are aiming to take into account the cluster structure property of sparse signals, of which the non‐zero coefficients appear in clustered blocks. By modelling simultaneously both sparsity and cluster prior within a hierarchical statistical Bayesian framework, a non‐parametric algorithm can be obtained through variational Bayes approach to recover original sparse signals. The proposed algorithm could be slightly considered as a generalisation of Bayesian CS (BCS), but with a consideration on cluster property. Consequently, the performance of the proposed algorithm is at least as good as BCS, which is verified by the experimental results.

In this letter, a sinusoidal time-frequency distribution based filtering (STFD-F) algorithm is proposed and analysed for estimating mono-component stationary sinusoidal signals embedded in strong noise. An initial frequency estimation of the sinusoidal signal is required in the STFD-F algorithm. We theoretically derive the closed-form expressions of the variance and the bias of the estimated signal using the STFD-F, and show that the performance of the STFD-F is dependent on the frequency estimation accuracy, which can be gradually refined by performing an iterative STFD-F procedure. Computer simulations on synthetic sinusoidal signals are presented to corroborate the theoretical analysis.

Armed with structures, group sparsity can be exploited to extraordinarily improve the performance of adaptive estimation. In this letter, the adaptive estimation algorithm for cluster structured sparse signals, called A-CluSS, is proposed. In particular, a hierarchical Bayesian model is built, where both sparse prior and cluster structured prior are exploited simultaneously. The adaptive updating formulas for statistical variables are obtained via the variational Bayesian inference and the resulted algorithms can adaptively estimate the cluster structured sparse signals without knowledge of block size, block numbers and block locations. Superiority of proposed A-CluSS is demonstrated via various simulations.

In traditional framework of Compressive Sensing (CS), only sparse prior on the property of signals in time or frequency domain is adopted to guarantee the exact inverse recovery. Besides sparse prior, cluster prior is introduced in this paper in order to investigate a class of structural sparse signals, called clustered sparse signals. A hierarchical statistical model is employed via Bayesian approach to model both the sparse prior and cluster prior and Markov Chain Monte Carlo (MCMC) sampling is implemented for the inference. Unlike the state-of-the-art algorithms based on the cluster prior, the proposed algorithm solves the inverse problem without any prior knowledge of the cluster parameters, even without the knowledge of the sparsity. The experimental results show that the proposed algorithm outperforms many state-of-the-art algorithms.

Compressive sensing is a new methodology to capture signals at sub-Nyquist rate. To guarantee exact recovery from compressed measurements, one should choose specific matrix, which satisfies the Restricted Isometry Property (RIP), to implement the sensing procedure. In this letter, we propose to construct the sensing matrix with chaotic sequence following a trivial method and prove that with overwhelming probability, the RIP of this kind of matrix is guaranteed. Meanwhile, its experimental comparisons with Gaussian random matrix, Bernoulli random matrix and sparse matrix are carried out and show that the performances among these sensing matrix are almost equal.