This paper introduces the Spatio-Temporal Adaptive Context Optimization (STACO) method, which enhances the quality of contextual prediction across various resolutions, essential for subsequent compression. The STACO takes predicted contexts$C_t^{\{1,2,3\}}$as input and improves their quality by aligning them better with decoded features$f_{t}$, thus boosting coding efficiency. The STACO comprises Quality Perception Units and Consistency Synergy Modules, arranged in a hierarchical stacked architecture. This multi-scale design enables simultaneous processing of contexts at different spatial resolutions and facilitates information exchange through upsampling and downsampling. Enhanced contexts$\tilde{C}_{t}^{\{1,2,3\}}$are output after passing through residual connections, ensuring better alignment with reconstructed features. Using VTM-11.0 as anchor, the STACO significantly improves compression efficiency on common test condition (CTC) in HEVC, achieving an average BD-rate reduction of 17.99% for PSNR and 43.84% for MS-SSIM. By incorporating spatial quality mapping and temporal propagation, STACO offers a significant advancement in video compression.

Neural Video Compression (NVC) has advanced significantly in recent years, with improvements in inter prediction techniques. In inter prediction, most NVC approaches utilize pixel information or temporal features from neighboring frames as reference information, while using optical flow to represent motion information. In this paper, we introduce an innovative and efficient method for Dynamic Temporal Reference Aggregation (DTRA). The proposed DTRA consists of two components: Temporal Information Compensation (TIC) and Feature Level Motion Information Enhancement (MIE). The TIC module generates compensation information by leveraging long-term temporal information from the decoding buffer, enriching the semantic content of the reference features and enhancing their texture details. The MIE module refines the motion features at the encoder side and divides the motion information into multiple groups for diverse motion alignment at the decoder side, thereby improving the motion compensation. Extensive experiments demonstrate the effectiveness of the proposed method, achieving an average bitrate savings of 9.67% compared to state-of-the-art (SOTA) approaches.

Recently, the bio-inspired spike camera with continuous motion recording capability has attracted tremendous attention due to its ultra high temporal resolution imaging characteristic. Such imaging feature results in huge data storage and transmission burden compared to that of traditional camera, raising severe challenge and imminent necessity in compression for spike camera captured content. Existing lossy data compression methods could not be applied for compressing spike streams efficiently due to integrate-and-fire characteristic and binarized data structure. Considering the imaging principle and information fidelity of spike cameras, we propose a novel Reconstruction-based Contextual Spike Compression (RCSC) framework, which contains scene reconstruction, contextual image compression and spike generation. To our knowledge, it is the first learning-based model for efficient and robust spike stream compression with informative fidelity. Extensive experimental results show that our model outperforms the state-of-the-art conventional codec VVC intra by 6.14% and surpasses the state-of-the-art learned codec by 2.53% in BD-rate reduction, establishing a strong baseline for spike compression.

The advent of Neuromorphic spike cameras has garnered significant attention for their ability to capture continuous motion with unparalleled temporal resolution. However, this imaging attribute necessitates considerable resources for binary spike data storage and transmission. In light of compression and spike-driven intelligent applications, we present the notion of Spike Coding for Intelligence (SCI), wherein spike sequences are compressed and optimized for both bit-rate and task performance. Drawing inspiration from the mammalian vision system, we propose a dual-pathway architecture for separate processing of spatial semantics and motion information, which is then merged to produce features for compression. A refinement scheme is also introduced to ensure consistency between decoded features and motion vectors. We further propose a temporal regression approach that integrates various motion dynamics, capitalizing on the advancements in warping and deformation simultaneously. Comprehensive experiments demonstrate our scheme achieves state-of-the-art (SOTA) performance for spike compression and analysis. We achieve an average 17.25% BD-rate reduction compared to SOTA codecs and a 4.3% accuracy improvement over SpiReco for spike-based classification, with 88.26% complexity reduction and 42.41% inference time saving on the encoding side.

The widespread adoption of bio-inspired cameras has catalyzed the development of spike-based intelligent applications. Despite its innovative imaging principle allows for functionality in extreme scenarios, the intricate nature of spike signals poses processing challenges to achieve desired performance. Traditional methods struggles to deliver visual perception and temporal prediction simultaneously, and they lack the flexibility needed for diverse intelligent applications. To address this problem, we analyze the spatio-temporal correlations between spike information at different temporal scales. A novel spike processing method is introduced for compact spike representations that utilizes intra-scale correlation for higher predictive accuracy. Additionally, we propose a multi-scale spatio-temporal aggregation unit (MSTAU) that further leverages inter-scale correlation to achieve efficient perception and precise prediction. Experimental results show noticeable improvements in scene reconstruction and object classification, with increases of 3.49dB in scene reconstruction quality and 2.20% in accuracy, respectively. Besides, the proposed method accommodate different visual applications via switching analysis models, offering a novel perspective for spike processing.