Spike camera is a kind of neuromorphic camera that records dynamic scenes by firing a stream of binary spikes with extremely high temporal resolution. It demonstrates great potential for vision tasks in high-speed scenarios. One limitation in its current implementation is the relatively low spatial resolution. This paper develops a network called Spk2SRImgNet to super-resolve high resolution images from low resolution spike stream. However, fluctuations in spike stream hinder the performance of spike camera super resolution. To address this issue, we propose a motion aligned collaborative filtering (MACF) module, which is motivated by key ideas in classic image restoration schemes to mitigate fluctuations in spike data. MACF leverages the temporal similarity of spike stream to acquire similar features from neighboring moments via motion alignment. To separate disturbances from features, MACF filters these similar features jointly in transform domain to exploit representation sparsity, and generates refinement features that will be used to update initial fluctuated features. Specifically, MACF designs an inverse motion alignment operation to map these refinement features back to their original positions. The initial features are aggregated with the repositioned refinement features to enhance reliability. Experimental results demonstrate that the proposed method achieves state-of-the-art performance compared with existing methods.

Spike camera is a retina-inspired neuromorphic camera which can capture dynamic scenes of high-speed motion by firing a continuous stream of spikes at an extremely high temporal resolution. The limitation in the current design is that each spike only represents the arrival of a fixed amount of photons. It can not deal with strong light areas in which the amount of accumulated photons reaches the pre-specified threshold multiple times within a single readout interval. In this paper, we propose a new spike camera model of high-speed imaging for high dynamic range scenarios. In this scheme, each pixel accumulates the incoming photons persistently and generates a new type of spike stream in which each spike symbol may be associated with different levels, indicating the arrival of different amounts of photons since the last readout. This enables the camera to support dynamic scenes with wider dynamic range. To achieve this, we propose a two-level buffer mechanism, one for photon accumulation and one for spike-firing encoding. We use a register to hold the number of spike-firings which has not been read out yet. At each readout time, the major part in the counter is read out via a carefully designed exponential encoding and the counter is updated. Such encoding and readout strategy enables a very efficient expansion of the dynamic range using a small number of encoding bits. Furthermore, we propose an image reconstruction scheme for the proposed camera, utilizing both spike intervals and spike levels to recover the light intensity. We incorporate Mamba and propose a temporal-spatial selective scan mechanism to extract temporal-spatial correlation within spike streams. We employ a pyramid adaptive filtering and alignment module to achieve coarse-to-fine feature alignment. Experimental results show that the proposed scheme can achieve better imaging quality and outperform the existing spike camera in high dynamic range scenarios.

With the prevalence of emerging computer vision applications, the demand for capturing dynamic scenes with high-speed motion has increased. A kind of neuromorphic sensor called spike camera shows great potential in this aspect since it generates a stream of binary spikes to describe the dynamic light intensity with a very high temporal resolution. Color spike camera (CSC) was recently invented to capture the color information of dynamic scenes via a color filter array (CFA) on the sensor. This paper proposes a long short-term temporal aggregation strategy of spike signals. First, we utilize short-term temporal correlation to adaptively extract temporal features of each time point. Then we align the features and aggregate them to exploit long-term temporal correlation, suppressing undesired motion blur. To implement the strategy, we design a CSC reconstruction network. Based on adaptive short-term temporal aggregation, we propose a spike representation module to extract temporal features of each color channel, leveraging multiple temporal scales. Considering the long-term temporal correlation, we develop an alignment module to align the temporal features. In particular, we perform motion alignment of red and blue channels with the guidance of the higher-sampling-rate green channel, leveraging motion consistency among color channels. Besides, we propose a module to aggregate the aligned temporal features for the restored color image, which exploits color channel correlation. We have also developed a CSC simulator for data generation. Experimental results demonstrate that our method can restore color images with fine texture details, achieving state-of-the-art CSC reconstruction performance.

As a neuromorphic camera with high temporal resolution, spike camera can capture dynamic scenes with high-speed motion. Recently, spike camera with a color filter array (CFA) has been developed for color imaging. There are some methods for spike camera demosaicing to reconstruct color images from Bayer-pattern spike streams. However, the demosaicing results are bothered by severe noise in spike streams, to which previous works pay less attention. In this paper, we propose an iterative joint demosaicing and denoising network (SJDD-Net) for spike cameras based on the observation model. Firstly, we design a color spike representation (CSR) to learn latent representation from Bayer-pattern spike streams. In CSR, we propose an offset-sharing deformable convolution module to align temporal features of color channels. Then we develop a spike noise estimator (SNE) to obtain features of the noise distribution. Finally, a color correlation prior (CCP) module is proposed to utilize the color correlation for better details. For training and evaluation, we designed a spike camera simulator to generate Bayer-pattern spike streams with synthesized noise. Besides, we captured some Bayer-pattern spike streams, building the first real-world captured dataset to our knowledge. Experimental results show that our method can restore clean images from Bayer-pattern spike streams. The source codes and dataset are available at https://github.com/csycdong/SJDD-Net.

As a bio-inspired vision sensor with ultra-high speed, spike cameras exhibit great potential in recording dynamic scenes with high-speed motion or drastic light changes. Different from traditional cameras, each pixel in spike cam-eras records the arrival of photons continuously by firing binary spikes at an ultra-fine temporal granularity. In this process, multiple factors impact the imaging, including the photons' Poisson arrival, thermal noises from circuits, and quantization effects in spike readout. These factors intro-duce fluctuations to spikes, making the recorded spike in-tervals unstable and unable to reflect accurate light intensi-ties. In this paper, we present an approach to deal with spike fluctuations and boost spike camera image reconstruction. We first analyze the quantization effects and reveal the unbi-ased estimation attribute of the reciprocal of differential of spike firing time (DSFT). Based on this, we propose a spike representation module to use DSFT with multiple orders for fluctuation suppression, where DSFT with higher or-ders indicates spike integration duration between multiple spikes. We also propose a module for inter-moment feature alignment at multiple granularities. The coarser alignment is based on patch-level cross-attention with a local search strategy, and the finer alignment is based on deformable convolution at the pixel level. Experimental results demon-strate the effectiveness of our method on both synthetic and real-captured data. The source code and dataset are avail-able at https://github.com/ruizhao26/BSF.

Spike camera is a neuromorphic camera that recurrently accumulates photons and fires spikes to record the incident light intensity at very high temporal resolution, making it particularly suitable for recording high dynamic scenes. This paper addresses the problem of image reconstruction for spike camera. Due to the Poisson effect of photon arrival and the quantization effect of spike readout, the spike interval calculated from a single spike cycle cannot reflect the light intensity accurately. Firstly, this paper analyzes the error of spike interval estimation under static light intensity. Then, this paper focuses on the temporal correlation of continuous spikes under dynamic light intensity. Specifically, it considers 3D space time similarity to weighted average multiple continuous spike intervals for the intensity estimation at a certain pixel. Experimental results demonstrate that the proposed method achieves better performance in both objective and subjective aspects compared with previous reconstruction methods.

For capturing dynamic scenes with ultra-fast motion, neuromorphic cameras with extremely high temporal resolution have demonstrated their great capability and potential. Different from the event cameras that only record relative changes in light intensity, spike camera fires a stream of spikes according to a full-time accumulation of photons so that it can recover the texture details for both static areas and dynamic areas. Recently, color spike camera has been invented to record color information of dynamic scenes using a color filter array (CFA). However, demosaicing for color spike cameras is an open and challenging problem. In this paper, we develop a demosaicing network, called CSpkNet, to reconstruct dynamic color visual signals from the spike stream captured by the color spike camera. Firstly, we develop a light inference module to convert binary spike streams to intensity estimates. In particular, a feature-based channel attention module is proposed to reduce the noises caused by quantization errors. Secondly, considering both the Bayer configuration and object motion, we propose a motion-guided filtering module to estimate the missing pixels of each color channel, without undesired motion blur. Finally, we design a refinement module to improve the intensity and details, utilizing the color correlation. Experimental results demonstrate that CSpkNet can reconstruct color images from the Bayer-pattern spike stream with promising visual quality.

Recently, machine learning-based image compression has attracted increasing interests and is approaching the state-of-the-art compression ratio. But unlike traditional codec, it lacks a universal optimization method to seek efficient representation for different images. In this paper, we develop a plug-and-play optimization framework for seeking higher compression ratio, which can be flexibly applied to existing and potential future compression networks. To make the latent representation more efficient, we propose a novel latent optimization algorithm to adaptively remove the redundancy for each image. Additionally, inspired by the potential of side information for traditional codecs, we introduce side information into our framework, and integrate side information optimization with latent optimization to further enhance the compression ratio. In particular, with the joint side information and latent optimization, we can achieve fine rate control using only single model instead of training different models for different rate-distortion trade-offs, which significantly reduces the training and storage cost to support multiple bit rates. Experimental results demonstrate that our proposed framework can remarkably boost the machine learning-based compression ratio, achieving more than 10% additional bit rate saving on three different representative network structures. With the proposed optimization framework, we can achieve 7.6% bit rate saving against the latest traditional coding standard VVC on Kodak dataset, yielding the state-of-the-art compression ratio.

Occlusion and motion blur make it challenging to interpolate video frame, since estimating complex motions between two frames is hard and unreliable, especially in highly dynamic scenes. This paper aims to address these issues by exploiting spike stream as auxiliary visual information between frames to synthesize target frames. Instead of estimating motions by optical flow from RGB frames, we present a new dual-modal pipeline adopting both RGB frames and the corresponding spike stream as inputs (SVFI). It extracts the scene structure and objects' outline feature maps of the target frames from spike stream. Those feature maps are fused with the color and texture feature maps extracted from RGB frames to synthesize target frames. Benefited by the spike stream that contains consecutive information between two frames, SVFI can directly extract the information in occlusion and motion blur areas of target frames from spike stream, thus it is more robust than previous optical flow-based methods. Experiments show SVFI outperforms the SOTA methods on wide variety of datasets. For instance, in 7 and 15 frame skip evaluations, it shows up to 5.58 dB and 6.56 dB improvements in terms of PSNR over the corresponding second best methods BMBC and DAIN. SVFI also shows visually impressive performance in real-world scenes.

Spike camera is a kind of neuromorphic sensor that uses a novel ``integrate-and-fire'' mechanism to generate a continuous spike stream to record the dynamic light intensity at extremely high temporal resolution. However, as a trade-off for high temporal resolution, its spatial resolution is limited, resulting in inferior reconstruction details. To address this issue, this paper develops a network (SpikeSR-Net) to super-resolve a high-resolution image sequence from the low-resolution binary spike streams. SpikeSR-Net is designed based on the observation model of spike camera and exploits both the merits of model-based and learning-based methods. To deal with the limited representation capacity of binary data, a pixel-adaptive spike encoder is proposed to convert spikes to latent representation to infer clues on intensity and motion. Then, a motion-aligned super resolver is employed to exploit long-term correlation, so that the dense sampling in temporal domain can be exploited to enhance the spatial resolution without introducing motion blur. Experimental results show that SpikeSR-Net is promising in super-resolving higher-quality images for spike camera.

The recently invented spike camera can capture high-speed motion in dynamic scenes by accumulating incoming photons continuously and firing spikes at very high temporal resolution. This paper addresses the demosaicing problem in spike camera color imaging. Specifically, we propose the 3D residual interpolation (3DRI) method to convert raw spike frames to color image frames. Due to the Poisson effect of photon arrivals and the quantization effect of spike readout, the instantaneous intensity recovered from the spike stream may suffer from undesired noise. To handle the noise, we estimate the missing color pixels along motion trajectories to exploit the temporal correlation among neighboring frames. In addition, by utilizing the color channels correlation, we design a residual-based demosaicing pipeline that uses the green pixels to guide the estimation of the red or blue missing pixels. Experimental results demonstrate our proposed 3DRI can produce color images from spike streams, achieving a good objective and perceptual quality for high-motion scenes.

Spike camera is an emerging bio-inspired vision sensor with ultra-high temporal resolution. It records scenes by accumulating photons and outputting continuous binary spike streams. Optical flow is a key task for spike cameras and their applications. A previous attempt has been made for spike-based optical flow. However, the previous work only focuses on motion between two moments, and it uses graphics-based data for training, whose generalization is limited. In this paper, we propose a tailored network, Spike2Flow that extracts information from binary spikes with temporal-spatial representation based on the differential of spike firing time and spatial information aggregation. The network utilizes continuous motion clues through joint correlation decoding. Besides, a new dataset with real-world scenes is proposed for better generalization. Experimental results show that our approach achieves state-of-the-art performance on existing synthetic datasets and real data captured by spike cameras. The source code and dataset are available at \url{https://github.com/ruizhao26/Spike2Flow}.

Benefiting from the high temporal resolution, the spike camera shows promising potential in capturing high-speed scenes via accumulating luminance intensity and firing spikes. Although the spike camera compared to the high-speed camera is quite cost-effective, its performance in capturing low-light scenes is poor. Specifically, it takes more time for the spike camera to accumulate enough light signal for firing a spike in low-light scenes, while the scenes may have already changed because of the high-speed motion. There may be no effective spikes for a long time due to the insufficient incident light, and the signal-to-noise ratio of spike streams in low-light scenes is unsatisfactory. Thus, it's easy to introduce noise and motion blur while reconstructing, especially in rapidly changing scenes. To address this issue, we propose a low-light scene reconstruction method for the spike camera. In particular, we first develop a Brightness-Adaptive Light Inference (BALI) method to preliminarily reconstruct the low-light scene according to the brightness, which utilizes both the spike interval and the spike number. Considering the motion, we then estimate optical flow and filter the preliminary restored frames iteratively to handle the noise via temporal correlation. After that, there is still some noise, and we further handle it by a spatial filter according to the brightness. As a result, we restore a clear image based on both temporal and spatial correlation. The experimental results demonstrate that our method achieves good visual quality in low-light scene reconstruction.

The recently invented retina-inspired spike camera has shown great potential for capturing dynamic scenes. Different from the conventional digital cameras that compact the photoelectric information within the exposure interval into a single snapshot, the spike camera produces a continuous spike stream to record the dynamic light intensity variation process. For spike cameras, image reconstruction remains an important and challenging issue. To this end, this paper develops a spike-to-image neural network (Spk2ImgNet) to reconstruct the dynamic scene from the continuous spike stream. In particular, to handle the challenges brought by both noise and high-speed motion, we propose a hierarchical architecture to exploit the temporal correlation of the spike stream progressively. Firstly, a spatially adaptive light inference subnet is proposed to exploit the local temporal correlation, producing basic light intensity estimates of different moments. Then, a pyramid deformable alignment is utilized to align the intermediate features such that the feature fusion module can exploit the long-term temporal correlation, while avoiding undesired motion blur. In addition, to train the network, we simulate the working mechanism of spike camera to generate a large-scale spike dataset composed of spike streams and corresponding ground truth images. Experimental results demonstrate that the proposed network evidently outperforms the state-of-the-art spike camera reconstruction methods.

Recently, a novel retina-inspired camera, namely spike camera, has shown great potential for recording high-speed dynamic scenes. Unlike conventional digital cameras that compact the visual information within an exposure interval into a single snapshot, the spike camera continuously outputs binary spike streams to record the dynamic scenes, yielding a very high temporal resolution. Most of the existing reconstruction methods for spike camera focus on reconstructing images with the same resolution as spike camera. However, as a trade-off of high temporal resolution, the spatial resolution of spike camera is limited, resulting in inferior details of the reconstruction. To address this issue, we develop a spike camera super-resolution framework, aiming to super resolve high-resolution intensity images from the low-resolution binary spike streams. Due to the relative motion between the camera and the objects to capture, the spikes fired by the same sensor pixel no longer describes the same points in the external scene. In this paper, we exploit the relative motion and derive the relationship between light intensity and each spike, so as to recover the external scene with both high temporal and high spatial resolution. Experimental results demonstrate that the proposed method can reconstruct pleasant high-resolution images from low- resolution spike streams.

Worker recruitment in mobile crowdsensing systems aims to recruit the most suitable users to perform tasks with high quality and in real-time. Many worker recruitment or task matching mechanisms have been proposed, especially for crowdsourcing platforms, where content information of tasks from the implicit feedback of workers' attendance is extensively exploited to help workers find preferred tasks efficiently. Different from traditional crowdsourcing systems, tasks in mobile crowdsensing systems are usually time-sensitive and location-dependent which also play a crucial role in worker recruitment. However, these context information have not been effectively explored for user recruitment in mobile crowdsensing systems. In this paper, we propose a novel personalized task-oriented worker recruitment mechanism for mobile crowdsensing systems based on a careful characterization of workers' preference. In particular, we fully exploit the content information (e.g., task category, task description) together with the context information (e.g., task time, task location) from the implicit feedback of workers' attendance to accurately model workers' preference on tasks. Moreover, we regard the task-worker fitness prediction as a binary classification problem and utilize the Logit model to integrate the heterogeneous factors into a single framework to predict the matching probability of each task-worker pair. Finally, the workers with the highest matching probability are recruited proactively for each new task. Extensive experiments on real-world datasets demonstrate that the proposed mechanism achieves better performance than the benchmarks.

A new retina-inspired bionic spike camera has recently shown great potential for capturing high speed movements. Unlike conventional cameras with a fixed low sampling rate, retina-inspired spike camera can well record fast-moving scenes by continuously accumulating luminance intensity and firing spikes. To restore the captured high-speed motion scenes from spike data, several reconstruction methods have been proposed. A typical method utilizes two neighbouring spikes to infer the instantaneous luminance intensity. Although high temporal resolution imaging can be achieved, the signal to noise ratio (SNR) of reconstructions is generally unsatisfactory. For improving the SNR, some methods propose to average the spikes in big time window. However, the reconstructions may suffer from undesired motion blur, especially when there are objects moving very fast in scenes. To address this issue, we develop a new image reconstruction approach for potential retina-inspired spike camera to recover high-speed motion scenes. Specially, we take the motion of objects into consideration and exploit optical flow to align the scenes of different moments. After motion alignment, a filtering along motion trajectory can be employed to the signals to take the advantage of temporal correlations while not introducing undesired motion blur. Experimental results demonstrate that our proposed method achieves better visual quality than previous reconstruction schemes.

With the development of emerging computer vision applications, there is an increasing demand for capturing the scenes with high-speed motion. Recently, a novel retina-inspired spike camera has shown great potential for recording the dynamic scenes at high temporal resolution. Different from the conventional digital cameras that capture the visual scene by a single snapshot, the spike camera monitors the incoming light persistently, with each pixel producing a continuous stream of spikes. Recovering the motion process from the spike data sequence is an important problem to study for the spike camera, as it is the foundation of many other tasks, such as image reconstruction, object tracking and object detection. In this paper, we carefully analyze the characteristics of spike data and develop a motion estimation algorithm to recover the continuous high-speed motion process from the spike camera data sequence. Based on the assumption that the spike intervals passed by the same motion trajectories usually have the similar spike densities, we establish a data term constraint to model the temporal consistency of spike intervals. In addition, we integrate a local smoothness constraint with the proposed data term constraint to further improve the estimation accuracy. Experimental results demonstrate that our proposed algorithm can recover high-speed motion process from the captured spike data, and the recovered motion information is beneficial for i mage reconstruction.

Incentive mechanisms have been commonly proposed to encourage people to participate in mobile crowdsensing (MCS). However, most of them set unchangeable rewards for sensing tasks, while the inherent inequality and on-demand feature of sensing tasks have been long ignored, especially for location-dependent sensing tasks (LDSTs). In this paper, we focus on location-dependent MCS systems and propose a demand-driven dynamic incentive mechanism that dynamically changes the rewards of sensing tasks at each sensing round in an on-demand way to balance their popularity. A demand indicator is introduced to characterize the demand of each sensing task by considering its deadline, completing progress, and number of potential participants. At each sensing round, we use the Analytic Hierarchy Process (AHP) to calculate the relative demands of all sensing tasks and then determine their rewards accordingly. Moreover, we consider two task selection problem with participatory users and opportunistic users, respectively, and prove that both of them are NP-hard. We propose an optimal dynamic programming based solution for participatory scenario and an optimal backtracking based solution for opportunistic scenario to help each user select tasks while maximizing its profit. Extensive experiments show that the demand-driven dynamic incentive mechanism outperforms existing incentive mechanisms.

Due to the fast inference and excellent learning capability, deep learning has become an effective means for image denoising and attracted considerable attention recently. However, for the images with rich textures and structures, the performance of deep learning approaches is still unsatisfactory. To address this issue, we develop a new convolutional neural network (CNN) for subband image denoising and name it SDCNN. In the proposed approach, we first decompose images into transform domain and denoise the coefficients of various subbands. By incorporating frequency information with spatial context, SDCNN is more effective in recovering image details. In particular, the introduced procedure of subband transform also plays the role of downsampling and enlarges the receptive field without increasing depth or sacrificing efficiency of network. Experimental results show that the SDCNN achieves promising results in terms of both objective and subjective performance.

With the rich sensing capacity and ubiquitous usage of smartphones, crowdsensing leveraging the power of the crowd of mobile users has become an effective technique to collect data for various sensing applications. Many incentive mechanisms have been proposed to encourage people to participate in crowdsensing. However, most of them set unchangeable rewards for sensing tasks, while the inherent inequality and on-demand feature of sensing tasks have been long ignored, especially for location-dependent sensing tasks. In this paper, we focus on location-dependent crowdsensing systems and propose a demand-based dynamic incentive mechanism that dynamically changes the rewards of sensing tasks at each sensing round in an on-demand way to balance their popularity. A demand indicator is introduced to characterize the demand of each sensing task by considering its deadline, completing progress, and number of potential participants. At each sensing round, we use the Analytic Hierarchy Process to calculate the relative demands of all sensing tasks and then determine their rewards accordingly. Moreover, we prove that the distributed task selection problem with time budget is NP-hard. We propose an optimal dynamic programming based solution and a greedy solution to help each user select tasks while maximizing its profit. Extensive experiments show that the demand-based dynamic incentive mechanism outperforms existing incentive mechanisms.

Recently, the SoftCast scheme has shown great potential for robust image/video communication in wireless scenarios, where the channel quality may fluctuate drastically and unpredictably. However, the analog-like transmission in Soft-Cast is not always efficient in terms of power usage, compared with digital approaches. In this paper, we propose a layered image/video SoftCast scheme, in which a coarse version of the image is transmitted by a base layer in digital way, while the residual details are delivered by an enhancement layer in pseudo-analog way. In order to achieve optimal overall transmission performance, this paper studies the problem of optimal bit rate selection for the base layer, by building a rate-residual model based on the relationship between the base layer bit rate and the residual signal spectrum. Experimental results show that the proposed scheme can improve the performance of original scheme remarkably, while still preserving the smooth quality degradation characteristic of SoftCast.

Recently, a scheme called SoftCast has shown great potential for wireless image/video communication in the scenarios where the channel quality may fluctuate drastically and unpredictably. The transmission is lossy in nature, with its transmission power allocated among coefficients unequally to minimize the distortion. One problem is that its performance is optimized using mean square errors (MSE) as the quality metric, which is known for not matching the perception of human eyes. Inspired by the researches in image quality assessment, this paper proposes a power allocation and optimization scheme that minimizes the perceptual distortion of reconstruction image. In particular, we establish a perception model to evaluate the perceptual importance of different transform coefficients, based on the structure similarity (SSIM) image quality metric. Experimental results show that the proposed scheme can improve the perceptual performance of the original SoftCast scheme.