Efficient video representation models are critical for many video analysis and processing tasks. In this paper, we present a framework based on the concept of finding the sparsest solution to model video frames. To model the spatio-temporal information, frames from one scene are decomposed into two components: (i) a common frame, which describes the visual information common to all the frames in the scene/segment, and (ii) a set of innovative frames, which depicts the dynamic behaviour of the scene. The proposed approach exploits and builds on recent results in the field of compressed sensing to jointly estimate the common frame and the innovative frames for each video segment. We refer to the proposed modeling framework by CIV (Common and Innovative Visuals). We show how the proposed model can be utilized to find scene change boundaries and extend CIV to videos from multiple scenes. Furthermore, the proposed model is robust to noise and can be used for various video processing applications without relying on motion estimation and detection or image segmentation. Results for object tracking, video editing (object removal, inpainting) and scene change detection are presented to demonstrate the efficiency and the performance of the proposed model.

Typical consumer digital cameras sense only one out of three color components per image pixel. The problem of demosaicing deals with interpolating those missing color components. In this paper, we present compressive demosaicing (CD), a framework for demosaicing natural images based on the theory of compressed sensing (CS). Given sensed samples of an image, CD employs a CS solver to find the sparse representation of that image under a fixed sparsifying dictionary Ψ. As opposed to state of the art CS-based demosaicing approaches, we consider a clear distinction between the interchannel (color) and interpixel correlations of natural images. Utilizing some well-known facts about the human visual system, those two types of correlations are utilized in a nonseparable format to construct the sparsifying transform Ψ. Our simulation results verify that CD performs better (both visually and in terms of PSNR) than leading demosaicing approaches when applied to the majority of standard test images.

In this paper, we explore the notion of using frames to project sensed colors within their inherently 3D space onto a larger number of color basis vectors. In particular, we develop a new frame design, Incoherent Color Frames (ICF), which can include an arbitrary number of incoherent color vectors. An ICF frame possesses key desired properties including the ability to sparsify colors in 3D and to decorrelate color channels utilizing a spatial-frequency selective strategy. We present a low complexity algorithm for constructing ICF frames targeted for the problem of image demosaicing. Our simulation results show that when incorporating the proposed ICF within a Compressive Demosaicing (CD) framework, significant visual improvements can be achieved when compared with traditional and Compressed Sesnsing-based demosaicing solutions.

The utility of Compressed Sensing (CS) for demosaicing of images captured using random panchromatic color filter arrays (CFA) has been investigated in [1]. Meanwhile, most camera manufacturers employ periodic CFAs such as the popular Bayer CFA. In this paper, we derive a CS-based solution to demosaicing images captured using the general class of periodic CFAs. It is well known that periodic CFAs can be designed to effectively separate luminance and chrominance frequency bands [2, 3]. We employ this ability to reduce artifacts associated with luminance-chrominance overlap at the solver side. We show that the modified compressive demo-saicing method coupled with the additional constraint that chrominance channels have smooth surfaces achieves further improved results for most periodic CFAs.

A typical consumer digital camera uses a Color Filter Array (CFA) to sense only one color component per image pixel. The original three-color image is reconstructed by interpolating the missing color components. This interpolation process (known as demosaicing) corresponds to solving an under-determined system of linear equations. In this paper, we show that by replacing the traditional CFA with a random panchromatic CFA, recent results in the emerging field of Compressed Sensing (CS) can be used to solve the demosaicing problem in a novel way. Specifically, during the image reconstruction process, we exploit the fact that the multi-dimensional color of each pixel has a compressible representation in a (possibly overcomplete) color system. While adhering to the “single color per pixel sensing” constraint at the sensing stage, during the reconstruction process we utilize the inter-pixel correlation by exploiting the compressible representation of the overall image in some sparsifying bases. Depending on the CFA, sparsifying bases and the color system, we form an underdetermined system of linear equations and find the sparsest solution for the color image by utilizing a CS solver. We illustrate that, for natural images, the proposed Compressive Demosaicing (CD) framework visually outperforms leading demosaicing methods in a consistent manner; in many cases it achieves clear visible improvements in a significant way.

We consider the problem of recovering a signal/image (x) with a k-sparse representation, from hybrid (complex and real), noiseless linear samples (y) using a mixture of complex-valued sparse and real-valued dense projections within a single matrix. The proposed Hybrid Compressed Sensing (HCS) employs the complex-sparse part of the projection matrix to divide the n-dimensional signal (x) into subsets. In turn, each subset of the signal (coefficients) is mapped onto a complex sample of the measurement vector (y). Under a worst-case scenario of such sparsity-induced mapping, when the number of complex sparse measurements is sufficiently large then this mapping leads to the isolation of a significant fraction of the k non-zero coefficients into different complex measurement samples from y. Using a simple property of complex numbers (namely complex phases) one can identify the isolated non-zeros of x. After reducing the effect of the identified non-zero coefficients from the compressive samples, we utilize the real-valued dense submatrix to form a full rank system of equations to recover the signal values in the remaining indices (that are not recovered by the sparse complex projection part). We show that the proposed hybrid approach can recover a k-sparse signal (with high probability) while requiring only m ≈ 3k3√(n/2k) real measurements (where each complex sample is counted as two real measurements). We also derive expressions for the optimal mix of complex-sparse and real-dense rows within an HCS projection matrix. Further, in a practical range of sparsity ratio (k/n) suitable for images, the hybrid approach outperforms even the most complex compressed sensing frameworks (namely basis pursuit with dense Gaussian matrices). The theoretical complexity of HCS is less than the complexity of solving a full-rank system of m linear equations. In practice, the complexity can be lower than this bound.

Leading compressed sensing (CS) methods require m = O (k log(n)) compressive samples to perfectly reconstruct a k-sparse signal x of size n using random projection matrices (e.g., Gaussian or random Fourier matrices). For a given m, perfect reconstruction usually requires high complexity methods, such as Basis Pursuit (BP), which has complexity O(n3). Meanwhile, low-complexity greedy algorithms do not achieve the same level of performance (as BP) in terms of the quality of the reconstructed signal for the same m. In this paper, we introduce a new CS framework, which we refer to as Randomness-in-Structured Ensemble (RISE) projection. RISE projection matrices enable compressive sampling of image coefficients from random locations within the k-sparse image vector while imposing small structured overlaps. We prove that RISE-based compressed sensing requires only m = ck samples (where c is not a function of n) to perfectly recover a k-sparse image signal. For the case of n ≤ O(k2), the complexity of our solver is O(nk) which is less than the complexity of the popular greedy algorithm Orthogonal Matching Pursuit (OMP). Moreover, in practice we only need m = 2k samples to reconstruct the signal. We present simulation results that demonstrate the RISE framework's ability to recover the original image with higher than 50 dB PSNR, whereas other leading approaches (such as BP) can achieve PSNR values around 30 dB only.