Generative adversarial network (GAN)-based image compression approaches reconstruct images with highly realistic quality at low bit rates. However, currently there is no published GAN-based image compression approach that utilizes advanced GAN losses, such as the Wasserstein GAN with gradient penalty loss (WGAN-GP), to improve the quality of reconstructed images. Meanwhile, existing deep learning-based image compression approaches require extra convolution layers to estimate and constrain the entropy during training, which makes the network larger and may require extra bits to send information to the decoder. In this paper, we propose a new GAN for image compression with novel discriminator and generator loss functions and a simple entropy estimation approach. Our new loss functions outperform the current GAN loss for low bitrate image compression. Our entropy estimation approach does not require extra convolution layers but still works well to constrain the number of bits during training.

Future frame prediction is an emerging, yet challenging task in the deep learning field due to its inherent uncertainty and complex spatiotemporal dynamics. The state-of-the-art methods achieve significant accuracy at the expense of complex, computationally intensive deep neural networks, which makes it difficult to deploy in mobile devices. In the light of recent wide popularity of Green AI which aims for efficient environment friendly solutions alongside accuracy, we propose a lightweight model using 3D separable convolutions, which can predict future video frames with reduced model size and reasonable accuracy-complexity tradeoffs as compared to the state-of-the-art methods.

Deep learning has been successfully applied to image and video compression. Specifically, generative adversarial network (GAN) can compress images at low bit rates with sharp details and high perceptual quality. In this work, we propose a novel generative video compression (GVC) model with a transformer-based discriminator (TD), which learns non-local correlations within video frames to improve adversarial training. Besides, our GVC model incorporates a new loss to train the generator, which combines a base loss, a discriminator-dependent feature loss, and a perceptual loss. Experiments on HEVC test sequences demonstrate that the proposed GVC model provides superior performance at extremely low bit rates, compared to existing learned and traditional video coding schemes.

With the continuous improvement of computer vision technology, more and more image information is consumed by machines rather than humans. Image coding for machines (ICM) is to compress image data such that they can be more efficiently sent to the receiver side for machines to conduct visual analysis. A typical deep learning-based ICM structure contains one codec network which compresses and transmits images through the Internet and one semantic analysis task network such as image classification and object recognition. In the codec part, the side information is the hyper-prior or hierarchical layers of hyper-priors for the compression of image latent representations. In this paper, we propose a Side Information Driven Image Coding (SIIC) framework based on deep learning. It only compresses and transmits the side information to the receiver for image classification tasks. We obtain a top-l accuracy of 70.38% on the ImageNet1K dataset with 0.046 bits per pixel.

Compressed sensing is an effective solution for signal acquisition and signal reconstruction at a much lower rate than the Nyquist rate. Traditional methods, such as orthogonal matching pursuit and basis pursuit, for image compressed sensing reconstruction have unsatisfying reconstruction quality and long reconstruction time. Researchers now focus on neural network and deep learning methods for the better reconstruction of compressed-sensed signals at a very low sampling rate and a fast speed. However, current neural network approaches for image compressed sensing do not consider the similarities between images or within images, or the types of image blocks; thus, performing poorly in images with complex contents. In this paper, we develop a novel neural network framework that utilizes the similarities between image blocks through Gaussian-mixture models without recording the similarity information to achieve better reconstruction quality than the state-of-the-art neural network methods for block-level image compressed sensing.

Deep learning methods have been actively applied in moving object detection and achieved great performance. However, many existing models render superior detection accuracy at the cost of high computational complexity and slow inference speed, which hindered the application on mobile and embedded devices with limited computing resources. In this paper, we propose a two-branch 3D separable convolutional neural network named “F3DsCNN” for moving object detection. The network extracts both high-level global features and low-level detailed features. It achieves a fast inference speed of 120 frames per second, suitable for tasks that need to be carried out in a timely manner on a computationally limited platform with high accuracy.

Compressed sensing (CS) is a signal processing framework that effectively recovers a signal from a small number of samples. Traditional compressed sensing algorithms, such as basis pursuit (BP) and orthogonal matching pursuit (OMP) have several drawbacks, such as low reconstruction performance at small compressed sensing rates and high time complexity. Recently, researchers focus on deep learning to get compressed sensing matrix and reconstruction operations collectively. However, they failed to consider sparsity in their neural networks to compressed sensing recovery; thus, the reconstruction performances are still unsatisfied. In this paper, we use 2D-discrete cosine transform and 2D-discrete wavelet transform to impose sparsity of recovered signals to deep learning in video frame compressed sensing. We find the reconstruction performance is significantly enhanced.

Compressed sensing (CS) is the theory and practice of sub-Nyquist sampling of sparse signals of interest. Exact reconstruction may then be possible with much fewer than the Nyquist-required number of data. In this paper, we consider a multiview video system in which multiple cameras at different locations perform independent CS to simultaneously capture different views of a scene. At the decoder, we propose a disparity- and motion-compensated total variation minimization algorithm to jointly reconstruct the multiview video sequence. The experimental results show that the proposed joint reconstruction algorithm successfully exploits simultaneously intra-frame, inter-frame, and inter-view sparsity and significantly outperforms existing independent-view reconstruction, residue-view reconstruction, and motion-adaptive reconstruction algorithms.

We develop a novel joint-view Kalman filter for causal reconstruction of compressed-sensed multiview videos. Compressed-sensed multiview video frames are initially reconstructed individually via ℓ1-norm minimization. Then, ajoint-view state transition model is established for each pair of neighboring views using motion or motion-disparity field estimates. Experimental results demonstrate significantly improved reconstruction quality compared to conventional CS reconstruction and independent-view (single-view) motion-compensated Kalman filtering.

We consider the problem of foreground and background extraction from compressed-sensed (CS) surveillance videos that are captured by a static CS camera. We propose, for the first time in the literature, a principal component analysis (PCA) approach that computes directly in the CS domain the low-rank subspace of the background scene. Rather than computing the conventionalL2-norm-based principal components, which are simply the dominant left singular vectors of the CS-domain data matrix, we compute the principal components under anL1-norm maximization criterion. The background scene is then obtained by projecting the CS measurement vector onto theL1principal components followed by total-variation (TV) minimization image recovery. The proposedL1-norm procedure directly carries out low-rank background representation without reconstructing the video sequence and, at the same time, exhibits significant robustness against outliers in CS measurements compared toL2-norm PCA. An adaptive CS-L1-PCA method is also developed for low-latency video surveillance. Extensive experimental studies described in this paper illustrate and support the theoretical developments.

Compressed sensing (CS) is the theory and practice of sub-Nyquist sampling of sparse signals of interest. Perfect reconstruction may then be possible with much fewer than the Nyquist required number of data. In this paper, we consider a distributed multi-view imaging system where each camera at a different location performs independent compressed sensing acquisition of the target scene. At the decoder, we propose a disparity-compensated total-variation (TV) minimization algorithm to jointly reconstruct the multiple views. Experimental results show that the proposed joint decoding algorithm outperforms significantly independent-view decoding as well as disparity-compensated residue-view reconstruction algorithm.

We consider a variable block size compressed sensing (CS) framework for high efficiency depth map coding. In this context, quad-tree decomposition is performed on a depth image to differentiate irregular uniform and edge areas prior to CS acquisition. To exploit temporal correlation and enhance coding efficiency, such quad-tree based CS acquisition is further extended to inter-frame encoding, where block partitioning is performed independently on the I frame and each of the subsequent residual frames. At the decoder, pixel domain total-variation minimization is performed for high quality depth map reconstruction. Experiments presented herein illustrate and support these developments.

In recent years, compressed sensing (CS) has been used for compressing depth maps for the multi-view video plus depth. In these compression schemes, every block of the depth map is sampled with a fixed non-adaptive sensing matrix and the algorithms are generally incorporated into conventional codecs like H.264/AVC, resulting in high computational complexity both at the encoder and decoder. In this paper, we present a novel block-based CS codec for compressing depth maps that has two major features. First, an adaptive measurement rate allocation algorithm is introduced that computes the measurement rate for each compressively sensed block using ratedistortion optimization (RDO). Second, a simple block classification and frame-differencing module is utilized to reduce encoding complexity while maintaining good RD performance. Simulation results clearly show that the proposed codec has superior rate-distortion (RD) performance in comparison to H.264/AVC Baseline Profile (up to 3 dB gain) and an encoding time reduction of up to 97%.

Compressed sensing is the theory and practice of sub-Nyquist sampling of sparse signals of interest. Perfect reconstruction may then be possible with much fewer than the Nyquist required number of data. In this paper, in particular, we consider a video system where acquisition is carried out in the form of direct compressive sampling (CS) with no other form of sophisticated encoding. Therefore, the burden of quality video sequence reconstruction falls solely on the receiver side. We show that effective implicit motion estimation and decoding can be carried out at the receiver or decoder side via sparsity-aware recovery. The receiver performs sliding-window interframe decoding that adaptively estimates Karhunen–Loève bases from adjacent previously reconstructed frames to enhance the sparse representation of each video frame block, such that the overall reconstruction quality is improved at any given fixed CS rate. Experimental results included in this paper illustrate the presented developments.