The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.

Glass-free compressive light field (CLF) display gains much attention due to their compatibility in holographic-like and three-dimensional (3D) demonstration. Opposite to other analogous devices, CLF display can provide binocular and motion parallaxes by stacking multiple liquid crystal screens without any extra accessories. It is possible to bring the immersive and accommodative experience upon a well-pleasing visual consequence. Conventionally, the excessive processing time impacts its practical value in commercial, along with the severe degradation of display brightness. Therefore, in this paper, we propose a learning-based factorization framework to promote the visual results and expedite the layer decomposition and display adaption. It utilizes the advantage of a dual-guided system and residual learning to implement pixel-wise information extraction and refinement. The experimental results illustrate the outperformance of our proposed method over the conventional iterative factorization. Furthermore, a three-layered CLF prototype has been assembled to verify the practicality of our method.

The knowledge of the bitrates and coding parameters is the prerequisite for implementing the rate control mechanism. Many previous works have been attempted to avoid the actual coding to obtain these factors by studying their regularity of correlation. However, these works only focus on the studies on the picture level or the coding unit (CU) level rather than on the sequence level. In this paper, we propose the YOCO (You Only Code Once) light-weight rate control model learning scheme, which can achieve the sequence-level rate control by managing the constant rate factor (CRF). It utilizes the unified information extracted from the compressed videos in the bitstream and pixel domains, leveraging the deep learning algorithm to learn the rate control model as these factors' algebraic relevance. For each video sequence, we can allocate the bitrates extremely approaching the target value with coding at the estimated CRF setting. We compare the application of pixel domain information on each rate control model (linear and the quadratic R-CRF models) to validate their effectiveness. The experimental results demonstrate the improvement of accuracy on the bitrate estimation in the different definitions, especially in the high ones.

Generally, adaptive bitrates for variable Internet bandwidths can be obtained through multi-pass coding. Referenceless prediction-based methods show practical benefits compared with multi-pass coding to avoid excessive computational resource consumption, especially in low-latency circumstances. However, most of them fail to predict precisely due to the complex inner structure of modern codecs. Therefore, to improve the fidelity of prediction, we propose a referenceless prediction-based R-QP modeling (PmR-QP) method to estimate bitrate by leveraging a deep learning algorithm with only one-pass coding. It refines the global rate-control paradigm in modern codecs on flexibility and applicability with few adjustments as possible. By exploring the potentials of bitstream and pixel features from the prerequisite of one-pass coding, it can reach the expectation of bitrate estimation in terms of precision. To be more specific, we first describe the R-QP relationship curve as a robust quadratic R-QP modeling function derived from the Cauchy-based distribution. Second, we simplify the modeling function by fastening one operational point of the relationship curve received from the coding process. Third, we learn the model parameters from bitstream and pixel features, named them hybrid referenceless features, comprising texture information, hierarchical coding structure, and selected modes in intra-prediction. Extensive experiments demonstrate the proposed method significantly decreases the proportion of samples' bitrate estimation error within 10% by 24.60% on average over the state-of-the-art.

Fall is one of the main reasons for body injuries among seniors. Traditional fall detection methods are mainly achieved by wearable and non-wearable techniques, which may cause skin discomfort or invasion of privacy to users. In this paper, we propose an automatic fall detection method with the assist of the mmWave radar signal to solve the aforementioned issues. The radar devices are capable to record the reflection from objects in both the spatial and temporal domain, which can be used to depict the activities of users with the support of a recurrent neural network (RNN) with long-short-term memory (LSTM) units. First, we employ the radar low-dimension embedding (RLDE) algorithm to preprocess the Range-angle reflection heatmap sequence converted from the raw radar signal for reducing the redundancy in the spatial domain. Then, the processed sequence is split into frames for inputting LSTM units one by one. Eventually, the output from the last LSTM unit is fed in a Softmax layer for classifying different activities. To validate the effectiveness of our proposed method, we construct a radar dataset with the assist of market radar module devices, to implement several experiments. The experimental results demonstrate that, compared to LSTM only and the widely used 3-D convolutional neural network (3-D CNN), combining RLDE and LSTM can achieve the best detection results with much less computational time consumption. In addition, we extend the proposed method to classify multiple human activities simultaneously and the satisfied performances are observed.

In this paper, we propose a two-pass encoding framework to handle the problem of sequence-level rate control. We consider the sequence-level encoding parameter constant rate factor (CRF) as the factor to be adjusted. The proposed framework mainly has two key contributions. First, we provide a second order model to characterize the relationship between the bitrate and CRF. The proposed second order model outperforms the traditional linear model significantly. Second, we adopt a shallow neural network to train the relationship between the content-dependent features with the second-order model parameters. The proposed neural network is quite simple but able to estimate the model parameters accurately. We implement the proposed algorithm under tensorflow. Experimental results show that our proposed method obviously outperforms the state-of-the-art method.

Current light field compression techniques lack robustness to handle both rate-distortion optimized motion compensation as well as latency during the encoding and decoding process. This paper focuses on a contribution approach that uses advanced frame prediction with affine and translational motion models and optimized view prediction structures. This method allows a significant compression performance gain over the current state of art of hierarchical temporal coding by 13.9%. The proposed method introduces an optimized encoding order that takes advantage of each group of pictures structure in order to leverage the dense perspective model of light field imagery. Both a global perspective model and a local affine model can be combined to show substantial distortion reduction at low processor costs. This contribution approach leads to an efficient and robust compression scheme for light field datasets.