This letter is dedicated to improving the quality of video sequences compressed by the H.265/HEVC standard. We propose to consider this problem as a signal recovery from incomplete measurements taken in the HEVC transform domain. The recovery could be obtained via l1-minimization using the Iterative Shrinkage-Thresholding Algorithm (ISTA) well known in compressive sensing (CS) framework. However, in case of HEVC the ISTA updating step cannot be performed directly via matrix multiplications, since the sensing is performed using high complex hybrid intra- and motion-compensated prediction in pixel domain, and the frame sensing matrix depends on the current and corresponding reference frames along with the encoder compression profile. In order to overcome these limitations, in this letter, we first propose to modify the HEVC decoder so that it also obtains the prediction values for each pixel taking into account all the coding modes within the input bitstream. Second, we propose to modify the ISTA updating step by introducing encoding and decoding operators applied instead of the matrix multiplication by sensing matrix and its transpose, respectively. These operators use the obtained prediction values, as well as prediction mode, motion vectors, quantization step, and transform type extracted for each coding unit from the input bitstream in order to replicate the encoding and the decoding process except the entropy coding. Herewith, the ISTA thre sholding stage is performed by an image or video enhancement neural network. Experimental results show that the proposed approach provides up to 1 dB improvement in Peak Signal-to-Noise Ratio (PSNR) compared to the state-of-the-art approaches such as Recursive Fusion and Deformable Spatiotemporal Attention (RFDA), Spatio-Temporal Detail Information Retrieval (STDR) and Coding Priors-Guided Aggregation (CPGA).

We propose an adaptive binary implementation of the range version of Asymmetric Numeral Systems (rANS). First, as rANS encoder processes symbols in reverse order, we estimate the probabilities in forward order, store them into the encoder memory, and use them during the reverse encoding. It guarantees that both the encoder and the decoder have exactly the same probability estimation for each symbol. Second, we show how this approach can be implemented using probability estimation via Virtual Sliding Window (VSW). Finally, we demonstrate that comparing to rANS with a static model, the proposed adaptive binary rANS provides better compression performance having similar decoding complexity.

In skeletal representation, intra-frame differences between body joints, as well as inter-frame dynamics between body skeletons contain discriminative information for action recognition. Conventional methods for modeling human skeleton sequences generally depend on motion trajectory and body joint dependency information, thus lacking the ability to identify the inherent differences of human skeletons. In this paper, we propose a spatio-temporal difference descriptor based on a directional convolution architecture that enables us to learn the spatio-temporal differences and contextual dependencies between different body joints simultaneously. The overall model is built on a deep symmetric positive definite (SPD) metric learning architecture designed to learn discriminative manifold features with the well-designed non-linear mapping operation. Experiments on several action datasets show that our proposed method achieves up to 3% accuracy improvement over state-of-the-art methods.

In this paper we continue to study the problem of efficient video coding based on compressive sensing. In our previous work we proposed CS-JPEG video codec, where dragon noiselet transform is applied to a differential frame computed as a residual between an input frame and its downsampled and upsampled version. The downsampled version of the frame is compressed via JPEG baseline, while the noiselet coefficients are subsampled, quantized, entropy encoded and embedded into the application header of the JPEG file. At the decoder side, an iterative soft thresholding algorithm (ISTA) with video block-matching and 3D filtering (VBM3D) used as the thresholding operator is applied. In this paper, we first introduce a fast ISTA based on simple temporal and spatial transforms with parameters which are randomly selected at each iteration, and show that the proposed approach reduces the recovery time by around 300 times. Second, we consider the codec parameters selection providing the best reconstruction quality for a given bit rate. Finally, we evaluate the codec performance on HD video sequences recorded by real-life street surveillance systems without camera motion and show that CS-JPEG with the fast decoding and the selected parameters provides 54.3% and 22.4% bit rate savings, in average, comparing with the fast implementations of H.264/AVC Intra or H.265/HEVC Intra, respectively.

Being a class of effective feature descriptors for action recognition, action representations based on skeleton sequences have yielded excellent recognition results. Most methods used to construct these action representations are based on the information from all the joint positions in actions. Unfortunately, some joints in the actions do not improve the accuracy of action recognition, and may even cause unnecessary inter-class errors. In this study, the authors propose a new method for action recognition by selecting active joints which are closely related to the movement of the body as the first step. Further, a skeleton is characterized as a set of its active-joint positions, and the set can be mapped to a point on pre-shape space to filter out the scale and translation variability. Then, a skeleton sequence (an action) can be regarded as points on the space. Because the timing-sequence relationship between skeletons is very valuable for action recognition, a tensor-based linear dynamical system (tLDS) is employed to model the temporal information of the action. To avoid using a finite-order sequence to estimate the infinite-order feature descriptor of a tLDS, the descriptor is mapped to a point on an infinite Grassmannian composed of the extended observability subspaces. The action is classified using sparse coding and dictionary learning (SCDL) on the infinite Grassmannian. Experimental results demonstrate that the recognition accuracies of the proposed method outperform state-of-the-art ones on four different action datasets.

This paper studies the problem of video coding based on compressive sensing (CS) framework. We propose a new approach based on global sensing model applied to a residual frame computed as a difference between an input frame and its downsampled and upsampled version. The downsampled version of the frame is compressed via JPEG baseline making the codec compatible with JPEG. The sensed measurements are quantized, subsampled using a predefined look-up table, and encoded via context-adaptive binary range coder. The resulting bit stream is divided into network abstract layer packets which are embedded into an application part of the JPEG header. Experimental results show that the proposed approach outperforms existing algorithms based on CS in rate-distortion performance and generates a bit stream, which is significantly more robust to packet losses compared to conventional codecs.

This paper studies the problem of decoding video sequences compressed by Motion JPEG (M-JPEG) at the best possible perceived video quality. We consider decoding of M-JPEG video as signal recovery from incomplete measurements known in compressive sensing. We take all quantized nonzero Discrete Cosine Transform (DCT) coefficients as measurements and the remaining zero coefficients as data that should be recovered. The output video is reconstructed via iterative thresholding algorithm, where Video Block Matching and 4-D filtering (VBM4D) is used as thresholding operator. To reduce non-linearities in the measurements caused by the quantization in JPEG, we propose to apply spatio-temporal pre-filtering before measurements calculation and recovery. Since temporal inconsistencies of the residual coding artifacts lead to strong flickering in recovered video, we also propose to apply motion-compensated deflickering filter as a post-filter. Experimental results show that the proposed approach provides 0.44-0.51 dB average improvement in Peak Signal to Noise Ratio (PSNR), as well as lower flickering level compared to the state-of-the-art method based on Coefficient Graph Laplacians (COGL). We have also conducted a subjective comparison study, indicating that the proposed approach outperforms state-of-the-art methods in terms of subjective video quality.

LIDAR (light detection and ranging) based real‐time 3D perception is crucial for applications such as autonomous driving. However, most of the convolutional neural network (CNN) based methods are time‐consuming and computation‐intensive. These drawbacks are mainly attributed to the highly variable density of LIDAR point cloud and the complexity of their pipelines. To find a balance between speed and accuracy for 3D object detection from LIDAR, authors propose RTL3D, a computationally efficient Real‐time LIDAR‐based 3D detector. In RTL3D, an effective voxel‐wise feature representation is utilised to organise unstructured point cloud. By employing a sparse feature learning network (SFLN) on voxelised 3D data, RTL3D exploits the sparsity of point cloud and down‐samples 3D data into 2D. Basing on the generated 2D feature map, an optimised dense detection network (DDN) is applied to regress the oriented bounding box without relying on any predefined anchor boxes. The authors also introduce an incremental data augmentation approach which greatly improves the performance of RTL3D. Empirical experiments on public KITTI benchmark demonstrate that RTL3D achieves a competitive performance with state‐of‐the‐art works on 3D detection task. Owning to the simplicity of its single‐stage and anchor‐free design, RTL3D has a real‐time inference speed of 40 FPS.

The authors consider the problem of compressive sensed video recovery via iterative thresholding algorithm. Traditionally, it is assumed that some fixed sparsifying transform is applied at each iteration of the algorithm. In order to improve the recovery performance, at each iteration the thresholding could be applied for different transforms in order to obtain several estimates for each pixel. Then the resulting pixel value is computed based on obtained estimates using simple averaging. However, calculation of the estimates leads to significant increase in reconstruction complexity. Therefore, the authors propose a heuristic approach, where at each iteration only one transform is randomly selected from some set of transforms. First, they present simple examples, when block‐based 2D discrete cosine transform is used as the sparsifying transform, and show that the random selection of the block size at each iteration significantly outperforms the case when fixed block size is used. Second, building on these simple examples, they apply the proposed approach when video block‐matching and 3D filtering (VBM3D) is used for the thresholding and show that the random transform selection within VBM3D allows to improve the recovery performance as compared with the recovery based on VBM3D with fixed transform.

An adaptive multialphabet arithmetic coding (AC) based on a generalized virtual sliding window (GVSW) is a novel multiplication-free AC scheme which can achieve better compression performance than existing implementations based on binarization and multiplication-free adaptive binary AC. In this brief, we propose an efficient hardware implementation of GVSW. We design a parallel update process of sliding windows, which can efficiently eliminate the sequentiality in the probability estimation procedure and introduce a partial unrolling for parallel execution of the renormalization procedure according to the statistical properties of an encoded data. The proposed architecture achieves an average throughput of 116.7 Mb/s for the Xilinx Kintex-7 KC705 platform.

In this letter, we present a novel compression algorithm of infrared video sequences captured during drone inspections based on iterative aerial map construction. In our approach, we first apply a stitching algorithm to construct a map of an inspected area assuming that a drone is flying at the same altitude by trajectory close to meander, so that each frame can have a partial overlap with other frame captured much earlier or later. Then, we extract position and rotation angle within the map for each frame and use them as a side information for the video coding. In order to compress an input video sequence, we utilize a multi-view H.265/HEVC with two views. First view is a virtual view generated utilizing the decoded frames of the second view and the side information, whereas the input video is considered as the second view, which is encoded utilizing the virtual view as a reference for the inter-view prediction. The proposed approach has two main benefits. First, the aerial map is generated during decoding utilizing the side information, i.e., the map is not embedded into a bit stream. Second, the inter-view prediction allows to exploit an additional redundancy, which is typical for a drone video. Experimental results show that the proposed algorithm provides 1.4%-2.4% bit rate savings comparing to H.265/HEVC. The maximum possible bit rate savings are estimated from 15.5% to 18.9% assuming that the drone is repeatedly flying many times at exactly the same trajectory.

In this paper, we present a novel drone high dynamic range infrared video coding algorithm based on aerial map prediction. First, at the encoder side we accumulate input frames in a buffer and use them to build an aerial map. Then we apply global motion estimation to extract the most similar frame from the aerial map and use it as an extra reference frame in the list of reference frames of H.265/HEVC video coding standard. The map is compressed by H.265/HEVC Intra and included into the overall bit stream. The global motion estimation may reflect the camera rotation which is typical for UAV video sequences. As a result, the overall coding performance is improved. Experimental results show that for a test video with camera rotation the proposed algorithm provides 3-35% bit rate savings comparing to the H.265/HEVC.

Compressive sensing theory asserts that, under certain conditions, a high dimensional but compressible signal can be recovered from a small number of random linear projections by utilizing computationally efficient algorithms. The a priori knowledge of the basis in which the signal of interest is sparse is the key assumption utilized by such algorithms. However, the basis in which the signal is the sparsest is unknown for many natural signals of interest. Instead there may exist multiple bases which lead to a compressible representation of the signal: e.g., an image is compressible in different wavelet transforms. We show that a significant performance improvement can be achieved by utilizing multiple estimates of the signal using sparsifying bases in the context of signal reconstruction from compressive samples. Further, we derive a customized interior-point method to jointly obtain multiple estimates of a 2-D signal (image) from compressive measurements utilizing multiple sparsifying bases as well as the fact that the images usually have a sparse gradient.

We propose a low-complexity High Dynamic Range (HDR) infrared image (IR) coding algorithm assuming the typical case of IR images with an active range of more than 8 bit depth, but less than 16 bit depth. First, we separate an input image into base and residual images with maximum 8 bit depth each. Then we compress each image by a JPEG baseline encoder and include the residual image bit stream into the application part of JPEG header of the base image. As a result, the base image can be reconstructed by JPEG baseline decoder. If the JPEG bit stream size of the residual image is higher than the raw data size, then we include the raw residual image instead. If the residual image contains only zero values or the quality factor for it is 0 then we do not include the residual image into the header. Experimental results show that compared with JPEG-XT Part 6 with `global Reinhard' tone-mapping, the proposed approach has lower complexity and similar rate-distortion performance on IR test images.

We propose a novel efficient multialphabet multiplication-free adaptive arithmetic coder. First, we generalize probability estimation via virtual sliding window for the multialphabet case and show that it does not require multiplications and provides a tradeoff between the probability adaptation speed and the precision of the probability estimation. Second, we show how the generalized virtual sliding window can be used to eliminate multiplications and divisions. Finally, we demonstrate that the proposed arithmetic coder provides better compression performance than existing implementations based on state-of-the-art multiplication-free binary arithmetic coders.

"View-plus-depth" is a popular 3D image representation format, in which the color 2D image is augmented with a gray-scale image representing the scene depth map aligned with the color pixels. In this paper, we propose a novel depth map compression method aimed at finding an optimal spatial depth scale and down-sampling (sparsifying) the depth image over it. The down-sampled depth image is then compressed by a combination of a new arithmetic and a predictive coder. Our approach is motivated by the current achievements in multi-sensor 3D scene sensing where low-resolution depth map captured by time-of-flight sensors is successfully up-sampled and aligned with high resolution RGB images. In our approach, color image segmentation in terms of super-pixels is used for finding the optimal depth scale and corresponding down-sampling. In contrast to other segmentation methods, it results in an isotropic and balanced low-resolution depth image, which is easily compressible. A bilateral regularizer is used for reconstructing the original-size depth map out of the low-resolution one and for splitting and predictive coding of segments with high reconstruction error. The scheme compares favorably with other methods for depth map compression.

In this paper, we propose a new hardware-efficient adaptive binary range coder (ABRC) and its very-large-scale integration (VLSI) architecture. To achieve this, we follow an approach that allows to reduce the bit capacity of the multiplication needed in the interval division part and shows how to avoid the need to use a loop in the renormalization part of ABRC. The probability estimation in the proposed ABRC is based on a lookup table free virtual sliding window. To obtain a higher compression performance, we propose a new adaptive window size selection algorithm. In comparison with an ABRC with a single window, the proposed system provides a faster probability adaptation at the initial encoding/decoding stage, and more accurate probability estimation for very low entropy binary sources. We show that the VLSI architecture of the proposed ABRC attains a throughput of 105.92 MSymbols/s on the FPGA platform, and consumes 18.15 mW for the dynamic part power. In comparison with the state-of-the-art MQ-coder (used in JPEG2000 standard) and the M-coder (used in H.264/Advanced Video Coding and H.265/High Efficiency Video Coding standards), the proposed ABRC architecture provides comparable throughput, reduced memory, and power consumption. Experimental results obtained for a wavelet video codec with JPEG2000-like bit-plane entropy coder show that the proposed ABRC allows to reduce the bit rate by 0.8%-8% in comparison with the MQ-coder and from 1.0%-24.2% in comparison with the M-coder.

Overtaking on rural roads may cause severe accidents when oncoming traffic is detected by a driver too late, or its speed is underestimated. Recently proposed cooperative overtaking assistance systems are based on real-time video transmission, where a video stream captured with a camera installed at the windshield of a vehicle is compressed, broadcast through the wireless channel, and displayed to the drivers of vehicles driving behind. In such a system, it is of ultimate importance to deliver video information about the opposite lane with low end-to-end latency and good visual quality. In this paper, we propose reallocating the wireless channel resources in favor of the part of the captured video frame containing the image of the oncoming vehicle. To achieve this goal, we apply automotive radar for oncoming vehicle detection, and we use the image of this vehicle as a region-of-interest (ROI) for the video rate control. We present the theoretical framework, which describes the basics of such an approach and can serve as a useful guideline for the future practical implementation of the overtaking assistance systems. The benefits of our proposal are demonstrated in relation to the practical scenario of H.264/Advance Video Coding (AVC), IEEE 802.11p/Wireless Access for Vehicular Environments (WAVE) intervehicle communication standards, and currently used automotive radars.

This paper is dedicated to fast video coding based on three-dimensional discrete wavelet transform. First, we propose a novel low-complexity bit-plane entropy coding of wavelet subbands based on Levenstein zero-run coder for low entropy contexts and adaptive binary range coder for other contexts. Second, we propose a rate-distortion efficient criterion for skipping 2-D wavelet transforms and entropy encoding based on parent-child subband tree. Finally, we propose one pass rate control which uses virtual buffer concept for adaptive Lagrange multiplier selection. Simulations results show that the proposed video codec has a much lower computational complexity (from 2 to 6 times) for the same quality level compared to the H.264/AVC standard in the low complexity mode.

In this paper we propose a novel efficient adaptive binary arithmetic coder which is multiplication-free and requires no look-up tables. To achieve this, we combine the probability estimation based on a virtual sliding window with the approximation of multiplication and the use of simple operations to calculate the next approximation after the encoding of each binary symbol. We show that the proposed algorithm is faster and provides a better compression efficiency compared to the M-coder in the CABAC entropy coding scheme of the H.264/AVC video coding standard.

Overtaking on rural roads often becomes dangerous when oncoming traffic is detected by the driver too late or its speed is underestimated. Up-to-date proposals for cooperative overtaking assistance systems rely either on the beaconing of status messages or on real-time video transmission. The beaconing-based system provides limited information to the driver about the upcoming traffic in the form of warnings. Introduction of the video-based system can extend the amount of driver information, thus, increasing the reliability of the assistance. However, when two platoons of vehicles are meeting each other, the quality of the video information can undergo significant degradation since a common multiple access communication channel is used. In this paper we demonstrate that the performance of a video-based overtaking assistant can be significantly improved if codec channel adaptation is undertaken by exploiting the information from the beacons about any forthcoming increase in the load of the multiple access channel used. The benefits of our approach are demonstrated in relation to the practical scenario of H.264/AVC video coding and IEEE 802.11p/WAVE inter-vehicle communication standards.

A high-throughput memory-efficient arithmetic coder architecture for the set partitioning in hierarchical trees (SPIHT) image compression is proposed based on a simple context model in this paper. The architecture benefits from various optimizations performed at different levels of arithmetic coding from higher algorithm abstraction to lower circuits' implementations. First, the complex context model used by software is mitigated by designing a simple context model, which just uses the brother nodes' states in the coding zerotree of SPIHT to form context symbols for the arithmetic coding. The simple context model results in a regular access pattern during reading the wavelet transform coefficients, which is convenient to the hardware implementation, but at a cost of slight performance loss. Second, in order to avoid rescanning the wavelet transform coefficients, a breadth first search SPIHT without lists algorithm is used instead of SPIHT with lists algorithm. Especially, the coding bit-planes of each zero tree are processed in parallel. Third, an out-of-order execution mechanism for different types of context is proposed that can allocate the context symbol to the idle arithmetic coding core with a different order that of the input. For the balance of the input rate of the wavelet coefficients, eight arithmetic coders are replicated in the compression system. And in one arithmetic coder, there exists four cores to process different contexts. Fourth, several dedicated circuits are designed to further improve the throughput of the architecture. The common bit detection (CBD) circuit is used for unrolling the renormalization stage of the arithmetic coding. The carry look-ahead adder (CLA) and fast multiplier-divider are also employed to shorten the critical path in the architecture. Moreover, an adaptive clock switch mechanism can stop some invalid bit-planes' clock for the power saving purpose according to the input images. Experimental results demonstrate that the proposed architecture attains a throughput of 902.464 Mb/s at its maximum and achieves savings of 20.08% in power consumption over full bit-planes coding scheme based on field-programmable gate arrays (FPGAs).

This paper disscuses latency-constrained video transmission over high-speed wireless personal area networks. Low-power single-layer video compression is proposed as an alternative to others video processing approaches. End-to-end distortion and end-to end latency in video transmission system are analyzed. A near-optimal video source rate control based on MINMAX quality criteria is introduced. Practical results for video encoder based on H.264/AVC standard are also given.