In recent years, the crucial task of image compression has been addressed by end-to-end neural network methods. However, achieving fine-grained rate control in this new paradigm has presented challenges. In our previous work, we explored mismatches in rate estimation during target-rate-oriented training and proposed heuristics involving costly parameter searches as a solution. This work proposes a lightweight approach, which dynamically adapts loss parameters to mitigate rate estimation issues, ensuring precise target rate attainment. Inspired by Reinforcement Learning, our method exhibits performance comparable to preceding approaches on the Kodak dataset in terms of PSNR. Additionally, it reduces computational training costs.

In recent years image compression techniques based on deep learning have achieved great success and their performances are gradually reaching the methods crafted by experts, such as JPEG, WebP, and Better Portable Graphics (BPG). A technique that is fundamental for modern image and video codecs is intra prediction, which takes advantage of local redundancy to predict the pixels from previously encoded neighbors. In this paper, we use Convolutional Neural Networks (CNN) to develop a new intra-picture prediction mode. More specifically, we propose a multi-mode intra prediction approach that uses two CNN-based prediction modes and all intra modes previously implemented in the High Efficiency Video Coding (HEVC) standard. We also propose a bit allocation technique that increases the bitstream only if the reconstruction error is significantly reduced. Experimental results evince a significant and consistent performance increase compared to other approaches that use a similar backbone architecture, with 28% bitrate reduction compared to the baseline codec.

Due to the large size of a light field image, compressing and transmitting the entire data to a client before rendering any image for observation would incur a significant startup delay. In response, in interactive light field streaming (ILFS) a server synthesizes and transmits a new viewpoint image as a combination of sub-aperture images (SAIs) per user request. However, in so doing the client relies entirely on the server for reconstruction of every requested image. In this paper, we extend a previous proposal of progressive light field data transmission strategy, where the client can incrementally learn SAIs over time. Specifically, requested focal-point images are synthesized using carefully chosen weighted linear combinations of SAIs, so that recovery of SAIs amounts to inversion of a lower-triangular weight matrix-a matrix structure that enables SAI recovery without amplifying quantization noise due to lossy image coding. We design an objective function to encourage specific combinations of SAIs to increase rank of the lower-triangular weight matrix for fast SAI recovery. This new proposal reduces the size of the initial user's cache and the total number of transmitted images compared to our previous work. Experimental results show that our scheme can outperform ILFS by up to 70% in terms of BD-rate.

Light field cameras capture the emanated light from a scene. This type of images allows for changing point of views or focal points by processing the captured information. Recently, a Progressive Light Field Communication (PLFC) was proposed. PLFC addresses an interactive Light Field (LF) streaming framework, where a client requests a certain view or focal point and a server synthesizes and transmits each requested image as a linear combination of Sub-Aperture Images (SAI). The main idea of PLFC is that as the virtual views are transmitted, the client gradually learns information about the LF, so eventually the client may posses enough information to locally create the virtual view at the required quality, avoiding the transmission of a new image. In order to PLFC work, an optimization algorithm which selects the SAIs that are used to create a certain virtual view is requested. Here, we improve over the previous PLFC proposal by presenting a method that focuses on a refinement algorithm for SAI selection, using dynamic Quantization Parameter (QP) during encoding, using an automatic method to determine the Lagrangian multiplier during optimization and modifying how the initial required cache is created. These proposed changes in the algorithm produce significant gains. The results shows gains up to 85.8% on BD-rate compared to trivial LF transmissions, whereas they're up to 32.8% compared to previous PLFC.

This paper presents algorithms designed for one-dimensional (1-D) and 2-D surface electromyographic (S-EMG) signal compression. The 1-D approach is a wavelet transform based encoder applied to isometric and dynamic S-EMG signals. An adaptive estimation of the spectral shape is used to carry out dynamic bit allocation for vector quantization of transformed coefficients. Thus, an entropy coding is applied to minimize redundancy in quantized coefficient vector and to pack the data. In the 2-D approach algorithm, the isometric or dynamic S-EMG signal is properly segmented and arranged to build a 2-D representation. The high efficient video codec is used to encode the signal, using 16-bit-depth precision, all possible coding/prediction unit sizes, and all intra-coding modes. The encoders are evaluated with objective metrics, and a real signal data bank is used. Furthermore, performance comparisons are also shown in this paper, where the proposed methods have outperformed other efficient encoders reported in the literature.

Light field (LF) imaging captures multiple intensities and directions of light per pixel during acquisition in a 3D scene, so that novel images of different viewpoints or focal points can be synthesized. However, transmitting all LF data before viewer observation incurs a large startup delay. To avoid such delay, we propose a new interactive LF streaming framework, where a client periodically requests viewpoint images, and in response a server synthesizes and transmits each requested image as a carefully chosen sparse linear combination of sub-aperture images. For each received synthesized image, the client “decodes” and recovers a new sub-aperture image using a cache of known sub-aperture images. As the cache of decoded sub-aperture images grows over time, the client becomes capable of synthesizing new view/focal point images, reducing overall transmission cost. Experimental results show that our proposed scheme can deliver synthesized images at high quality, even though only sparse sub-aperture images are used for synthesis. Moreover, compared with a scenario where the requested synthesized images are always transmitted, our proposed scheme achieves a significant reduction in accumulated rate when a sufficient number of images are transmitted.

Touchless multiview fingerprinting technology has been proposed as an alternative to overcome intrinsic problems of traditional touchbased systems. However, if one wants to benefit from the advantages presented by touchless scanners, the captured images must be processed in order to become compatible to touchbased systems. This paper proposes a two-step solution to the touchless-to-touch compatibility problem. First, it reproduces the texture of touchbased acquisition; and second, it performs a geometric transformation in order to approximate the nail-to-nail touchbased enrollment process. Two experiments were proposed, one to evaluate the quality of the processed images, another one to estimate the EER (equal error rate) for a set of 200 fingerprints (100 fingers, 2 images per finger). Results show that 90% of the images presented good, very good or excellent scores according to NFIQ. In addition, the observed EER was approximately 4%, demonstrating the viability of the proposed method.

In this work we propose an online verification system for both signature and isolated cursive words that uses histograms of angles, velocity and Fourier descriptors. The proposed system is designed to be used in a mobile device with limited computational capability and where users often draw using their fingertip instead of a stylus pen. The goal is to use signatures or isolated words to unlock certain applications, content or the device itself. In this context, isolated words have certain desirable properties that can be more useful than signatures. Different isolated words can be used to verify the user in different applications, combining knowledge-based security systems (i.e. passwords) with a behavioral biometric verification system. The proposed system creates a profile of several histograms sets, Fourier descriptors and statistical information based on a number of input samples. A simple Manhattan distance measure is used to compare the query sample with the profile in order to verify the identity of the user. This method can achieve an equal error rate as low as |$3.25\%$| for isolated words.

Typical H.265/High Efficiency Video Coding (HEVC) encoder implementations test a variety of prediction modes and select the optimal configuration for each block in terms of Rate-Distortion (RD) cost. A fast HEVC mode decision algorithm is proposed here referred to as Context Adaptive Mode Sorting (CAMS). The frequency of selection of modes and their RD costs are collected while encoding the training frames based on local parameters (the context). This information is then used to sort and restrict the prediction modes to test for each context, while the optimal mode found using CAMS on each CU is validated based on the RD cost distributions found during the training. Experimental results show that the method reduces total encoding time of fast HEVC implementations on average by 29.3%, at modest efficiency losses.

Although images as viewed from intermediate virtual viewpoints can be synthesized using texture and depth maps from nearby camera views via depth-image-based rendering (DIBR), the rendered images contain disocclusion holes - spatial regions that were not visible in the reference views due to foreground object occlusion - that requires proper filling. In this paper, we introduce a new signal prior into the hole filling problem formulation: given disocclusion holes are part of the background and background tends to have low visual saliency, the extrapolated signal into the holes must also be of low saliency. Mathematically, we add a low-saliency prior to an exemplar-based inpainting algorithm, so that the best-matched block has both small matching cost and is of low visual saliency. Moreover, we compute a suitable Lagrange multiplier value for the saliency cost term via analysis of the reference images. Experimental results show that using a low-saliency prior can improve performance by 0.5 dB over a previous hole filling scheme.

In this paper, a fast transcoding solution from H.264/AVC to HEVC bitstreams is presented. This solution is based on two main modules: a coding unit (CU) classification module that relies on a machine learning technique in order to map H.264/AVC macroblocks into HEVC CUs; and an early termination technique that is based on statistical modeling of the HEVC rate-distortion (RD) cost in order to further speed-up the transcoding. The transcoder is built around an established two-stage transcoding. In the first stage, called the training stage, full re-encoding is performed while the H.264/AVC and the HEVC information are gathered. This information is then used to build both the CU classification model and the early termination sieves, that are used in the second stage (called the transcoding stage). The solution is tested with well-known video sequences and evaluated in terms of RD and complexity. The proposed method is 3.83 times faster, on average, than the trivial transcoder, and 1.8 times faster than a previous transcoding solution, while yielding a RD loss of 4% compared to this solution.

Free-viewpoint video conferencing allows a participant to observe the remote 3D scene from any freely chosen viewpoint. An intermediate virtual viewpoint image is typically synthesized using two pairs of transmitted texture and depth maps from two neighboring captured viewpoints via depth-image-based rendering (DIBR). To maintain high quality of synthesized images, it is imperative to contain the adverse effects of network packet losses that may arise during texture and depth video transmission. Towards this goal, we develop an integrated approach that exploits the representation redundancy inherent in the multiple streamed videos-a voxel in the 3D scene visible to two captured views is sampled and coded twice in the two views. In particular, at the receiver we first develop an error concealment strategy that adaptively blends corresponding pixels in the two captured views during DIBR, so that pixels from the more reliable transmitted view are weighted more heavily. We then couple it with a sender-side optimization of reference picture selection (RPS) during real-time video coding, so that blocks containing pixel samples of voxels that are visible in both views are more error-resiliently coded in one view only, given adaptive blending will mitigate errors in the other view. Further, synthesized view distortion sensitivities to texture versus depth errors are analyzed, so that relative importance of texture and depth code blocks can be computed for system-wide RPS optimization. Finally, quantization parameter (QP) is adaptively selected per frame, optimally trading off source distortion due to compression with channel distortion due to potential packet losses. Experimental results show that the proposed scheme can outperform previous work by up to 2.9 dB at 5% packet loss rate.

In free viewpoint video, texture and depth maps from two camera-captured viewpoints are transmitted, so that at receiver, a novel virtual view chosen by the client can be synthesized via depth-image-based rendering (DIBR). When irrecoverable packet losses occur during transmission-typically affecting less important spatial regions in the video given unequal error protection (UEP) is deployed-appropriate error concealment strategies must be used at decoder to minimize resulting visual degradation in the synthesized view. Towards this goal, we propose a new optimization framework based on visual saliency to combine two different concealment techniques. First, given a pixel in the virtual view is typically constructed as a convex combination of corresponding pixels in the left and right captured views, weighted pixel blending (WPB) readjusts the weights in the linear sum to reflect the expected error in code blocks that contain the corresponding pixels. Second, exemplar-based patch matching (EPM) finds the most similar patches in the known spatial region to complete missing pixels in the unknown region. To choose between candidates constructed using the two techniques when filling a given pixel patch in the synthesized view, we first compute a weighted sum of expected error and visual saliency for each candidate patch. The candidate with the smaller sum (one with small expected error and visual saliency, so that even if errors do occur, they do not stand out visually) is selected for pixel completion. Experimental results show that our scheme can outperform the use of co-located blocks from a previous frame by up to 0.7dB in PSNR and improve subjective visual quality.

The emerging video coding standard, HEVC, was developed to replace the current standard, H.264/AVC. However, in order to promote inter-operability with existing systems using the H.264/AVC, transcoding from H.264/AVC to the HEVC codec is highly needed. This paper presents a transcoding solution that uses machine learning techniques in order to map H.264/AVC macroblocks into HEVC coding units (CUs). Two alternatives to build the machine learning model are evaluated. The first uses a static training, where the model is built offline and used to transcode any video sequence. The other uses a dynamic training, with two well-defined stages: a training stage and a transcoding stage. In the training stage, full re-encoding is performed while the H.264/AVC and the HEVC information are gathered. This information is then used to build a model, which is used in the transcoding stage to classify the HEVC CU partitioning. Both solutions are tested with well-known video sequences and evaluated in terms of rate-distortion (RD) and complexity. The proposed method is on average 2.26 times faster than the trivial transcoder using fast motion estimation, while yielding a RD loss of only 3.6% in terms of bitrate.

This paper reports the results of a psychometric experiment aimed at investigating the visibility and annoyance of packet-loss artifacts, with a focus on understanding to what extent their presence influences viewing behavior. The rationale behind this study is that packet loss artifacts might “distract” visual attention, creating visual saliency themselves, thereby becoming more visible. In turn, higher artifact visibility might impact their annoyance. Our study involved seven videos, compressed at a very high bitrate (thus free of spatial artifacts), impaired by discarding packets at different packet loss ratios. We tracked the observers' eye-movements while (1) they assessed the annoyance of impaired videos and (2) looked freely at pristine videos. Our results show that the viewing behavior significantly changes from pristine to impaired videos. This change is related to both properties of the video and to the annoyance of the artifacts presented.

In a free-viewpoint video conferencing system, the viewer can choose any desired viewpoint of the 3D scene for observation. Rendering of images for arbitrarily chosen viewpoint can be achieved through depth-image-based rendering (DIBR), which typically employs “texture-plus-depth” video format for 3D data exchange. Robust and timely transmission of multiple texture and depth maps over bandwidth-constrained and loss-prone networks is a challenging problem. In this paper, we optimize transmission of multiview video in texture-plus-depth format over a lossy channel for free viewpoint synthesis at decoder. In particular, we construct a recursive model to estimate the distortion in synthesized view due to errors in both texture and depth maps, and formulate a rate-distortion optimization problem to select reference pictures for macroblock encoding in H.264 in a computation-efficient way, in order to provide unequal protection to different macroblocks. Results show that the proposed scheme can outperform random insertion of intra refresh blocks by up to 0.73 dB at 5% loss.

This paper proposes a hybrid pattern matching/transform-based compression engine for scanned compound documents. The novelty of this approach is demonstrated by using a modified version of the HEVC (High Efficiency Video Coding) Test Model as a compound document compressor, here conveniently referred to as HEDC (High Efficiency Document Coder). The proposed method uses segments of a document to create a video sequence, which is then encoded by HEDC. The idea is to explore interframe prediction as a pattern matching algorithm for coding units pre-classified as text; and intraframe prediction for coding units pre-classified as image. Results show that HEDC outperforms AVC-I, HEVC-I (H.264/AVC and HEVC operating in pure intra mode), H.264/AVC and JPEG2000 by up to 3.3, 2.5, 1.7 and 5 dB, respectively. Furthermore, for most documents the proposed method yields practically the same rate-distortion performance as regular HEVC, but is approximately 5% to 20% faster due to a pre-classification algorithm that prevents it of performing all possible inter/intra prediction tests for each prediction unit.

Inspired by recent results showing that Wyner-Ziv coding using a combination of source and channel coding may be more efficient than pure channel coding, we have applied coset codes for the source coding part in the transform domain for Wyner-Ziv coding of video. The framework is a mixed-resolution approach where reduced encoding complexity is achieved by low resolution encoding of non-reference frames and regular encoding of the reference frames. Different from our previous works, no motion estimation is carried at the encoder, neither for the low resolution frames nor the reference frames. The entropy coders of the H.264/AVC codec were tuned to improve their performance for encoding the cosets. An encoding mode with lowest encoding complexity than H.264/AVC intra mode is achieved. Experimental results show a competitive rate-distortion performance especially at low bit rates.

We propose a mixed resolution framework based on full resolution key frames and spatial-reduction-based Wyner-Ziv coding of intermediate nonreference frames. Improved rate-distortion performance is achieved by enabling better side-information generation at the decoder side and better rate-allocation at the encoder side. The framework enables reduced encoding complexity by low resolution encoding of the nonreference frames, followed by Wyner-Ziv coding of the Laplacian residue. The quantized transform coefficients of the residual frame are mapped to cosets without the use of a feedback channel. A study to select optimal coding parameters in the creation of the memoryless cosets is made. Furthermore, a correlation estimation mechanism that guides the parameter choice process is proposed. The decoder first decodes the low resolution base layer and then generates a super-resolved side-information frame at full resolution using past and future key frames. Coset decoding is carried using side-information to obtain a higher quality version of the decoded frame. Implementation results are presented for the H.264/AVC codec.

In this paper we present a statistical model used to select coding parameters for a mixed resolution Wyner-Ziv framework implemented using the H.264/AVC standard. This paper extends the results of a previous work for the H.263+ case to the H.264/AVC coder, since the parameters need to be recalculated for the H.264 case. The proposed correlation estimation mechanism guides the parameter choice process, and also yields the statistical model used for decoding. This mechanism is proposed based on extracting edge information and residual error rate in co-located blocks from the low resolution base layer that is available at both ends.

A motion-based side-information generation scheme with semi super-resolution for a scalable Wyner-Ziv coder framework is introduced. It is known that the performance of any Wyner-Ziv coder is heavily dependent on the efficiency of the side-information generation. We propose an iterative block based scheme to generate a semi super-resolution frame using the past and future reference frames which should be coded at full-resolution. To enable this side-information generation the framework should allow for low encoding complexity, reducing the spatial resolution only in the non-reference frames. The enhancement layer is produced using a residual frame of the reduced resolution encoded frame. The decoder first decodes the low resolution base layer and iteratively generates the side-information, along with channel decoding, to obtain a higher quality version of the decoded frame. Results of the implementation of the framework using the motion-based side-information in the H.263+ and H.264 standards are presented.

A spatial-resolution reduction based framework for incorporation of a Wyner-Ziv frame coding mode in existing video codecs is presented, to enable a mode of operation with low encoding complexity. The core Wyner-Ziv frame coder works on the Laplacian residual of a lower-resolution frame encoded by a regular codec at reduced resolution. The quantized transform coefficients of the residual frame are mapped to cosets to reduce the bit-rate. A detailed rate-distortion analysis and procedure for obtaining the optimal parameters based on a realistic statistical model for the transform coefficients and the side information is also presented. The decoder iteratively conducts motion-based side-information generation and coset decoding, to gradually refine the estimate of the frame. Preliminary results are presented for application to the H.263+ video codec.