This paper presents a quantitative approach to demonstrate the robustness of neural networks for tabular data. These data form the backbone of the data structures found in most industrial applications. We analyse the effect of various widely used techniques we encounter in neural network practice, such as regularization of weights, addition of noise to the data, and positivity constraints. This analysis is performed by using three state-of-the-art techniques, which provide mathematical proofs of robustness in terms of Lipschitz constant for feed-forward networks. The experiments are carried out on two prediction tasks and one classification task. Our work brings insights into building robust neural network architectures for safety critical systems that require certification or approval from a competent authority.

The activity of generic flight path monitoring of an aircraft on the radar screen is one the major problem which results in air traffic controllers (ATCOs) losing awareness of it. A novel eye signature, known as the radar command group (RCG) time entropy signature, is thus presented in this paper. This signature seeks to model the monitoring behavior of ATCOs using eye-tracking technique during this activity. Acquisition and representation of this monitoring behavior is achieved by first identifying the respective fixation count and mouseover label movement on the aircraft to establish a RCG. The regularity of a RCG exhibited on an aircraft is then calculated using the information entropy formula. Real time simulations are conducted for 88 one-hour experimental sessions with licensed and non-licensed participants comprising of three expertise levels, using scenarios that mimic actual air traffic, consisting of various flight path configuration. Test results of the RCG time entropy signature showed that, an aircraft flying on a flight path with longer distance and no change in altitude likely results in a more regular generic flight path monitoring. Furthermore, it can also be used to differentiate the novice participants from intermediates and experts, as novices lack the training. An ATCO's expertise level can thus be determined and benchmarked accordingly, allowing this signature to be applied on future ATCO training.

Transmission and compression technologies advancement over the past decade led to a shift of multimedia content towards cloud systems. Multiple copies of the same video are available through numerous distribution systems. Different compression levels, algorithms and resolutions are used to match the requirements of particular applications. As 4k display technologies are rapidly adopted, resolution enhancement algorithms are of vital importance. Current solutions do not take into account the particularities of different video encoders, while video reconstruction methods from compressed sources do not provide resolution enhancement. In this paper, we propose a multi source compressed video enhancement framework, where each description can have a different compression level and resolution. Using a variational formulation based on a modern proximal dual splitting algorithm, we efficiently combine multiple descriptions of the same video. Two applications are proposed: combining two compressed low resolution (LR) descriptions of a video sequence into a high resolution (HR) description and enhancing a compressed HR video using a LR compressed description. Tests are performed over multiple video sequences encoded with high efficiency video coding, at different compression levels and resolutions obtained through multiple down-sampling methods.

This paper addresses the problem of encoding the video generated by the screen of an airplane cockpit. As other computer screens, cockpit screens consist of computer-generated graphics often atop a natural background. Existing screen content coding schemes fail notably in preserving the readability of textual information at the low bitrates required in avionic applications. We propose a screen coding scheme where textual information is encoded according to the relative semantics rather than in the pixel domain. The encoder localizes textual information, and the semantics of each character are extracted with a convolutional neural network and predictively encoded. Text is then removed via inpainting, and the residual background video is compressed with a standard codec and transmitted to the receiver together with the text semantics. At the decoder side, text is synthesized using the decoded semantics and superimposed over the decoded residual video recovering the original frame. Our proposed scheme offers two key advantages over a semantics-unaware scheme that encodes text in the pixel domain. First, the text readability at the decoder is not compromised by compression artifacts, whereas the relative bitrate is negligible. Second, removal of high-frequency transform coefficients associated with the inpainted text drastically reduces the bitrate of the residual video. Experiments with real cockpit video sequences show BD-rate gains up to 82% and 69% over a reference H.265/HEVC encoder and its screen content coding extension. Moreover, our scheme achieves quasi-errorless character recognition already at very low bitrates, whereas even HEVC-SCC needs at least three or four times more bitrate to achieve a comparable error rate.

Video enhancement methods enable to optimize the viewing of video content at the end-user side. Most approaches do not consider the compressed nature of the available content. In the present work, we build upon a recently proposed video enhancement approach that explicitly models a compression stage. To apply the enhancement framework on compressed representations requires to extract specific syntax elements during their decoding. This additional information embeds the enhanced result in a domain that closely fits the observation. We evaluate the framework performance in a single source resolution enhancement scenario, and show the method efficiency with respect to state-of-the-art approaches.

Infrastructure surveillance is an important requirement for many companies. With the advancement of technology, drones can now provide an efficient tool for such applications. A possible future scenario is the automated surveillance of railroads. Whereas numerous algorithms that provide railroad detection exist, they have mainly focused either on satellite images or for small, low altitude drones which are unsuitable for our particular scenario. In this paper we propose a railroad detection algorithm tailored for large, high altitude enduring drones. More specifically, we use Hough Transform to detect lines and perform a line clustering in the Rho and Theta space. A score model is also proposed in order to identify the railroad. We test our method on several sequences supplied by Airbus Defense & Space and show our algorithm to provide a detection rate of 93.23% in average.

Advancements over the last decade in video acquisition and display technologies lead to a continuous increase of video content resolution. These aspects combined with the shift towards cloud multimedia services and the underway adoption of High Efficiency Video Coding standards (HEVC) create a lot of interest for Super-Resolution (SR) and video enhancing techniques. Recent works showed that proximal based convex optimization approaches provide a promising direction in video restoration. An important aspect in the definition of a SR model is the metric used in defining the objective function. Most techniques are based on the classical I2norm. In this paper we further investigate the use of other norms and their behavior w.r.t. multiple quality evaluation metrics. We show that significant gains of up to 0.5 dB can be obtained when using different norms.

This paper proposes an extended selective encryption (SE) method for both H.264/advanced video coding (AVC) (CABAC) and High Efficiency Video Coding (HEVC) streams, addressing the main security issue that SE is facing: content protection, related to the amount of information leakage through a protected video. Our contribution is the improvement in the visual distortion induced by SE approaches. Previous works on both H.264/AVC (CABAC) and HEVC limit encryption to bins treated by one specific mode of CABAC-its bypass mode-which has the advantage of preserving the overall bitrate, we propose here to also rely on the encryption of the more widely used mode of CABAC-its regular mode. This allows encryption of a major codeword for video reconstruction, the prediction modes for intra blocks/units. Disturbing their statistics may cause bitrate overhead, which is the tradeoff for improving the content security level of the SE approach. A comprehensive study of this compromise between the improvement in the scrambling efficiency and the undesirable aftereffects is presented in this paper, and a specific security analysis of the proposed CABAC regular mode encryption is conducted.

Optimal rate allocation is among the most challenging tasks to perform in the context of predictive video coding, because of the dependencies between frames induced by motion compensation. In this paper, using a recursive rate-distortion model that explicitly takes into account these dependencies, we approach the frame-level rate allocation as a convex optimization problem. This technique is integrated into the recent HEVC encoder, and tested on several standard sequences. Experiments indicate that the proposed rate allocation ensures a better performance (in the rate-distortion sense) than the standard HEVC rate control, and with a little loss with respect to an optimal exhaustive research, which is largely compensated by a much shorter execution time.

The demand for 3D content has increased over the last years as 3D displays are now widespread. View synthesis methods, such as depth-image-based-rendering, provide an efficient tool in 3D content creation or transmission, and are integrated in coding solutions for multiview video content such as 3D-HEVC. In this paper, we propose a view synthesis method that takes advantage of temporal and inter-view correlations in multiview video sequences. We use warped motion vector fields computed in reference views to obtain temporal predictions of a frame in a synthesized view and blend them with depth-image-based-rendering synthesis. Our method is shown to bring gains of 0.42dB in average when tested on several multiview sequences.

Super Resolution (SR) addresses the problem of image and video upscaling. Most of the best performing SR methods do not take into account any compression prior into the degradation model. Consequently, compression artifacts can be undesirably amplified during SR. In the present work, we propose a novel HEVC-dedicated approach for embedding SR results into a domain that closely fits the compressed observation. Our main contribution is the inclusion of HEVC syntax (block size, quantization parameters etc.) into the degradation model. A recent convex optimization approach is used to solve the associated minimization problem. Over a wide range of resolutions and bitrates, we show that our method improves the results obtained with state of the art SR.

Depth maps are typically made of smooth regions separated by sharp edges. Following this rationale, this paper presents a novel coding scheme where depth data is represented by a set of contours defining the various regions together with a compact representation of the values inside each region. The proposed coding scheme is based on elastic curves, which make possible to compactly represent the contours exploiting also the temporal consistency in different frames. A 3D surface prediction algorithm is then used to obtain an accurate estimation of the depth field from the coded contours and a subsampled version of the data. Finally, an ad-hoc coding strategy for the low resolution data and the prediction residuals is presented. Experimental results prove how the proposed approach is able to obtain a very high coding efficiency outperforming the HEVC coder at medium-low bitrates.

Optimal rate allocation is among the most challenging tasks to perform in the context of multi-view video coding, because of the dependency between frames induced by motion compensation and depth image-based rendering. In this paper, using a recursive rate-distortion model that explicitly takes into account these dependencies, we approach the frame-level rate allocation as a convex optimization problem. Within this framework, we provide an efficient algorithm for exactly solving the above problem with recent convex optimization tools. Experiments on standard sequences demonstrate the interest of considering the proposed rate allocation method and confirm that our approach ensures a better performance (in ratedistortion sense) than the standard MV-HEVC rate control.

Low-bitrate video compression is a challenging task, particularly with the increasing complexity of video sequences. Re-shaping video data before its compression with modern hybrid encoders has provided interesting results in the low and ultra-low bit rate domains. In this work, we propose a novel saliency guided preprocessing approach, which combines adaptive re-sampling and background texture removal, to achieve efficient ROI-oriented compression. Evaluated with HEVC, we show that our solution improves the ROI encoding over a wide range of resolutions and bit rates whilst maintaining a high background intelligibility level.

As 3D media became more and more popular over the last years, new technologies are needed in the transmission, compression and creation of 3D content. One of the most commonly used techniques for aiding with the compression and creation of 3D content is known as view synthesis. The most effective class of view synthesis algorithms are using Depth-Image-Based-Rendering techniques, which use explicit scene geometry to render new views. However, these methods may produce geometrical distortions and localized artifacts which are difficult to evaluate as they are inherently different from encoding errors and they are perceived differently by human subjects. In this paper, we propose a region-of-interest evaluation technique for view synthesis based on DIBR methods. Based on the assumption that certain areas determined by the geometrical properties of the scene are prone to distortions, we select a ROI by analyzing the multiple DIBR methods together with the ground truth. The approach is tested using a subjective evaluation view synthesis database and show that our method improves the SSIM correlation with subjective scores We also test another similar method and traditional metrics.

Multiview video (MVV) plus depths formats use view synthesis to build intermediate views from existing adjacent views at the receiver side. Traditional view synthesis exploits the disparity information to interpolate an intermediate view by considered inter-view correlations. However, temporal correlation between different frames of the intermediate view can be used to improve the synthesis. We propose a new coding scheme for 3-D High Efficiency Video Coding (HEVC) that allows us to take full advantage of temporal correlations in the intermediate view and improve the existing synthesis from adjacent views. We use optical flow techniques to derive dense motion vector fields (MVF) from the adjacent views and then warp them at the level of the intermediate view. This allows us to construct multiple temporal predictions of the synthesized frame. A second contribution is an adaptive fusion method that judiciously selects between temporal and inter-view prediction to eliminate artifacts associated with each prediction type. The proposed system is compared against the state-of-the-art view synthesis reference software 1-D Fast technique used in 3-D HEVC standardization. Three intermediary views are synthesized. Gains of up to 1.21-dB Bjontegaard Delta peak SNR are shown when evaluated on several standard MVV test sequences.

Augmented reality, interactive navigation in 3D scenes, multiview video, and other emerging multimedia applications require large sets of images, hence larger data volumes and increased resources compared with traditional video services. The significant increase in the number of images in multiview systems leads to new challenging problems in data representation and data transmission to provide high quality of experience on resource-constrained environments. In order to reduce the size of the data, different multiview video compression strategies have been proposed recently. Most of them use the concept of reference or key views that are used to estimate other images when there is high correlation in the data set. In such coding schemes, the two following questions become fundamental: 1) how many reference views have to be chosen for keeping a good reconstruction quality under coding cost constraints? And 2) where to place these key views in the multiview data set? As these questions are largely overlooked in the literature, we study the reference view selection problem and propose an algorithm for the optimal selection of reference views in multiview coding systems. Based on a novel metric that measures the similarity between the views, we formulate an optimization problem for the positioning of the reference views, such that both the distortion of the view reconstruction and the coding rate cost are minimized. We solve this new problem with a shortest path algorithm that determines both the optimal number of reference views and their positions in the image set. We experimentally validate our solution in a practical multiview distributed coding system and in the standardized 3D-HEVC multiview coding scheme. We show that considering the 3D scene geometry in the reference view, positioning problem brings significant rate-distortion improvements and outperforms the traditional coding strategy that simply selects key frames based on the distance between cameras.

This paper analyzes the impact on the stability of the TCP-Reno congestion control mechanism when a network coding (NC) layer is inserted in the TCP/IP stack. A model of the dynamics of the TCP-NC protocol combined with random early detection (RED) as active queue management mechanism is considered to study the network equilibrium and stability properties. The existence and uniqueness of an equilibrium point is demonstrated and characterized in terms of average throughput, loss rate, and queue length. Global stability is proved in absence of forward delay, and the effects of the NC redundancy factor and of the delay on the local stability of TCP-NC-RED are studied around the equilibrium. The fairness of TCP-NC with respect to TCP-Reno-like protocols is also studied. A version of TCP-NC with adaptive redundancy factor (TCP-NCAR) is also introduced. Results provided by the proposed model are compared to those obtained by simulation for N sources sharing a single link. TCP-NC-RED becomes unstable when delay or capacity increases, as TCP-Reno does, but also when the redundancy factor increases. Its stability region is characterized as a function of the redundancy factor. If TCP-NC and TCP-Reno share the same links, TCP-NC is fair with TCP-Reno-like protocols when no redundancy is added. Simulations show that TCP-NCAR is able to compensate losses on the wireless parts of the network.

View synthesis received increasing attention over the last years, as it offers a wide range of practical applications like Free Viewpoint Television, 3D video, video gaming, etc. The main issues in view synthesis are the filling of disoccluded areas and the warping of real views. In this paper we propose a new hole filling method, it uses temporal correlations in the real views to extract information on disoccluded areas from different time instants in the synthetic view. We also propose a sub-pixel warping technique that takes into account depth and can be used for both the warping of the real view as well as for motion compensation. Our method is proved to bring gains of up to 0.31dB in average over several multiview test sequences.

The growth in the use of high definition (HD) and above video resolutions streams has outstripped the rate at which network infrastructure has been deployed. Video streaming applications require appropriate rate control techniques that make use of the specific characteristics of the video content, such as the regions of interest (ROI). With the introduction of high efficiency video coding (HEVC) streams, we consider new coding features to make a novel ROI-based rate control (RC) algorithm. The proposed approach introduces tiling in a ROI-based rate control scheme. It aims at enhancing the quality of important regions (i.e. faces for a videoconferencing system) considering independently coded regions lying within an ROI and helps evaluating the ROI quality under poor channel conditions. Our work consists of two major steps. First, we designed a RC algorithm based on an independent processing of tiles of different regions. Second, we investigate the effect of ROI- and tile-based rate control algorithm on the decoded quality of the stream transmitted over a lossy channel.

Multi-view video plus depth is emerging as the most flexible format for 3D video representation, as witnessed by the current standardization efforts by ISO and ITU. The depth information allows synthesizing virtual view points, and for its compression various techniques have been proposed. It is generally recognized that a high quality view rendering at the receiver side is possible only by preserving the contour information since distortions on edges during the encoding step would cause a sensible degradation on the synthesized view and on the 3D perception. As a consequence recent approaches include contour-based coding of depths. However, the impact of contour-preserving depth-coding on the perceived quality of synthesized images has not been conveniently studied. Therefore in this paper we make an investigation by means of a subjective study to better understand the limits and the potentialities of the different techniques. Our results show that the contour information is indeed relevant in the synthesis step: preserving the contours and coding coarsely the rest typically leads to images that users cannot tell apart from the reference ones, even at low bit rate. Moreover, our results show that objective metrics that are commonly used to evaluate synthesized images may have a low correlation coefficient with MOS rates and are in general not consistent across several techniques and contents.

The side information (SI) in Distributed Video Coding (DVC) is estimated using the available decoded frames and exploited for the decoding and reconstruction of other frames. The quality of the SI has a strong impact on the performance of DVC. Here, we propose a new approach that combines both global and local SI to improve coding performance. Since the background pixels in a frame are assigned to global estimation and the foreground objects to local estimation, one needs to estimate foreground objects in the SI using the backward and forward foreground objects, the background pixels are directly taken from the global SI. Specifically, elastic curves and local motion compensation are used to generate the foreground objects masks in the SI. Experimental results show that, as far as the rate-distortion performance is concerned, the proposed approach can achieve a PSNR improvement of up to 1.39 dB for a group of picture (GOP) size of 2, and up to 4.73 dB for larger GOP sizes, with respect to the reference DISCOVER codec.

Sparsity inducing penalizations are useful tools in variational methods for machine learning. In this paper, we design a learning algorithm for multiclass support vector machines that allows us to enforce sparsity through various nonsmooth regularizations, such as the mixed ℓ1, p-norm with p ≥ 1. The proposed constrained convex optimization approach involves an epigraphical constraint for which we derive the closed-form expression of the associated projection. This sparse multiclass SVM problem can be efficiently implemented thanks to the flexibility offered by recent primal-dual proximal algorithms. Experiments carried out for handwritten digits demonstrate the interest of considering nonsmooth sparsity-inducing regularizations and the efficiency of the proposed epigraphical projection method.

Optimal rate allocation is among the most challenging tasks to perform in the context of predictive video coding, because of the dependencies between frames induced by motion compensation. In this paper, we derive an analytical rate-distortion model that explicitly takes into account the dependencies between frames. The proposed approach allows us to formulate the frame-level optimal rate allocation as a convex optimization problem. Within this framework, we are able to achieve the exact solution in limited time (even for large-size problems), thanks to the flexibility offered by recent convex optimization techniques. Experiments on standard sequences demonstrate the interest of considering the proposed rate-distortion model and confirm that the optimal rate allocation ensures a better distribution of the total bit budget, with superior results (in the rate-distortion sense) with respect to the standard H.264/AVC rate control.

The objective of this paper is to study rate-distortion properties of a quantized Bernoulli-Generalized Gaussian source. Such source model has been found to be well-adapted for signals having a sparse representation in a transformed domain. We provide here accurate approximations of the entropy and the distortion functions evaluated through a p-th order error measure. These theoretical results are then validated experimentally. Finally, the benefit that can be drawn from the proposed approximations in bit allocation problems is illustrated for a wavelet-based compression scheme.

In this paper, we propose a new rate control scheme designed for the newest high efficiency video coding (HEVC) standard, and aimed at enhancing the quality of regions of interest (ROI). Our approach allocates a higher bit rate to the region of interest while keeping the global bit rate close to the assigned target value. This algorithm is developed for a videoconferencing system, where the ROIs (typically, faces) are automatically detected and each coding unit is classified in a region of the interest map. This map is given as input to the rate control algorithm and the bit allocation is made accordingly. Experimental results show that the proposed scheme achieves accurate target bit rates and provides an improvement in the region of interest quality, both in objective metrics and based on subjective quality evaluation.

Super Multi-View (SMV) video is a key enabler for future 3D video services that allows a glasses-free visualization and eliminates many causes of discomfort existing in current available 3D video technologies. SMV video content is composed of tens or hundreds of views, that can be aligned in horizontal only or both horizontal and vertical directions, providing respectively horizontal parallax or full parallax. This paper compares several coding schemes and coding orders, and proposes a coding structure that exploits inter-view correlations in the two directions, providing BD-rate gains up to 29.1% when compared to a basic anchor structure. Additionally, Neighboring Block Disparity Vector (NBDV) and Inter-View Motion Prediction (IVMP) coding tools are further improved to efficiently exploit coding structures in two dimensions, with BD-rate gains up to 4.2% reported over the reference 3D-HEVC encoder.

Multiview image and video systems with large number of views lead to new problems in data representation, transmission and user interaction. In order to reduce the data volumes, most distributed multiview coding schemes exploit the inter-view redundancies at the decoder side, using view synthesis from key views. In the situation where many views are considered, the two following questions become fundamental: i) how many key views have to be chosen for keeping a good reconstruction quality with reasonable coding cost? ii) where to place them optimally in the multiview sequences? We propose in this paper an algorithm for selecting the key views in a distributed multiview coding scheme. Based on a novel metric for the correlation between the views, we formulate an optimization problem for the positioning of the key views such that both the distortion of the reconstruction and the coding rate cost are effectively minimized. We then propose a new optimization strategy based on shortest path algorithm that permits to determine both the optimal number of key views and their positions in the image set. We experimentally validate our solution in a practical distributed multiview coding system and we show that considering the 3D scene geometry in the key view positioning brings significant rate-distortion improvements compared to distance-based key view selection as it is commonly done in the literature.

The 3D video extension of High Efficiency Video Coding (3D-HEVC) exploits texture-depth redundancies in 3D videos using intercomponent coding tools. It also inherits the same quadtree coding structure as HEVC for both components. The current software implementation of 3D-HEVC includes encoder shortcuts that speed up the quadtree construction process, but those are always accompanied by coding losses. Furthermore, since the texture and its associated depth represent the same scene, at the same time instant and view point, their quadtrees are closely linked. In this paper, an intercomponent tool is proposed in which this link is exploited to save both runtime and bits through a joint coding of the quadtrees. If depth is coded before the texture, the texture quadtree is initialized from the coded depth quadtree. Otherwise, the depth quadtree is limited to the coded texture quadtree. A 31% encoder runtime saving, a -0.3% gain for coded and synthesized views and a -1.8% gain for coded views are reported for the second method.

Nonlocal total variation (NLTV) has emerged as a useful tool in variational methods for image recovery problems. In this paper, we extend the NLTV-based regularization to multicomponent images by taking advantage of the structure tensor (ST) resulting from the gradient of a multicomponent image. The proposed approach allows us to penalize the nonlocal variations, jointly for the different components, through various l(1, p)-matrix-norms with p ≥ 1. To facilitate the choice of the hyperparameters, we adopt a constrained convex optimization approach in which we minimize the data fidelity term subject to a constraint involving the ST-NLTV regularization. The resulting convex optimization problem is solved with a novel epigraphical projection method. This formulation can be efficiently implemented because of the flexibility offered by recent primal-dual proximal algorithms. Experiments are carried out for color, multispectral, and hyperspectral images. The results demonstrate the interest of introducing a nonlocal ST regularization and show that the proposed approach leads to significant improvements in terms of convergence speed over current state-of-the-art methods, such as the alternating direction method of multipliers.

In this paper, we develop an efficient bit allocation strategy for subband-based image coding systems. More specifically, our objective is to design a new optimization algorithm based on a rate-distortion optimality criterion. To this end, we consider the uniform scalar quantization of a class of mixed distributed sources following a Bernoulli-generalized Gaussian distribution. This model appears to be particularly well-adapted for image data, which have a sparse representation in a wavelet basis. In this paper, we propose new approximations of the entropy and the distortion functions using piecewise affine and exponential forms, respectively. Because of these approximations, bit allocation is reformulated as a convex optimization problem. Solving the resulting problem allows us to derive the optimal quantization step for each subband. Experimental results show the benefits that can be drawn from the proposed bit allocation method in a typical transform-based coding application.

Blind Source Separation (BSS) deals with the recovery of source signals from a set of observed mixtures, when little or no knowledge of the mixing process is available. BSS can find an application in the context of network coding, where relaying linear combinations of packets maximizes the throughput and increases the loss immunity. By relieving the nodes from the need to send the combination coefficients, the overhead cost is largely reduced. However, the scaling ambiguity of the technique and the quasi-uniformity of compressed media sources makes it unfit, at its present state, for multimedia transmission. In order to open new practical applications for BSS in the context of multimedia transmission, we have recently proposed to use a non-linear encoding to increase the discriminating power of the classical entropy-based separation methods. Here, we propose to append to each source a non-linear message digest, which offers an overhead smaller than a per-symbol encoding and that can be more easily tuned. Our results prove that our algorithm is able to provide high decoding rates for different media types such as image, audio, and video, when the transmitted messages are less than 1.5 kilobytes, which is typically the case in a realistic transmission scenario.

Multiview distributed video coding (DVC) has gained much attention in the last few years because of its potential in avoiding communication between cameras without decreasing the coding performance. However, the current results are not matching the expectations mainly due to the fact that some theoretical assumptions are not satisfied in the current implementations. For example, in distributed source coding the encoder must know the correlation between the sources, which cannot be achieved in the traditional DVC systems without having a communication between the cameras. In this work, we propose a novel multiview distributed video coding scheme in which the depth maps are used to estimate the way two views are correlated with no exchanges between the cameras. Only their relative positions are known. We design the complete scheme and further propose a rate allocation algorithm to efficiently share the bit budget between the different components of our scheme. Then, a rate allocation algorithm for depth maps is proposed in order to maximize the quality of synthesized virtual views. We show, through detailed experiments, that our scheme significantly outperforms the state-of-the-art DVC system.

TV-like constraints/regularizations are useful tools in variational methods for multicomponent image restoration. In this paper, we design more sophisticated non-local TV constraints which are derived from the structure tensor. The proposed approach allows us to measure the non-local variations, jointly for the different components, through various ℓ1,pmatrix norms with p ≥ 1. The related convex constrained optimization problems are solved through a novel epigraphical projection method. This formulation can be efficiently implemented thanks to the flexibility offered by recent primal-dual proximal algorithms. Experiments carried out for color images demonstrate the interest of considering a Non-Local Structure Tensor TV and show that the proposed epigraphical projection method leads to significant improvements in terms of convergence speed over existing numerical solutions.

In Blind Source Separation, or BSS, a set of source signals are recovered from a set of mixed observations without knowledge of the mixing parameters. Originated for real signals, BSS has recently been applied to finite fields, enabling more practical applications. However, classical entropy-based techniques do not perform well in finite fields. Here, we propose a non-linear encoding of the sources to increase the discriminating power of the separation methods. Our results show that the encoding improves the success rate of the separation for sources with few samples in large finite fields, both conditions met in practical networking applications. Our results open new possibilities in the context of network coding-wherein linear combinations of packets are sent in order to maximize throughput and increase loss immunity- by relieving the nodes from the need to send the combination coefficients, thus reducing the overhead cost.

In this paper, a new spatio-temporal saliency model is presented. Based on the idea that both spatial and temporal features are needed to determine the saliency of a video, this model builds upon the fact that locally contrasted and globally rare features are salient. The features used in the model are both spatial (color and orientations) and temporal (motion amplitude and direction) at several scales. To be more robust to moving camera a module computes the global motion and to be more consistent in time, the saliency maps are combined together after a temporal filtering. The model is evaluated on a dataset of 24 videos split into 5 categories (Abnormal, Surveillance, Crowds, Moving camera, and Noisy). This model achieves better performance when compared to several state-of-the-art saliency models.

A test model for an HEVC-based 3D video coding standard (3D-HEVC) has recently been drafted. 3D-HEVC exploits inter-view redundancies by including disparity-compensated prediction (DCP) for efficient dependent view coding. It also uses the Merge coding mode to reduce the cost of motion / disparity parameters. However, the candidates in the Merge list are mostly temporal motion vectors. DCP does not often benefit from accurate predictors and is thus costly. Consequently, motion-compensated prediction (MCP) remains largely preferred. In this paper, we propose to reduce the cost of DCP by modifying the Merge candidate list to always include a disparity vector candidate. Two methods are proposed: the new candidate is either added in the secondary or in the primary list of candidates. The latter method, which achieves average bitrate reductions of 0.6% for dependent views, and 0.2% for coded and synthesized views, was adopted in both the 3D-HEVC working draft and software.

The up-and-coming extension of HEVC for 3D video (3D-HEVC) includes various tools to exploit different redundancies in a 3D video signal. Inter-view redundancies are in particular exploited using Inter-View Motion Prediction (IVMP) and Inter-View Residual Prediction (IVRP). Both of these tools compensate disparity-wise the current prediction unit (PU) in order to find its corresponding PU in a base view, from which some prediction information for the current PU is retrieved. The disparity vector (DV) used for disparity compensation is currently derived using a neighboring search process (NBDV) for a DV across spatial and temporal neighbors. The first DV found is selected as the final DV used in IVMP and IVRP, with no guarantee of optimality. In this paper, the NBDV derivation process is changed: all found DVs from different neighbors are stored in a list. Redundant vectors in this list are removed, and a median computation on the remaining vectors is performed. The resulting DV is set as the DV used for IVMP. Average bitrate reductions of 0.6% and 0.8% for the two dependent views and 0.2% on synthesized views are reported with only a slight increase in encoder and decoder runtimes.

The quality of side information plays a key role in distributed video coding. In this paper, we propose a new approach that consists of combining global and local motion compensation at the decoder side. The parameters of the global motion are estimated at the encoder using scale invariant feature transform features. Those estimated parameters are sent to the decoder in order to generate a globally motion compensated side information. Conversely, a locally motion compensated side information is generated at the decoder based on motion-compensated temporal interpolation of neighboring reference frames. Moreover, an improved fusion of global and local side information during the decoding process is achieved using the partially decoded Wyner-Ziv frame and decoded reference frames. The proposed technique improves significantly the quality of the side information, especially for sequences containing high global motion. Experimental results show that, as far as the rate-distortion performance is concerned, the proposed approach can achieve a PSNR improvement of up to 1.9 dB for a Group of Pictures (GOP) size of 2, and up to 4.65 dB for larger GOP sizes, with respect to the reference DISCOVER codec.

The rate-distortion performance of a distributed video coding system strongly depends on the characteristics of the side information. One could naïvely think that the best side information is the one with the largest PSNR with respect to the original corresponding image. However, previous works have shown that this is not always the case and a reduction of the side information MSE does not always translate into better rate-distortion performance for the complete system. The scope of this paper is to explore a set of metrics other than the PSNR and explicitly designed to classify the side information with respect to its impact on the end-to-end compression performance. A first contribution is to define an experimental framework that can be used to meaningfully compare different metrics for side information evaluation. As a second contribution, our analysis allows to understand why in some cases PSNR-based metrics provide a fairly reliable estimation of the side information quality, while in other cases they do not. This analysis also allows us to introduce a set of new metrics that are better adapted for side information effectiveness evaluation, and that are based on a suitable power of the absolute difference between side information and the original image, or on the Hamming distance between the respective transform coefficients. Besides their theoretical interest, these new metrics can also improve the rate-distortion performance of some distributed video coding systems such as the hash-based ones. We observe improvement up to 74% rate reduction in a simple study case.

The concept of cosparsity has been recently introduced in the arena of compressed sensing. In cosparse modelling, the ℓ0(or ℓ1) cost of an analysis-based representation of the target signal isminimized under a data fidelity constraint. By taking benefit from recent advances in proximal algorithms, we show that it is possible to efficiently address a more general framework where a convex block sparsity measure is minimized under various convex constraints. The main contribution of this work is the introduction of a new epigraphical projection technique, which allows us to consider more flexible data fidelity constraints than the standard linear or quadratic ones. The validity of our approach is illustrated through an application to an image reconstruction problem in the presence of Poisson noise.

Network coding (NC) can achieve the maximum information flow in the network by allowing nodes to combine received packets before retransmission. Several papers have shown NC to be beneficial in mobile ad-hoc networks, but the delay introduced by buffered decoding raises a problem in real-time streaming applications. Here we propose to use NC jointly with multiple description coding (MDC) to allow instant decoding of the received packets. The optimal encoding coefficients are chosen via distributed optimisation of the expected video quality. Nodes receive up-to-date information about the network topology through a recently proposed protocol, originally designed for real-time streaming of MDC video. Results show that, due to the limitations imposed by instant decoding to the coding window size, our approach consistently outperforms the popular technique of random linear network coding.

Oversampled filter banks perform a subband decomposition with redundancy representation. This redundancy has been shown to be useful to combat channel impairments, when the subbands are transmitted over a wireless channel, as well as quantization noise. This paper describes an implementation of the maximum a posteriori and the minimum mean-square error (MMSE) estimators of the input signal from the noisy quantized subbands obtained at the output of some transmission channel. The relations between the input samples and the noisy subband samples are described using a factor graph. Belief propagation is then applied to get the posterior marginals of the input samples. The experimental results show that when the channel is clear, a linear MMSE estimate performs quite well but the proposed approaches perform significantly better than a reconstruction using the linear MMSE estimator when the channel is noisy: a gain in terms of channel SNR of more than 2 dB is observed.

We propose a full reference visual quality metric to evaluate a semantic coding system which may not preserve exactly the position and/or the shape of objects. The metric is based on Scale-Invariant Feature Transform (SIFT) points. More specifically, Structural SIMilarity (SSIM) on windows around the SIFT points measures the compression artifacts (SSIM_SIFT). Conversely, the standard deviation of the matching distance between the SIFT points measures the geometric distortion (GEOMETRIC_SIFT). We validate our metric with subjective evaluation and reach a Spearman correlation of 0.86 for SSIM_SIFT and 0.74 for GEOMETRIC_SIFT.

Many existing works related to lossy-to-lossless image compression are based on the lifting concept. In this paper, we present a sparse optimization technique based on recent convex algorithms and applied to the prediction filters of a two-dimensional non separable lifting structure. The idea consists of designing these filters, at each resolution level, by minimizing the sum of the ℓ1-norm of the three detail subbands. Extending this optimization method in order to perform a global minimization over all resolution levels leads to a new optimization criterion taking into account linear dependencies between the generated coefficients. Simulations carried out on still images show the benefits which can be drawn from the proposed optimization techniques.

Traditional video codecs like H.264/AVC encode video sequences to minimize the Mean Squared Error (MSE)at a given bitrate. Seam carving is a content-aware resizing method. In this paper, we propose a semantic video compression scheme based on seam carving. Its principle is to suppress non salient parts of the video by seam carving. The reduced sequence is then encoded with H.264/AVC and the seams are represented and encoded with our proposed approach. The main idea is to encode the seams by regrouping them. Compared to our earlier work, the main contributions of this paper are: a new energy map with better temporal robustness, a new way to define groups of seams using k-median clustering, and an improved background synthesis. Experiments show that, compared to a traditional H.264/AVC encoding, we reach a bitrate saving between 10% and 24%%with the same quality of the salient objects.

The objective of this paper is to design an efficient bit allocation algorithm in the subband coding context based on an analytical approach. More precisely, we consider the uniform scalar quantization of subband coefficients modeled by a Generalized Gaussian distribution. This model appears to be particularly well-adapted for data having a sparse representation in the wavelet domain. Our main contribution is to reformulate the bit allocation problem as a convex programming one. For this purpose, we firstly define new convex approximations of the entropy and distortion functions. Then, we derive explicit expressions of the optimal quantization parameters. Finally, we illustrate the application of the proposed method to wavelet-based coding systems.

To enable synthesis of any desired intermediate view between two captured views at decoder via depth-image-based rendering (DIBR), both texture and depth maps from the captured viewpoints must be encoded and transmitted in a format known as texture-plus-depth. In this paper, we focus on the compression of depth maps across time to lower the overall bitrate in texture-plus-depth format. We observe that depth maps are not directly viewed, but are only used to provide geometric information of the captured scene for view synthesis at decoder. Thus, as long as the resulting geometric error does not lead to unacceptable synthesized view quality, each depth pixel only needs to be reconstructed at the decoder coarsely within a tolerable range. We first formalize the notion of tolerable range per depth pixel as don't care region (DCR), by studying the synthesized view distortion sensitivity to the pixel value - a sensitive depth pixel will have a narrow DCR, and vice versa. Given per-pixel DCRs, we then modify inter-prediction modes during motion prediction to search for a predictor block matching per-pixel DCRs in a target block (rather than the fixed ground truth depth signal in a target block), in order to lower the energy of the prediction residual for the block. We implemented our DCR-based motion prediction scheme inside H.264; our encoded bitstreams remain 100% standard compliant. We show experimentally that our proposed encoding scheme can reduce the bitrate of depth maps coded with baseline H.264 by over 28%.

The Parallel Proximal Algorithm (PPXA+) has been recently introduced as an efficient tool for solving convex optimization problems. It has proved particularly effective in the context of stereo vision, used as the methodological core of a novel disparity estimation technique. In this work, the main methodological issues limiting the efficient parallelization of this technique are addressed, and further modifications are proposed to enable and optimize the design of parallel implementations. Finally, actual implementations that fit both the multi-core CPU and GPU devices are provided and tested to validate the performance potential of the proposed technique.

Multi-view video streaming is an emerging video paradigm that enables new interactive services, such as free viewpoint television and immersive teleconferencing. However, it comes with a high bandwidth cost, as the equivalent of many single-view streams has to be transmitted. Network coding (NC) can improve the performance of the network by allowing nodes to combine received packets before retransmission. Several works have shown NC to be beneficiai in wireless networks, but the delay introduced by buffering before decoding raises a problem in real-time streaming applications. Here, we propose to use Expanding Window NC (EWNC) for multi-view streaming to allow immediate decoding of the received packets. The order in which the packets are included in the coding window is chosen via RD-optimization for the current sending opportunity. Results show that our approach consistently outperforms both classical NC applied on each view independently and transmission without NC.

This paper considers a multicast scenario and compares the average reception quality obtained when combining multiple description coding (MDC) and network coding (NC). Plain (single description) network coding (NC-SDC) serves as reference. In the considered scenario, a single source is multicast to several receivers with various channel conditions. Contrary to a NC-SDC scheme, unable to recover the coded packets when not enough combinations of packets have been received, NC of MDC packets allows a more progressive quality improvement with the number of received packets, and a reduction of the effect of the quantization noise when MDC is performed via frame expansion before quantization. Considering a probability distribution for the bit transition probability during transmission to any user in the multicast group, the expected signal-to-noise ratio is evaluated. Performance comparisons are made for various error distributions, field sizes, and MDC methods (via frame expansion and correlating transform).

In this paper, we address the challenge of delivering a video stream, encoded with multiple descriptions, in a mobile ad-hoc environment with low-latency constraints. This kind of application is meant to provide an efficient and reliable video communication tool in scenarios where the deployment of an infrastructure is not feasible, such as military and disaster relief applications. First, we present a recently proposed protocol that employs a reliable form of one-hop broadcast to build an efficient overlay network according to a multi-objective function that minimizes the number of packets injected in the network and maximizes the path diversity among descriptions. Then, we introduce the main contribution of this paper: a cross-layer congestion control strategy where the MAC layer is video-coding aware and adjusts its transmission parameters (namely, the RTS retry limit) via congestion/distortion optimization. The main challenge in this approach is providing a reliable estimation of congestion and distortion, given the limited information available at each node. Our simulations show that, if a stringent constraint of low delay is imposed, our technique grants a consistent gain in terms of both PSNR and delay reduction, for bitrates up to a few megabits per second.

This paper addresses the problem of rate distortion analysis in the context of multi-view image coding, where images are predicted via disparity compensation based on depth map. We first present an analytical model for the variance of the residual error in a predicted frame when the prediction is done with the help of a compressed depth map. This residual variance model presents a convenient expression that separates the different error origins (reference frame quantization, depth map coding, and motion activity). We then validate the novel analytical model by testing separately its different underlying hypotheses. Finally, we illustrate an application of our analytical model in a simple bit allocation problem where the objective is to determine the optimal distribution of a global bit budget among reference frame, depth map and disparity-compensated frame. We observe that the optimal allocation given by the analytical model corresponds in practice to the best rate distribution for high bitrate, which confirms the potential of the proposed model in the design of rate-controlled multi-view coding algorithms.

Stereo matching is an active area of research in image processing. In a recent work, a convex programming approach was developed in order to generate a dense disparity field. In this paper, we address the same estimation problem and pro pose to solve it in a more general convex optimization frame work based on proximal methods. More precisely, unlike previous works where the criterion must satisfy some restrictive conditions in order to be able to numerically solve the minimization problem, this work offers a great flexibility in the choice of the involved criterion. The method is validated in a stereo image coding framework, and the results demonstrate the good performance of the proposed parallel proximal algorithm.

In this paper we propose the use of intra-frame prediction with lapped transforms for image coding. Both lapped transforms and intra prediction exploit the redundancies of neighboring blocks and the combination of the two techniques results in a very efficient image coding scheme. The difficulty to combine them comes from the necessity to use for the prediction of the current block pixels in the causal neighborhood that have not been completely processed by the overlapping transform. We show how to overcome this difficulty, and thus the system presented here outperforms the traditional one, using intra-prediction and DCT, and the direct application of lapped transforms in all tested images.

This paper propose the use of lapped transforms in a predictive video coding scheme. Our approach involves an inter-frame, block-based prediction based on lapped trans forms and allows rate-distortion optimization, different block sizes and integration with intra-frame prediction. Moreover, this coding scheme strongly reduces the blocking artifacts associated to block-based transforms and permits a better exploitation of the redundancies beyond block borders. The proposed method presents promising results in comparison to the standard implementation of the H.264/AVC, especially for high-definition sequences.

Multiple-views video is commonly believed to be the next significant achievement in video communications, since it enables new exciting interactive services such as free viewpoint television and immersive teleconferencing. However the interactivity requirement (i.e. allowing the user to change the viewpoint during video streaming) involves a trade-off between storage and bandwidth costs. Several solutions have been proposed in the literature, using redundant predictive frames, Wyner-Ziv frames, or a combination of them. In this paper, we adopt distributed video coding for interactive multiview video plus depth (MVD), taking advantage of depth image based rendering (DIBR) and depth-aided inpainting to fill the occlusion areas. To the authors' best knowledge, very few works in interactive MVD consider the problem of continuity of the playback during the switching among streams. Therefore we survey the existing solutions, we propose a set of techniques for MVD coding and we compare them. As main results, we observe that DIBR can help in rate reduction (up to 13.36% for the texture video and up to 8.67% for the depth map, wrt the case where DIBR is not used), and we also note that the optimal strategy to combine DIBR and distributed video coding depends on the position of the switching time into the group of pictures. Choosing the best technique on a frame-to-frame basis can further reduce the rate from 1% to 6%.

Three dimensional digital video services are gathering a lot of attention in recent years, thanks to the introduction of new and efficient acquisition and rendering devices. In particular, 3D video is often represented by a single view and a so called depth map, which gives information about the distance between the point of view and the objects. This representation can be extended to multiple views, each with its own depth map. Efficient compression of this kind of data is of course a very important topic in sight of a massive deployment of services such as 3D-TV and FTV (free viewpoint TV). In this paper we consider the application of distributed coding techniques to the coding of depth maps, in order to reduce the complexity of single view or multi view encoders and to enhance interactive multiview video streaming. We start from state-of-the-art distributed video coding techniques and we improve them by using high order motion interpolation and by exploiting texture motion information to encode the depth maps. The experiments reported here show that the proposed method achieves a rate reduction up to 11.06% compared to state-of-the-art distributed video coding technique.

With the raise of cloud computing infrastructures on one side and the increased accessibility of parallel computational devices on the other, such as GPUs and multi-core CPUs, parallel programming has recently gained a renewed interest. This is particularly true in the domain of video coding, where the complexity and time consumption of the algorithms tend to limit the access to the core technology. In this work, we focus on the motion estimation problem, well-known to be the most time consuming step of a majority of video coding techniques. By relying on the use of the OpenCL standard, which provides a cross-platform framework for parallel programming, we propose here a scalable CPU/GPU implementation of the full search motion estimation algorithm (FSBM), and study its performances also with respect to the issues raised by the use of OpenCL.

Multiple description coding (MDC) is a framework designed to improve the robustness of video content transmission in lossy environments. In this work, we propose an MDC technique using a legacy coder to produce two descriptions, based on separation of even and odd frames. If only one description is received, the missing frames are reconstructed using temporal high-order motion interpolation (HOMI), a technique originally proposed for distributed video coding. If both descriptions are received, the frames are reconstructed as a block-wise linear combination of the two descriptions, with the coefficient computed at the encoder in a RD-optimised fashion, encoded with a context-adaptive arithmetic coder, and sent as side information. We integrated the proposed technique in a mobile ad-hoc streaming protocol, and tested it using a group mobility model. The results show a non-negligible gain for the expected video quality, with respect to the reference technique.

A wavelet-based multiview video coding scheme is presented in this paper. It uses a 4-D wavelet transform, which is composed of a 1-D temporal wavelet transform, namely motion compensated temporal filtering, a 1-D view-directional wavelet transform, namely disparity compensated view filtering and a 2-D spatial wavelet transform. Since the presented framework can make use of the inherent scalability properties of the wavelet transforms involved, it allows full scalability of the coded bitstream in the temporal, view, spatial, and quality dimensions. Coding performance close to the H.264/advanced video coding based standard multiview video codec is shown. Enhancements of the view transform, in order to better account for brightness and color variations across views are introduced. Additionally, the use of a signal adaptive anisotropic wavelet packet (WP) transform as a generalization of WP transforms for the spatial decomposition is proposed. Both enhancements lead to a decrease of bit rate of up to 11% compared with the baseline version of the codec.

Many existing works related to lossy-to-lossless image compression are based on the lifting concept. However, it has been observed that the separable lifting scheme structure presents some limitations because of the separable processing performed along the image lines and columns. In this paper, we propose to use a 2D non separable lifting scheme decomposition that enables progressive reconstruction and exact decoding of images. More precisely, we focus on the optimization of all the involved decomposition operators. In this respect, we design the prediction filters by minimizing the variance of the detail signals. Concerning the update filters, we propose a new optimization criterion which aims at reducing the inherent aliasing artefacts. Simulations carried out on still and stereo images show the benefits which can be drawn from the proposed optimization of the lifting operators.

The Laplacian model is the standard distribution for correlation noise estimation at the turbodecoder in Wyner-Ziv coding schemes. In practice, this hypothesis is not always satisfied and, regularly, the estimated model sensibly differs from the error distribution. In this work, we prove that using a model better fitted to the true distribution improves the performances, and we thus propose to use the more general exponential power distribution (EPD) which has never been tested in a distributed video coding context. Gains in rate-distortion over the Laplacian model are illustrated by results on several video sequences, showing that the EPD model outperforms the Laplacian one in off-line (oracle) as well as in on-line (practical implementation) modes. These results also indicate that, in some cases, the online EPD model reduces the bitrate even over the off-line Laplacian model.

A key step in distributed video coding is the generation of the side information (SI) i.e. the estimation of the Wyner-Ziv frame (WZF). This step is also frequently called image interpolation. State-of-the-art techniques perform a motion estimation between adjacent key frames (KFs) and linear interpolation in order to assess object positions in the WZF, and then the SI is produced by motion compensating the KFs. However the uniform motion model underlying this approach is not always able to produce a satisfying estimation of the motion, which can result in a low SI quality. In this paper we propose a new method for the generation of SI, based on higher order motion interpolation. We use more than two KFs to estimate the position of the current WZF block, which allows us to correctly estimate more complex motion (such as, for example, uniform accelerated motion). We performed a number of tests for the fine tuning of the parameters of the method. Our experiments show that the new interpolation technique has a small computational cost increase with respect to state of the art, but provides remarkably better performance with up to 0.5 dB of PSNR improvement in SI quality. Moreover the proposed method performs consistently well for several GOP sizes.

In this paper, we propose a joint source-channel (JSC) decoding scheme for 3D ESCOT-based video coders, such as Vidwav. The embedded bitstream generated by such coders is very sensitive to transmission errors unavoidable on wireless channels. The proposed JSC decoder employs the residual redundancy left in the bitstream by the source coder combined with bit reliability information provided by the channel or channel decoder to correct transmission errors. When considering an AWGN channel, the performance gains are in average 4 dB in terms of PSNR of the reconstructed frames, and 0.7 dB in terms of channel SNR. When considering individual frames, the obtained gain is up to 15 dB in PSNR.

Compressed sensing is applied to multiview image sets and inter-image disparity compensation is incorporated into image reconstruction in order to take advantage of the high degree of inter-image correlation common to multiview scenarios. Instead of recovering images in the set independently from one another, two neighboring images are used to calculate a prediction of a target image, and the difference between the original measurements and the compressed-sensing projection of the prediction is then reconstructed as a residual and added back to the prediction in an iterated fashion. The proposed method shows large gains in performance over straightforward, independent compressed-sensing recovery. Additionally, projection and recovery are block-based to significantly reduce computation time.

In a multiview-imaging setting, image-acquisition costs could be substantially diminished if some of the cameras operate at a reduced quality. Compressed sensing is proposed to effectuate such a reduction in image quality wherein certain images are acquired with random measurements at a reduced sampling rate via projection onto a random basis of lower dimension. To recover such projected images, compressed-sensing recovery incorporating disparity compensation is employed. Based on a recent compressed-sensing recovery algorithm for images that couples an iterative projection-based reconstruction with a smoothing step, the proposed algorithm drives image recovery using the projection-domain residual between the random measurements of the image in question and a disparity-based prediction created from adjacent, high-quality images. Experimental results reveal that the disparity-based reconstruction significantly outperforms direct reconstruction using simply the random measurements of the image alone.

Joint source-channel decoding (JSCD) exploits residual redundancy in compressed bitstreams to improve the robustness to transmission errors of multimedia coding schemes. This paper proposes an architecture to introduce some additional side information in compressed streams to help JSCD. This architecture exploits a reference decoder already present or introduced at the encoder side. An application to the robust decoding of 3D-ESCOT encoded bitstreams generated within the Vidwav video coder is presented. The layered bitstream generated by this encoder allows SNR scalability, and moreover, when processed by a JSCD, provides increased robustness to transmission errors compared with a single layered bitstream.

Depth Image Based Rendering (DIBR) technique has been recognized as a promising tool for supporting advanced 3D video services required in MultiView Video (MVV) systems. However, an inherent problem with DIBR is to fill newly exposed areas (holes) caused by disocclusions. This paper addresses the disocclusion problem. To deal with small disocclusions, hole-filling strategies have been designed by the state-of-the-art through pre-processing techniques of the depth video. For larger disocclusions, where depth pre-processing has some limitations, we propose an inpainting approach to retrieve missing pixels. Specifically, we propose in the texture and structure propagation process to take into account the depth information by distinguishing foreground and background parts of the scene. Experimental results illustrate the efficiency of the proposed method.

Multiple description coding is a framework adapted to noisy transmission environments. In this work, we use H.264 to create two descriptions of a video sequence, each of them assuring a minimum quality level. If both of them are received, a suitable algorithm is used to produce an improved quality sequence. The key technique is a temporal image interpolation using motion compensation, inspired to the distributed video coding context. The interpolated image blocks are weighted with the received blocks obtained from the other description. The optimal weights are computed at the encoder and efficiently sent to the decoder as side information. The proposed technique shows a remarkable gain for central decoding with respect to similar methods available in the state of the art.

Side information construction in Wyner-Ziv video coding is a sensible task which strongly influences the final ratedistortion performance of the scheme. This side information is usually generated through an interpolation of the previous and next images. Some of the zones of a scene however, such as the occlusions, cannot be estimated with other frames. In this paper we propose to avoid this problem by sending some hash information for these unpredictable zones of the image. The resulting algorithm is described and tested here. The obtained results show the advantages of using localized hash information for the high error zones in distributed video coding.

Side information generation is a critical step in distributed video coding systems. This is performed by using motion compensated temporal interpolation between two or more key frames (KFs). However, when the temporal distance between key frames increases (i.e. when the GOP size becomes large), the linear interpolation becomes less effective. In a previous work we showed that this problem can be mitigated by using high order interpolation. Now, in the case of long duration GOP, state-of-the-art algorithms propose a hierarchical algorithm for side information generation. By using this procedure, the quality of the central interpolated image in a GOP is consistently worse than images closer to the KFs. In this paper we propose a refinement of the central WZFs by higher order interpolation of the already decoded WZFs, that are closer to the WZF to be estimated. So we reduce the fluctuation of side information quality, with a beneficial impact on final rate-distortion characteristics of the system. The experimental results show an improvement on the SI up to 2.71 dB with respect the state-of-the-art and a global improvement of the PSNR on the decoded frames up to 0.71 dB and a bit rate reduction up to 15%.

Compressed sensing is applied to multiview image sets and the high degree of correlation between views is exploited to enhance recovery performance over straightforward independent view recovery. This gain in performance is obtained by recovering the difference between a set of acquired measurements and the projection of a prediction of the signal they represent. The recovered difference is then added back to the prediction, and the prediction and recovery procedure is repeated in an iterated fashion for each of the views in the multiview image set. The recovered multiview image set is then used as an initialization to repeat the entire process again to form a multistage refinement. Experimental results reveal substantial performance gains from the multistage reconstruction.

Differential motion estimation produces dense motion vector fields which are far too demanding in terms of coding rate in order to be used in video coding. However, a pel-recursive technique like that introduced by Cafforio and Rocca can be modified in order to work using only the information available at the decoder side. This allows to improve the motion vectors produced in the classical predictive modes of H.264. In this paper we describe the modification needed in order to introduce a differential motion estimation method into the H.264 codec. Experimental results will validate a coding mode, opening new perspectives in using differential-based motion estimation techniques into classical hybrid codecs.

This paper addresses the problem of video compression for robust transmission on lossy Internet networks. The approach developed relies on multiple description coding (MDC) principles. Predictive multiple description video coding has already been considered for robust video transmission over lossy channels. However, MDC, when combined with motion-compensated prediction, is known to suffer from predictive mismatch: in presence of losses, the prediction signal available at the decoder may differ from the one used at the encoder. In this paper, we describe a video compression scheme based on MDC and Wyner-Ziv (WZ) coding. The input sequence is structured into groups of pictures which contain one key frame and one WZ frame. Each frame is first transformed with a wavelet transform and the resulting frequency bands are quantized with a multiple description scalar quantizer. Two balanced descriptions of the video input are thus generated. The quantization indexes of the WZ frames are coded with an low-density parity-check accumulate-based Slepian-Wolf (SW) coder. The lateral receivers first decode the received key frame lateral descriptions, and then construct the side information needed for decoding the corresponding WZ frame descriptions by motion-compensated interpolation. In the case where the two WZ data descriptions are received, the central decoder can perform a separate SW decoding of the two sequences of quantization indexes. A joint iterative decoding approach of the two WZ descriptions is also described which improve the central WZ data decoding performance, however, at the expense of increased decoding complexity. The influence of the proposed iterative decoding technique and of the amount of redundancy on the lateral and central rate-distortion performance of the algorithm is studied.

In a typical competition-based coding, the pertinence of a prediction mode does not only depend on its own efficiency but also on the fact that it is complementary with the other modes. The method proposed in this paper to improve the intra coding of the H.264/AVC standard relies on this remark; it shows how the cost of signaling predictors that are quite similar can be avoided. Indeed, at low bitrates, the information related to the predictor signaling in intra coding reaches up to 25% of the total bitrate for the whole set of standard VCEG test sequences. In order to reduce this cost, a method reproducible at the decoder side is proposed to eliminate some predictors from the intra predictor set. The proposed method exploits the proximity of the predictors in the transform domain in order to obtain a representative and non-redundant set of predictors.

Motion estimation methods based on differential techniques proved to be very useful in the context of video analysis, but have a limited employment in classical video compression because, though accurate, the dense motion vector field they produce requires too much coding resource and computational effort. On the contrary, this kind of algorithm could be useful in the framework of distributed video coding (DVC). In this paper we propose a differential motion estimation algorithm which can run at the decoder in a DVC scheme, without requiring any increase in coding rate. This algorithm allows a performance improvement in image interpolation with respect to state-of-the-art algorithms.

In this paper, we propose the use of multiple description coding to increase the robustness of distributed video coding while keeping good rate-distortion performance. The video sequence is structured into key frames and Wyner-Ziv frames. For each type of frame, two descriptions are generated by a multiple description scalar quantizer and sent on a loss-prone channel. When both Wyner-Ziv descriptions are received, they are jointly decoded along with the side information. We investigate the influence of the amount of redundancy and of the iterative decoding of the descriptions on the performance.

Distributed video coding (DVC) is a recent paradigm which aims at transferring part of the coding complexity from the encoder to the decoder. The performance of such a coding scheme strongly depends on the capacity to estimate correlation at the decoder and, consequently, on the side information quality. In this paper we consider a multi-view DVC framework and propose a very efficient dense disparity estimation technique for side information construction, based on a variational formulation. The simulation results show that our approach clearly outperforms the existing methods for inter-view side-information generation.

In this paper, we present a variational framework for joint disparity and motion estimation in a sequence of stereo images. The problem involves the estimation of four dense fields: two motion fields and two disparity fields. In order to reduce computational complexity and improve estimation accuracy, the two motion fields, for the left and right sequences, and the disparity field of the current stereo pair are jointly estimated, using the stereo-motion consistency constraint. In the proposed variational framework, the joint estimation problem is formulated as a convex programming problem in which a convex objective function is minimized under specific convex constraints. This minimization is achieved using an efficient parallel block-iterative algorithm. Experimental results involving real stereo sequences indicate the feasibility and robustness of our approach.

Motion estimation (ME) methods based on differential techniques provide useful information for video analysis, and moreover it is relatively easy to embed into them regularity constraints enforcing for example, contour preservation. On the other hand, these techniques are rarely employed for video compression since, though accurate, the dense motion vector field (MVF) they produce requires too much coding resource and computational effort. However, this kind of algorithm could be useful in the framework of distributed video coding (DVC), where the motion vector are computed at the decoder side, so that no bit-rate is needed to transmit them. Moreover usually the decoder has enough computational power to face with the increased complexity of differential ME. In this paper we introduce a new image interpolation algorithm to be used in the context of DVC. This algorithm combines a popular DVC technique with differential ME. We adapt a pel-recursive differential ME algorithm to the DVC context; moreover we insert a regularity constraint which allows more consistent MVFs. The experimental results are encouraging: the quality of interpolated images is improved of up to 1.1 dB w.r.t. to state-of-the-art techniques. These results prove to be consistent when we use different GOP sizes.

Research in stereo image coding has focused on the disparity estimation/compensation process to exploit the cross-view redundancies. Most of the reported methods use a classical block-based technique in order to estimate the disparity field. However, this estimation technique does not always provide an accurate disparity map, which may affect the disparity compensation step. In this paper, we propose to use an estimation method that produces a dense and smooth disparity map. Then, on the one hand, this map is segmented and efficiently coded by exploiting the high correlation between neighboring disparity values. On the other hand, we integrate the disparity information into a vector lifting scheme for stereo image coding. Experimental results indicate that the proposed coding scheme outperforms the conventional methods employing a block-based disparity estimation.

This paper proposes a video streaming system optimizing resource utilization when the media server only disposes of long term feedbacks from the client. Based on a partial knowledge of the network, we developed a scheduling algorithm that exploits the scalable video coding (SVC) properties to estimate packets importance and that takes into account packet delay dependencies to better anticipate congestion situations. Compared to more conventional streaming systems, experimental results show that our approach allows to better face network condition degradation like bandwidth reduction or packet error rate increase.

This paper presents a novel algorithm for content adaptive GOP size control in distributed video coding. The GOP size is dynamically varied along the sequence, depending on motion activity. Automatic mode selection allows the system to switch between H.264 intra-coding and Wyner-Ziv coding modes to optimize the overall performance. Furthermore, the encoder determines a suitable compression ratio for the Wyner-Ziv frames without the need for a feedback channel. Simulation results show significant improvement in the average system performance, compared to fixed GOP Wyner-Ziv and H.264 intra-coding.

This work aims at improving the quality of the side information in distributed video coding. Based on genetic algorithms, our proposed technique combines several frames, interpolated using previously developed methods, in a fusion-based approach. Simulation results show a significant improvement in the side information quality compared to other interpolation techniques available in the literature, which greatly improves the rate-distortion performance of a distributed video codec, where the gain in PSNR can reach 6 dB.

Multiview video coding is an emerging application where, in addition to classical temporal prediction, an efficient disparity prediction should be performed in order to achieve the best compression performance. A popular coder is the multiview video coding (MVC) extension of H.264/AVC, which uses a block-based disparity estimation (just like temporal prediction in H.264/AVC). In this paper, we propose to improve the MVC extension by using a dense estimation method that generates a smooth disparity map with ideally infinite precision. The obtained disparity is then segmented and efficiently encoded by using a rate-distortion optimization technique. Experimental results show that significant gains can be obtained compared to the block-based disparity estimation technique used in the MVC extension.

The lifting scheme represents an easy way of implementing the wavelet transform and of constructing new content-adapted transforms. However, the adaptive version of lifting schemes can result in strongly non-isometric transforms. This can be a major limitation, since all most successful coding techniques rely on the distortion estimation in the transform domain. In this paper we focus on the problem of evaluating the reconstruction distortion (due to quantization noise) in the wavelet domain when a non-isometric adaptive-prediction lifting scheme is used. The problem arises since these transforms are nonlinear, and so common techniques for distortion evaluation cannot be used in this case. We circumvent the difficulty by computing an equivalent time-varying linear filter, for which it is possible to generalize the distortion computation technique. In addition to the theoretical formulation of the distortion estimation, in this paper we provide experimental results proving the reliability of this estimation, and the consequent improvement of RD performance, thanks to a more effective resource allocation which can be performed in the transform domain.

This paper presents an adaptation of peer-to-peer diffusion strategies of video over mobile links based on the use of multiple description coding. The system architecture, including the used overlay algorithm (the unstructured GIA algorithm) and video coding with multiple description coding (MDC), are presented as well as their interest in this context. Simulation results illustrate the approach, showing the interest of using MDC vs. a single description coding solution. The proposed simulation framework involves an H.264/AVC video codec with two descriptions and a simple adaptation of the overlay algorithm.

Many research efforts have been devoted to the improvement of stereo image coding techniques for storage or transmission. In this paper, we are mainly interested in lossy-to-lossless coding schemes for stereo images allowing progressive reconstruction. The most commonly used approaches for stereo compression are based on disparity compensation techniques. The basic principle involved in this technique first consists of estimating the disparity map. Then, one image is considered as a reference and the other is predicted in order to generate a residual image. In this paper, we propose a novel approach, based on vector lifting schemes (VLS), which offers the advantage of generating two compact multiresolution representations of the left and the right views. We present two versions of this new scheme. A theoretical analysis of the performance of the considered VLS is also conducted. Experimental results indicate a significant improvement using the proposed structures compared with conventional methods.

We consider the uniform scalar quantization of a class of mixed distributed memoryless sources, namely sources having a Bernoulli Generalized Gaussian (BGG) distribution. Both for low and high resolutions, asymptotic expressions of the distortion for a pth-order moment error measure, and close approximations of the entropy are provided for these sources. Operational rate-distortion functions at high bit rate and their slope factors at low bit rate are derived. The dependence of these results on p and the distribution parameters as well as the relation to the Shannon optimal rate-distortion bound are then discussed. The application of these results to transform coding in two simple cases is finally highlighted.

In this paper, we provide operational rate-distortion results for memoryless generalized Gaussian sources. Close approximations of the entropy are provided for these sources, after a uniform scalar quantization at low/high resolution. Asymptotic expressions of the distortion for an arbitrary p-th order error measure are also given. The resulting approximations at low/high bitrate of the operational rate-distortion function are thus compared with the Shannon optimal bound showing the overall good performance of uniform quantization rules.

An approach to scalable multi-view video coding with joint best basis wavelet packets is examined in this paper. A 4-D wavelet transform is used to decorrelate the multi-view video data temporally, view-directionally, and spatially for efficient scalable compression. Motion compensated temporal filtering (MCTF) is used for temporal, and disparity compensated view filtering (DCVF) for view- directional decomposition. Adaptive wavelet packets as a generalized wavelet decomposition are presented for spatial decomposition. Two algorithms to find the best basis wavelet packets are evaluated and compared with classical dyadic wavelet transform: a low complexity entropy based best basis search and a search algorithm in a rate-distortion framework. In both cases, the joint best basis is determined for a group of frames rather than for each frame individually. Therefore, the rate to spend for the tree description is minimal while advantage is taken of the similarity of frames within a temporal-view-directional subband.

Multi-view video and 3D television are emerging applications raising the problem of efficient encoding of a depth map, in addition to classical texture images. This paper investigates depth image coding via an adaptive wavelet lifting scheme. Switching between long filters in homogeneous areas and short filters over the edges of the depth map is decided based on the contours detected in the texture image. The method takes thus into consideration the correlation existing between the edges in the texture and in the depth image, leading to an improved encoding of the latter one.

Abstract—This paper addresses the problem of dense disparity estimation from a pair of color stereo images. Based on a convex set theoretic formulation, the stereo matching problem is cast as a convex programming problem in which a color-based objective function is minimized under specific convex constraints. These constraints arise from prior knowledge and rely on various properties of the disparity field to be estimated. The resulting multi-constrained optimization problem is solved via an efficient parallel block-iterative algorithm. Four different color spaces have been tested in order to evaluate their suitability for stereo matching. Experiments on standard stereo images show that the matching results have been efficiently improved when using color information instead of grey values. I.

In this paper we study the problem of evaluating the reconstruction distortion in the wavelet domain when adaptive lifting schemes (ALS) are used for the direct and inverse transform. The distortion evaluation is necessary in order to perform efficient resource allocation over the transform coefficients. ALS is a non-linear transformation, which prevents using common techniques for distortion evaluation. However we show the equivalence of this non-linear scheme with a time-varying linear filter, and we generalize the distortion computation technique to it. Experiments show that the proposed method allows a reliable estimation of the distortion in the transform domain. This results in improved coding performance.

In this paper, we present a rate-distortion analysis for a distributed video coding system, which allows for the selection of a suitable quantization parameter for the H.264 coded key frames, under rate constraints. On the other hand, an adaptive rate allocation and quantization algorithm with a frame dropping strategy is used for the transmission of Wyner-Ziv frames in a multi-user scenario, without the need for a permanent feedback channel. The available bandwidth is dynamically shared between Wyner-Ziv and Key frames. Turbo-codes are used for the joint source-channel coding of the Wyner-Ziv frames and for the protection of the H.264 bitstream from channel errors. Simulation results show the superior performance of the proposed technique, compared to traditional systems with fixed quantization parameters and compression rates.

The H.264/MPEG4-AVC video coding standard has achieved a higher coding efficiency compared to its predecessors. The significant bitrate reduction is mainly obtained by efficient motion compensation tools, as variable block sizes, multiple reference frames, 1/4-pel motion accuracy and powerful prediction modes (e.g., SKIP and DIRECT). These tools have contributed to an increased proportion of the motion information in the total bit- stream. To achieve the performance required by the future ITU-T challenge, namely to provide a codec with 50% bitrate reduction compared to the current H.264, the reduction of this motion information cost is essential. This paper proposes a competing framework for better motion vector coding and SKIP mode. The predictors for the SKIP mode and the motion vector predictors are optimally selected by a rate-distortion criterion. These methods take advantage from the use of the spatial and the temporal redundancies in the motion vector fields, where the simple spatial median usually fails. An adaptation of the temporal predictors according to the temporal distances between motion vector fields is also described for multiple reference frames and B-slices options. These two combined schemes lead to a systematic bitrate saving on Baseline and High profile, compared to an H.264/MPEG4-AVC standard codec, which reaches up to 45%.

The H.264/MPEG4-AVC video coding standard has achieved a higher coding efficiency compared to its predecessors. The significant bit rate reduction is mainly obtained by efficient motion compensation tools, as variable block sizes, multiple reference frames, 1/4-pel motion accuracy and powerful prediction modes (e.g., SKIP and DIRECT). These tools have contributed to an increased proportion of the motion information in the total bit stream. To achieve the performance required by the future ITU-T challenge, namely to provide a codec with 50% bit rate reduction compared to the current H.264, the reduction of this motion information cost is essential.

In this paper, we consider a two description coding scheme based on a general frame synthesis operator. Through some approximations of the original rate-distortion problem, the design of the efficiently encoded coefficients is formulated as a convex optimization problem. We also show that there exists a close link between the proposed coding strategy and compressed sensing problems. Simulations results are provided to show the validity of our approach.

We propose an adaptive wavelet transform that results in fewer large detail coefficients while preserving image contours in the approximation subband. The transform is constructed with a three-step nonlinear lifting scheme: a fixed prediction followed by a space-varying update and a non-additive prediction. The interest of the proposed scheme is demonstrated for nearly lossless compression.

The geometric features of images, such as edges, are difficult to represent. When a redundant transform is used for their extraction, the compression challenge is even more difficult. In this paper we present a new rate-distortion optimization algorithm based on graph theory that can encode efficiently the coefficients of a critically sampled, non-orthogonal or even redundant transform, like the contourlet decomposition. The basic idea is to construct a specialized graph such that its minimum cut minimizes the energy functional. We propose to apply this technique for rate-distortion Lagrangian optimization in subband image coding. The method yields good compression results compared to the state-of-art JPEG2000 codec, as well as a general improvement in visual quality.

Depth image-based rendering (DIBR) is the process of synthesizing some new "virtual" views from one "real" view and the associated per-pixel depth information. The most important problem in this process is to deal with the newly exposed areas (holes) appearing in the virtual images. One common solution to decrease the number of holes is to pre-process the depth map, before the warping. In this paper, we present a new filtering technique for depth image-based rendering. In order to reduce or completely remove the newly exposed areas an efficient smoothing is necessary for the sharp depth changes near object boundaries. In the meantime it is useless to filter the smooth areas in the depth map. Our solution is based on a weighted Gaussian filter taking into account the distance to the contours. By this way, the geometric distortions and the computation time are reduced compared to a uniform filtering of the depth map. We present some results in the context of creation of stereoscopic views for 3D TV.

Transform image coding and more particularly the subclass of block based transformations are widely used to compress images. The JPEG standard for still images and MPEG codec specifications for video are very efficient implementations, but these algorithms perform image reconstruction without taking into account the quantization operations performed on the transform coefficients. In this paper, we propose an adaptive algorithm to tune the inverse transformation matrix as a function of the quantization level in order to minimize the reconstruction error. The developed algorithm provides quality scalability features and also integrates resizing operations into the inverse transformation process leading to a spatial scalability of fractional factors.

H.264/MPEG4-AVC is the latest video codec provided by the Joint Video Team, gathering ITU-T and ISO/IEC experts. Technically there are no drastic changes compared to its predecessors H.263 and MPEG-4 part 2. It however significantly reduces the bitrate and seems to be progressively adopted by the market. The gain mainly results from the addition of efficient motion compensation tools, variable block sizes, multiple reference frames, 1/4-pel motion accuracy and powerful Skip and Direct modes. A close study of the bits repartition in the bitstream reveals that motion information can represent up to 40% of the total bitstream. As a consequence reduction of motion cost is a priority for future enhancements. This paper proposes a competition-based scheme for the prediction of the motion. It impacts the selection of the motion vectors, based on a modified rate-distortion criterion, for the Inter modes and for the Skip mode. Combined spatial and temporal predictors take benefit of temporal redundancies, where the spatial median usually fails. An average 7% bitrate saving compared to a standard H.264/MPEG4-AVC codec is reported. In addition, on the fly adaptation of the set of predictors is proposed and preliminary results are provided.

Three-dimensional (t+2D) wavelet coding schemes have been demonstrated to be efficient techniques for video compression applications. However, the separable wavelet transform used for removing the spatial redundancy allows a limited representation of the 2D texture because of spatial isotropy of the wavelet basis functions. In this case, anisotropic transforms, such as fully separable wavelet transforms (FSWT), can represent a solution for spatial decorrelation. FSWT inherits the separability, the computational simplicity and the filter bank characteristics of the standard 2D wavelet transform, but it improves the representation of directional textures, as the ones which can be found in temporal detail frames of t + 2D decompositions. The extension of both classical wavelet and wavelet-packet transforms to fully separable decompositions preserve at the same time the low-complexity and best-bases selection algorithms of these ones. We apply these transforms in t + 2D video coding schemes and compare them with classical decompositions.

A joint source-channel coding scheme for transmission of video over flat Rayleigh fading channels is described. The coding scheme consists of a spatiotemporal motion-compensated wavelet decomposition, a vector quantization of the coefficients through maximum-diversity lattices, and a linear labeling which minimizes simultaneously the source and channel distortion. Modulation diversity via rotated constellations produces the maximum-diversity lattices which increase robustness to channel fading without the addition of redundancy. Experimental results compare the proposed system to a prominent scalable video coder protected by more traditional convolutional codes, and superior performance is observed for high levels of channel noise.

3D video codecs have attracted recently a lot of attention, due to their compression performance comparable with that of state-of-art hybrid codecs and due to their scalability features. In this work, we propose a least mean square (LMS) based adaptive prediction for the temporal prediction step in lifting implementation. This approach improves the overall quality of the coded video, by reducing both the blocking and ghosting artefacts. Experimental results show that the video quality as well as PSNR values are greatly improved with the proposed adaptive method, especially for video sequences with large contrast between the moving objects and the background and for sequences with illumination variations

A simple and computationally lightweight video coder employing shape-adaptive, embedded intraframe coding and wavelet-domain conditional replenishment is proposed. Robustness to packet losses arises from packetization of the embedded bitstream with unequal error protection which is assigned to the packets with a fast, locally optimal procedure. Experimental results reveal that, when compared to H.264/AVC configured for low-complexity, error-resilient operation, not only does the proposed coder usually produce substantially superior rate-distortion performance as packet losses increase, it also achieves a significantly faster encoding speed.

In an H.264 video encoder, the important task of intra coding is performed in the residual domain after a spatial transform has been applied to the block data. Although the intra prediction is very efficient and shows the ability to capture some orientations of the image, after this spatial prediction the intra residual may still show regular structures. To take this possible regularity into account, especially orientation, we propose a preprocessing stage that selects the best rotation to be applied to the block among a set of predefined rotations. To this end, we utilize a rate-distortion algorithm. We further propose to calculate the real orientation of the block to decrease the amount of computation. Experimental results show that compared to the standard intra-prediction mode of H.264, the proposed method can gain up to 1 dBs

Three-dimensional (3D) wavelet coding schemes compression applications. The scalability property of such schemes is one of the most important issues for video surveillance systems. In this paper we introduce a new lifting-based method of temporal decomposition which provides a scalability factor of 5 in a motion-compensated subband video coding scheme. Depending on the sequence characteristics, motion model etc., this structure can provide high coding performance. Also, it gives a better leading to an improved temporal scalability. It addresses video surveillance applications, where the motion is very low in most cases

A scalable video coder consisting of motion-compensated temporal filtering coupled with structured vector quantization plus a linear mapping of quantizer indexes that minimizes simultaneously source and channel distortions is presented. The linear index assignment takes the form of either a direct, uncoded mapping or a coded mapping via Reed-Muller codes. Experimental results compare the proposed system to a similar scheme using unstructured vector quantization as well as to a prominent scalable video coder protected by more traditional convolutional codes. The proposed system consistently outperforms the other two schemes by a significant margin for very noisy channel conditions.

Subband image coding with classical wavelet transforms inherently provides dyadic scalability since the low-pass approximation band can be used as a half-sized version of the original image. The multiresolution approach further provides scalability factors that are powers of two, but no other factors can be easily reached. On the other hand, a hierarchical scheme based on M-bandtransforms will lead only to scalability factors that are powers of M. In this letter, we present how an M-band synthesis filter bank can be modified in order to directly reconstruct an output image of a resolution reduced by a factor M/P, where P is an integer lower than M. As the analysis filter bank is not modified, this technique allows the design of subband decoding schemes that support fractional scalability. With this feature, the same compressed bitstream can be directly decoded by different receivers at several resolutions, depending on their computational and bandwith capabilities.

Recent breakthroughs in motion-compensated temporal wavelet filtering have finally enabled implementation of highly efficient scalable and error-resilient video codecs. These new wavelet codecs provide numerous advantages over nonscalable conventional solutions techniques based on motion-compensated prediction, such as no recursive predictive loop, separation of noise and sampling artifacts from the content through use of longer temporal filters, removal of long range as well as short range temporal redundancies, etc. Moreover, these wavelet video coding schemes can provide flexible spatial, temporal, signal-to-noise ratio and complexity scalability with fine granularity over a large range of bit rates, while maintaining a very high coding efficiency. However, most motion-compensated wavelet video schemes are based on classical two-band decompositions that offer only dyadic factors of temporal scalability. In this paper, we propose a three-band temporal structure that extends the concept of motion-compensated temporal filtering (MCTF) that was introduced in the classical lifting framework. These newly introduced structures provide higher temporal scalability flexibility, as well as improved compression performance compared with dyadic Haar MCTF.

The paper presents a solution of temporal scalability for video encoded H.264/MPEG-4 AVC bitstreams. Achieved through the concept of adaptive M-band hierarchical filterbanks, the temporal scalability is performed thanks to the application of a frame shuffling operation which allows backward compatibility with the standard to be kept. Simulation results show that this scalability is obtained with no degradation in terms of subjective and objective quality.

The paper presents the design of a multivariate statistical approach for multicomponent image denoising in the wavelet transform domain. We extend an approach that we have recently proposed, where the wavelet coefficients of all the image channels at the same spatial position, in a given orientation and at the same resolution level, are grouped into a vector, and a multivariate Bernoulli-Gaussian distribution is used as a prior model. The paper develops low-complexity maximum a posteriori rules that exploit jointly the intra- and interscale redundancies between the wavelet coefficients. Experimental results carried out on remote sensing multispectral images show that the proposed procedure improves the state-of-the-art wavelet-based denoising methods.

A joint source-channel coding scheme of t + 2D decomposed video sequences and an iterative bit allocation are presented. The joint source-channel coding scheme consists of a vector quantization and a linear labelling by lattice constellations minimizing simultaneously the channel and the source distortion. The channel distortion, due to the linear labelling, is minimized and depends on the value of the noise variance and the variance of the source. The iterative algorithm results in an optimal codebook allocation subject to a global bit rate and a nonnegativity constraint. The overall flexible coding scheme is proved to be very efficient in noisy environments.

The objective of this paper is the design of adaptive quincunx lifting schemes for lossless compression of multiband images. More precisely, the operators of the lifting scheme are modified according to the local activity of the multivariate input signal. To this respect, a block-based adaptive strategy is adopted: the image is partitioned into a quadtree structure and a couple of optimal operators is assigned to each resulting volumetric segmented block. Our main contribution consists of a suitable quadtree partitioning rule that takes into account simultaneously the spatial and spectral redundancies. Simulations performed on real satellite images show that the proposed adaptive method outperforms the conventional non-adaptive lifting schemes.

In wavelet-based scalable video coding schemes, temporal interframe redundancy is exploited by applying a temporal wavelet transform along the motion trajectories. A spatial decomposition of the temporal subbands is further performed to take advantage of the spatial redundancy of the filtered frames. However, most of the t+2D video codecs do not take into account the very different spatial characteristics of the temporal subband frames and use indifferently the same spatial 9/7 biorthogonal transform to decompose them. In this paper, we present a spatial transform based on a four-band filter bank, whose frequency selectivity properties are shown to be more suited to represent detail frames. We give the analytical form of a linear-phase, orthogonal and regular four-band filter bank and we show by experimental results conducted on video sequences that significant improvements in terms of PSNR can be obtained using the proposed filter bank to decompose the detail frames.

Multiple description coding (MDC) recently appeared as a joint source-channel coding technique specifically designed for real-time multimedia applications over best effort switched packet networks such as the Internet, in order to cope with packet losses due to transmission errors or network congestion. In this paper we compare several redundant wavelet decompositions within the framework of multiple description of scalable video coding. Special attention is paid to the optimal design of the central decoder. Simulation results are provided for motion-compensated filter banks so as to evaluate the efficiency of the central and side decoding strategies. Compared with other techniques, a key factor of the proposed analysed schemes is their reduced redundancy factor.

Joint source-channel coding of scalable video using motion-compensated temporal filtering is considered. The proposed coding scheme consists of a structured vector quantizer based on lattice constellations and a linear index assignment, which minimizes simultaneously the channel and source distortions. Both uncoded linear index assignment as well as partially coded linear index assignment via Reed-Muller codes are considered. The proposed system is compared to an unstructured quantizer with minimax index assignment. Simulation results indicate that, for a Gaussian channel, the structured-codebook scheme is very robust, maintaining near-noiseless performance even when the channel is very noisy. Additionally, the proposed structured-quantizer scheme outperforms its unstructured counterpart when channel noise levels are high.

Some of the most powerful schemes for scalable video coding are based on the so-called (t + 2D) paradigm and provide coding performance competitive with state-of-the-art codecs. However, the temporal multiresolution schemes used in such codecs introduce a non-negligible delay, preventing their use in applications which require low latency, like video conferencing or video streaming. In this paper, we provide a flexible approach to reduce the end-to-end delay involved in motion-compensated temporal filtering schemes. We show by simulations results the trade-offs observed between coding efficiency and low end-to-end delay.

We consider the problem of image transmission on error-prone networks with little or no error protection. To this end, we build a multiple description scheme based on classical biorthogonal filter banks, that achieves good reconstruction even at low bitrates. The novelty of the proposed approach mainly consists in building a 2D wavelet frame representation with low redundancy. Another contribution of this work is the use of a convex optimization approach at the decoder end in order to best take advantage of all the received information. The quantization constraints define convex sets, which allow us to apply fast iterative projection techniques to find a feasible solution of the decoding problem.

We present a new class of adaptive wavelet decompositions that can capture the directional nature of picture information. Our method exploits the properties of seminorms to build lifting structures able to choose between different update filters, the choice being triggered by a local gradient of the input. In order to discriminate between different geometrical information, the system makes use of multiple criteria, giving rise to multiple choice of update filters. We establish the conditions under which these decisions can be recovered at synthesis, without the need for transmitting overhead information.

Motion-compensated temporal filtering (MCTF) is an essential ingredient of the recently developed wavelet-based scalable video coding schemes. Lifting implementation of these decompositions represents a versatile tool for spatiotemporal optimizations, and several improvements have already been proposed in this framework. In this paper, we propose new spatiotemporal update operators based either on weighted averages of multiple connected pixels or on nonlinear filtering of these pixels. We demonstrate the improved performance of the proposed operators, both theoretically and experimentally.

We present a framework for the systematic analysis of video encoding complexity, measured in terms of the number of motion estimation (ME) computations, that we illustrate on motion compensated wavelet video coding schemes. We demonstrate the graceful complexity scalability of these schemes through the modification of the spatiotemporal decomposition structure and the ME parameters, and the use of spatiotemporal prediction. We generate a wide range of rate-distortion-complexity (R-D-C) operating points for different sequences, by modifying these options. Using our analytical framework we derive closed form expressions for the number of ME computations for these different coding modes and show that they accurately capture the computational complexity independent of the underlying content characteristics. Our framework for complexity analysis can be combined with rate-distortion modeling to determine the encoding structure and parameters for optimal R-D-C tradeoffs.

We present a joint source-channel coding scheme developed for transmitting scalable video over a noisy channel. This scheme is based on a 2D+t motion-compensated wavelet decomposition of the video sequence, on a vector quantization and on an optimal linear labelling. In P. Knagenhjelm and E. Agrell's work (IEEE Trans. Information Theory, vol. IT 42, pp. 1139-1151, 1996), it is shown that on binary discrete channels, the channel distortion is minimized by linear labelling. In this paper, we provide a vector quantization method dependent on a linear labelling which minimizes not only the channel distortion but at the same time the distortion of our video source. Moreover, by exploiting a statistical model of the spatio-temporal wavelet coefficients we adapt the vector quantizer to the non-stationarity of the source. Simulation results demonstrate the robustness of this scheme under noisy channel conditions.

Motion-compensated temporal filtering is an essential ingredient of recently developed wavelet-based scalable video coding schemes. The lifting implementation of these decompositions represents a versatile tool for spatio-temporal optimizations and numerous improvements have thus been proposed. An additional feature has been introduced lately by providing a 3-band lifting temporal decomposition, allowing non-dyadic scalability factors. The paper presents a motion-compensated scheme which is the 3-band equivalent of the 5/3 temporal filterbank. The proposed structure does not enter the classical lifting framework and therefore its invertibility is studied. Simulation results show higher coding efficiency, improved visual quality and more flexibility compared to the previously introduced Haar-like 2 or 3-band schemes.

Motion-compensated temporal filtering is an essential ingredient of recently developed wavelet-based scalable video coding schemes. Lifting implementation of these decompositions represents a versatile tool for spatio-temporal optimizations and numerous improvements have thus been proposed. In this paper, we propose an alternative structure for the temporal prediction in the 5/3 filterbank. It significantly reduces the ghosting artefacts in the temporal approximation subband frames, providing a higher quality scalability and improved compression performance, for an equivalent complexity.

Spatio-temporal motion-compensated wavelet decomposition is an increasingly popular method for scalable video coding, with coding efficiency which is competitive with state-of-the-art nonscalable codecs. In this paper, we propose a new spatio-temporal update operator in the lifting scheme allowing efficient implementation of these temporal decompositions. We demonstrate its improved performance both theoretically, by exhibiting a decrease in the reconstruction error and by simulation results.

We present a joint source-channel coding scheme developed for transmitting scalable video on a Rayleigh fading channel. Our scheme consists of a motion compensated temporal subband decomposition applied on groups of frames, followed by a spatial wavelet transform of the temporal subbands and a vector quantization. The vector quantization method is based on a linear labelling by "maximum component diversity" lattice constellations which minimizes not only the channel distortion but at the same time the distortion of our video source. These lattice constellations, which are multidimensional rotations generated from totally real cyclotomic fields, increase the diversity order on a Rayleigh fading channel. In addition, a rate-compatible punctured convolutional code in combination with a perfect interleaving is applied, providing even higher diversity.

Multiple description coding (MDC) is a joint source-channel coding technique specifically designed for real-time multimedia applications over best effort switched packet networks (such as Internet), in order to cope with packet losses due to transmission errors or network congestion. Error resilience of transmitted bitstreams is thus significantly increased, but this does not solve problems like bitstream adaptation to bandwidth variations or receiver characteristics, which are in turn addressed by scalable coding techniques. In this paper, we present a new method of multiple description coding of scalable video, combining the scalability features with MDC. We propose a redundant motion-compensated temporal scheme related to Haar multiresolution analysis. We also present an equivalent lifting implementation leading to simple central and lateral decoders.

Scalable video coding via motion-compensated spatio-temporal wavelet decompositions has gained a great interest in transmission over heterogeneous networks due to the flexibility of the resulting bitstream to accomodate various network conditions as well as user capabilities and demands. Meanwhile, the adaptation of the bitstream to the available bandwidth can lead to discarding the finest detail subbands during the transmission. The loss of these subbands would result in a low quality, oversmoothed reconstructed sequence. In this paper, we present a statistical spatio-temporal model between the wavelet coefficients and we show its efficiency in the prediction of the high frequency subbands and in the quality enhancement of the scalable video.

We propose applying an adaptive context-tree weighting (CTW) method in the H.264 video coders. We first investigate two different ways to incorporating the CTW method into an H.264 coder and compare the coding effectiveness of using the method with that of using the context models specified in the H.264 standard. We then describe a novel approach for automatically adapting the CTW method based on the syntactic element to be coded and the encoding parameters. We show that our CTW-based arithmetic coding method yields similar or better compression results compared with the context-based adaptive arithmetic coding method used in H.264, without having to specify so many context models.

The motion-compensated temporal filtering is an essential part of a scalable wavelet-based video coding scheme, which applies a temporal wavelet transform in the motion direction over the frames of a video sequence. The lifting structure of the temporal filter bank tackled in this paper involves a predict operator which makes use of two motion vector fields to bidirectionally predict frames from their neighbouring ones. We show in this paper that there exists an optimal algorithm allowing to estimate jointly these two motion vector fields subject to an optimization criterion directly related to the coding of the detail subbands. We provide an iterative suboptimal form of this algorithm implementing this approach. We show that this algorithm provides substantial gains in terms of PSNR, with the same complexity as a separate estimation of the two motion vector fields.

Classical linear wavelet representations of images have the drawback that they are not well suited to represent edge information. To overcome this problem, nonlinear multiresolution decompositions are being designed that can take into account the characteristics of the input signal/image. In our previous work [(G. Piella et al., July 2002), (H.J.A.M. Heijmans et al., 2002)] we have introduced an adaptive lifting framework that does not require bookkeeping but has the property that it processes edges and homogeneous regions in an image in a different fashion. The current paper discusses the effects of quantization in such adaptive wavelet decomposition. We provide conditions for recovering the original decisions at the synthesis and for relating the reconstruction error to the quantization error. Such an analysis is essential for the application of these adaptive decompositions in image compression algorithms.

Recently, a new generation of video compression techniques gained the interest of the research engineering community, by proposing state-of-the-art compression performance in addition to temporal/spatial/SNR scalability features. They exploit interframe redundancy through a temporal wavelet transform in the motion direction. While classical two-band decompositions offer dyadic factors of scalability, in this paper we present a three-band temporal lifting structure providing a factor 3 of scalability in a motion-compensated subband video coding scheme. More flexibility is thus available in order to adapt the bitstream to the bandwidth variations in heterogeneous networks. We also show improved compression performance compared with a dyadic Haar MCTF analysis.

Motion compensated temporal filtering (MCTF) is used in wavelet video coding schemes to remove temporal redundancy. Current extensions to MCTF use bi-directional filtering and multiple reference frames to improve the coding performance significantly. However, these extensions lead to an additional cost of coding extra sets of motion vectors. In this paper we propose the use of temporal prediction to reduce the bits needed to code motion vectors. We describe several temporal prediction methods, and highlight the advantages of each scheme. We also use MV prediction during motion estimation, i.e. change the search center and the search range based on the temporal prediction, and describe the tradeoffs to be made between rate, distortion, and complexity. We perform several experiments to illustrate the gains of using temporal prediction, and identify the content dependent nature of such results.

This paper treats a class of adaptive update lifting schemes which do not require bookkeeping for perfect reconstruction. The choice of the update lifting filter is triggered by a binary threshold criterion based on a generalised gradient which is chosen in such a way that it only smoothes homogeneous regions. More precisely, the criterion can be chosen so that it ‘recognises’ signals that are polynomial up to a given order. The lifting filter updates the signal in these polynomial regions but leaves other parts unaffected.

The paper discusses a method for the construction of nonlinear 2D wavelet decompositions using an adaptive update lifting scheme. A very interesting aspect is that the decomposition does not require any bookkeeping, i.e., it is nonredundant, but that it, nevertheless, allows perfect reconstruction. The major ingredient of the construction is the so-called decision map which triggers the choice of the update filter. Another interesting point is the possibility of better preserving edges, even at low resolutions.

This letter treats a class of adaptive update-lifting schemes that do not require bookkeeping for perfect reconstruction. The choice of the update-lifting filter is triggered by a binary threshold criterion based on a generalized gradient that is chosen in such a way that it only smooths homogeneous regions. This criterion can be chosen so that it ignores portions of a signal that are polynomial up to a given order. The update-lifting filter modifies the signal in these polynomial regions but leaves other portions unaffected.

Three-dimensional wavelet decompositions are efficient tools for scalable video coding. We show a lifting formulation for these decompositions. The temporal wavelet transform is inherently nonlinear, due to the motion estimation step, and the lifting formalism allows us to provide several improvements to the scheme initially proposed by Choi and Woods: a better processing of the uncovered areas is proposed and an overlapped motion-compensated temporal filtering method is introduced in the multiresolution decomposition. As shown by simulations, the proposed method results in higher coding efficiency, while keeping the scalability functionalities.

Multimedia transmissions over heterogeneous networks require a high degree of flexibility from video compression systems. They are expected to be fully scalable, that is to say to be able to partly decode a video bitstream and to get a reconstruction quality proportional to the received amount of information. To achieve this functionality, we propose a video codec based on 2D+t subband decomposition. Groups of frames are first temporally filtered using motion compensation and then spatially decomposed with wavelets. The spatiotemporal coefficients are further scanned and compressed using a new SPIHT-like strategy, namely fully scalable zerotree coding, together with arithmetic encoding, which provides a combination of temporal, spatial and SNR scalability. In addition, scalable motion vector coding ensures a fully progressive bitstream.

We propose a separable decomposition approximating the Karhunen-Loeve transform for random fields. We show that this problem is related to a joint singular value decomposition of a set of matrices and we provide an efficient algorithm to compute it. Finally, we illustrate the interest of this new tool for image representation and approximation.

We address the problem of context modeling and coding in the framework of a three dimensional wavelet-based video compression scheme with temporal filtering in the motion direction. Embedded coding algorithms based on a 3D decomposition (like the 3D SPIHT algorithm) provide scalability, which is an important feature for delivering compressed bitstreams over heterogeneous networks. We propose a highly scalable video compression scheme, based on an efficient contextual modeling and a hierarchical encoding of the spatio-temporal trees of wavelet coefficients. We have shown that the efficiency of the final arithmetic encoding stage is increased by performing a statistical modeling based on the context tree weighting method. In order to reflect changes in the statistical behaviors of the different information sources that contribute to the bitstream, several statistical models have been distinguished. As shown by simulations, the proposed method results in significant improvements of the coding efficiency, while keeping the scalability functionality.

Nonstationary two-dimensional random fields with wide-sense stationary fractional increments are introduced using a fractional order for their increments in two directions of space. Some of their properties are studied and their value in modeling nonstationary, possibly anisotropic, fields is shown via examples. A separable wavelet packet analysis proves to be a useful tool in analyzing this kind of texture as it leads to stationary wavelet packet coefficients.

The problem of linear interpolation of nonstationary multidimensional processes with stationary increments is studied. The expressions for the interpolation filters and for the estimation error are derived, which generalize the results of the interpolation theory for stationary processes. Both finite and infinite extent interpolation are considered. An application to the interpolation of an underwater depth map is presented.