Web Real-Time Communication or Realtime communication in the Web (WebRTC/RTCWeb) is a prolific new standard and technology stack, providing full audio/video agnostic communications for the Web. Service providers implementing such technology deal with various levels of complexity ranging anywhere from: high service distribution, multi-client integration, P2P and Cloud assisted communication backends, content delivery, real-time constraints and across clouds resource allocation. This work presents a study of the joint factors including multi-cloud distribution, network performance, media parameters and back-end resource loads, in Cloud based Media Selective Forwarding Units for WebRTC infrastructures. The monitored workload is sampled from a large population of real users of our testing infrastructure, additionally the performance data is sampled both by passive user measurements as well as server side measurements. Patterns correlating such factors enable designing adaptive resource allocation algorithms and defining media Service Level Objectives (SLO) spanning over multiple data-centers or servers. Based on our analysis, we discover strong periodical load patterns even though the nature of user interaction with the system is mostly not predetermined with variable user churn.

Current telepresence systems, while being a great step forward in videoconferencing, still have important points to improve in what eye-contact, gaze and gesture awareness concerns. Many-to-many communications are going to greatly benefit from mature auto-stereoscopic 3D technology; allowing people to engage more natural remote meetings, with proper eye-contact and better spatiality feeling. For this purpose, proper real-time multi-perspective 3D video capture is necessary (often based on one or more View+Depth data sets). Given current state of the art, some sort of foreground segmentation is often necessary at the acquisition in order to generate 3D depth maps with hight enough resolution and accurate object boundaries. For this, one needs flicker-less foreground segmentations, accurate to borders, resilient to noise and foreground shade changes, and able to operate in real-time on performing architectures such as GPGPUs. This paper introduces a robust Foreground Segmentation approach used within the experimental immersive 3D Telepresence system from EU-FP7 3DPresence project. The proposed algorithm is based on a costs minimization using Hierarchical Believe Propagation and outliers reduction by regularization on oversegmented regions. The iterative nature of the approach makes it scalable in complexity, allowing it to increase accuracy and picture size capacity as GPGPUs become faster. In this work, particular care in the design of foreground and background cost models has also been taken in order to overcome limitations of previous work proposed in the literature.

This paper discusses the problem of high quality depth map estimation for real-time systems. Our work is based on the European FP7 project 3DPresence which aims to build a multi-view and multiuser 3D videoconferencing system. Based on new multi-view auto-stereoscopic display technology the remote conferees will be rendered as an integral part of a three dimensional virtual shared environment. In order to create the related views for the 3D displays as well as to virtually correct the eye contact problem robust depth maps are required. For this purpose, in this paper we will discuss the fusion of two competing approaches which have, from a camera configuration point of view, contrary to each other properties. Namely, we will combine the volumetric Visual Hull (VH) approach with the stereo matching based Hybrid Recursive Matching (HRM) to a new method which benefits from the advantages of both techniques and discards their weak points.

Over the years, many works have been published on the two-dimensional foreground segmentation task, describing different methods that treat to extract that part of the scene containing active entities. In most of the cases, the stochastic background process for each pixel is modeled first, and then the foreground pixels are classified as an exception to the model or using maximum a posteriori (MAP) or maximum likelihood (ML). The shadow is usually removed in a later stage and salt and pepper noise is treated with connected component analysis or mathematical morphology. In this paper, we propose a global method that classifies each pixel by finding the best possible class (foreground, background, shadow) examining the image globally. A Markov random field is used to represent the dependencies between all the pixels and classes and the global optimal solution is approximated with the belief propagation algorithm. The method can extend most local methods and increase their accuracy. In addition, this approach brings a probabilistic justification of the classification problem and it avoids the use of additional post-processing techniques.

3D technologies are becoming the more and more relevant in recent years. Visual communications, as well as image and video analysis, benefit in great manner from spatial information such as depth for various applications. Highly accurate visual depth estimation often involves complex optimization algorithms in order to fit proper estimation models to data. From a stereo/multiview matching perspective, local and global algorithms exist. Commonly, the latter are more complex and accurate, as data models are used to take the global structure into account. Belief Propagation has proven to be a good global algorithmic framework for depth estimation. By means of an iterative procedure, it is able to regularize, according to set of local smoothness and geometry constrains, an initial estimation of depth by a local approach such as simple block matching. However, information transfer from iteration to iteration by means of message passing can be excessively demanding in terms of memory bandwidth and usage. In this paper, a new Belief Propagation based algorithm with multiview matching with depth/color segmentation is proposed together with a strategy for message passing compression. Experimental results show the algorithm to be competitive with best performing ones in the state of the art, while reducing by a factor 10 the memory usage, with marginal loss in performance, of a typical Belief Propagation strategy.

In recent years, works on geometric multidimensional signal representations have established a close relation with signal expansions on redundant dictionaries. For this purpose, matching pursuits (MP) have shown to be an interesting tool. Recently, most important limitations of MP have been underlined, and alternative algorithms like weighted-MP have been proposed. This work explores the use of weighted-MP as a new framework for motion-adaptive geometric video approximations. We study a novel algorithm to decompose video sequences in terms of few, salient video components that jointly represent the geometric and motion content of a scene. Experimental coding results on highly geometric content reflect how the proposed paradigm exploits spatio-temporal video geometry. Two-dimensional weighted-MP improves the representation compared to those based on 2-D MP. Furthermore, the extracted video components represent relevant visual structures with high saliency. In an example application, such components are effectively used as video descriptors for the joint audio-video analysis of multimedia sequences.

Recent video coding strategies, such as H.264/AVC, incorporate an in-loop deblocking filter in order to reduce the effects of quantization noise. These techniques are limited to treating blocky artifacts on smooth regions. In order to solve this, sparsity-based filtering techniques have been recently proposed for efficient filtering of edge and textured areas. More recently, direction-adaptive sparsity-based filtering has also allowed to exploit directional features on intra encoded pictures. This paper proposes a high-performance in-loop filter for deartifacting intra and inter encoded video data. This work extends for Inter frames the use of direction-adaptive sparsity-based filtering techniques and it improves performance by adaptively selecting filtering thresholds consistent with quantization noise statistics, local encoding conditions, compression requirements and the original signal. Thresholds are both spatially and temporally adapted to optimize video quality and/or coding cost. Selected thresholds are encoded and transmitted as side information to the decoder. Experimental results show significant bit rate savings and visual quality enhancement when compared to the state-of-the- art H.264/AVC codec using in-loop deblocking filtering.

Modern video coding standards such as H.264/AVC use intra prediction for efficient coding of Intra pictures. These usually exploit local directional signal correlations. More recently, intra prediction modes using non-local signal information have been introduced. A very popular approach is the so called template matching prediction (TMP), which uses template based texture synthesis for signal prediction. This, combined with regular directional prediction, significantly improves intra coding efficiency compared to H.264/AVC. However, current TMP techniques have trouble synthesizing picture data with non-uniform illumination characteristics. They assume that similar picture regions resemble at the same time in structure and illumination, which is often not the case. In order to solve this, we propose a template matching technique with locally adaptive illumination compensation. The proposed technique is based on a linear compensation model with a scaling and an offset parameters to compensate for contrast and brightness disparities respectively. The model parameters are calculated by solving an auto-regressive Least Square problem during the template search for TMP. This permits to synthesize signal structures while capturing the local characteristics of illumination without needing extra side information. The total improvement in intra coding efficiency with respect to H.264/AVC can be of up to 18%.

Frame partitioning is a process of key importance in efficient video coding. Most recent video compression technologies, like H.264/AVC, use tree based frame partition. This reveals to be more efficient than simple uniform block partition, typically used in older video coding standards like MPEG-2 or H.263. However, tree based frame partition still does not code efficiently enough video information, as is unable to capture the geometric structure of 2D data. During last years, several works have been developed, mainly in the domain of still image representation and coding, in order to solve such limitations. An example is the use of wedge partitions. Based on these, in this paper, we study a way to better represent and code 2D video data by taking its 2D geometry into account. Our study is developed as an extension of H.264/AVC. Geometry-adaptive partitions are used to improve intra and inter prediction modes. Results obtained with the investigated method show that both better R-D and visual performance can be achieved.

Many modern video coding strategies, such as the H.264/AVC standard, use quadtree-based partition structures for coding intra macroblocks. Such a structure allows the coding algorithm to adapt to the complicated and non-stationary nature of natural images. Despite the adaptation flexibility of quadtree partitions, recent studies have shown that these are not efficient enough (in terms of rate-distortion performance) when images can be locally modeled as 2D piecewise-smooth signals. These observations motivate us to investigate the use of geometry based block partitioning for modeling intra data in video coding. In particular, in this paper, we study in detail the use of geometry-adaptive intra models, where wedgelet like discontinuities are used in order to define separate coding regions where different statistical/waveform modeling tools can be used. In order to implement this idea, we extend the existing H.264/AVC intra coding scheme by introducing two additional geometric modes: INTRA16X16GEO, and INTRA8X8GEO. Experimental results show that significantly improved R-D performance is achieved.

Modern video compression codecs achieve high compression efficiency by exploiting the temporal and spatial redundancies present in video sequences. Although state of the nit block based intra and inter prediction have been refined to better adapt to the content in the scene, they still exhibit limitations in fully extracting information from the decoded data. We propose the addition of an in-loop prediction refinement stage that is initialized with the conventional intra or inter predicted result and is capable of further reducing the spatial redundancies present in the sequence. By extracting additional information from the decoded neighborhood, the refinement stage is able to bring the prediction closer to the original data, thus improving compression efficiency. In this work, we study one possible method of refinement for the particular case of intra prediction. This uses sparse decomposition of blocks that overlap decoded neighboring blocks and the current predicted block in order to make prediction more coherent with already encoded data.

This paper explores a novel approach for ventricular and atrial activities estimation in electrocardiogram (ECG) signals, based on sparse source separation. Sparse decompositions of ECG over signal-adapted multi-component dictionaries can lead to natural separation of its components. In this work, dictionaries of functions adapted to ventricular and atrial activities are respectively defined. Then, the weighted orthogonal matching pursuit algorithm is used to unmix the two components of ECG signals. Despite the simplicity of the approach, results are very promising, showing the capacity of the algorithm to generate realistic estimations of atrial and ventricular activities.

In this work we explore the potentialities of a framework for the representation of audio-visual signals using decompositions on overcomplete dictionaries. Redundant decompositions may describe audio-visual sequences in a concise fashion, preserving good representation properties thanks to the use of redundant, well designed, dictionaries. We expect that this helps us overcome two typical problems of multimodal fusion algorithms. On one hand, classical representation techniques, like pixel-based measures (for the video) or Fourier-like transforms (for the audio), take into account only marginally the physics of the problem. On the other hand, the input signals have large dimensionality. The results we obtain by making use of sparse decompositions of audio-visual signals over redundant codebooks are encouraging and show the potentialities of the proposed approach to multimodal signal representation.

In this work, we explore a framework for the sparse representation of video sequences by means of spatio-temporal functions able to exploit the 2D nature of images and the temporal smoothness associated to object trajectories. Decomposition over redundant dictionaries formed by 2D functions capable to exploit image geometry, has shown to be well adapted for efficient sparse image approximations. Video representation by means of temporally evolving sets of such 2D functions seems thus a natural extension toward video approximation techniques. In the present paper we study the deformation of a geometry oriented image expansion based on matching pursuits (MP) (P. Vandergheynst and P. Frossard, May 2001), to obtain a parametric representation of frames transformation through time. We consider a modified MP approach based on a Bayesian decision criteria to deform geometrical primitives in a predictive fashion from frame to frame. Indeed, since motion stability is not guaranteed using a pure MP, a Bayesian framework is introduced to regularize motion among expansion terms of frames representations.

In this paper, we introduce a framework that merges classical ideas borrowed from scale-space and multiresolution segmentation with nonlinear partial differential equations. A non-linear scale-space stack is constructed by means of an appropriate diffusion equation. This stack is analyzed and a tree of coherent segments is constructed based on relationships between different scale layers. Pruning this tree proves to be a very efficient tool for unsupervised segmentation of different classes of images (e.g., natural, medical, etc.). This technique is light on the computational point of view and can be extended to nonscalar data in a straightforward manner.

The presented work extends the concept of sub-band video coding based on a 3D wavelet transform to a more adaptive approach. A formal comparison is presented between the performances inferred by the use of the 3D wavelet transform and the use of a 2D wavelet in the spatial domain extended by a locally adaptive transform in the temporal dimension. Some advantages are foreseen for the new scheme since it is able to deal better with certain signal models, like appearing and moving edges. An increased control of distortion spreading is expected and, consequently, a lower visual impact relevance.

Today's video codecs employ motion compensated prediction in combination with block matching techniques. These techniques, although achieving some level of adaptivity in their latest versions, continue to rely on the decomposition of frames on a set of artificial primitives: blocks. This paper presents a new approach to video coding. A geometrically adaptive image decomposition scheme using an over-complete basis is used to represent the scene. Using Matching Pursuit (MP), we are able to express local features such as position, anisotropic scale and orientation in terms of a set of spatio-frequential primitives. In order to perform frame prediction, only the changes in the parameters that determine these functions from frame to frame will have to be coded. Such an approach, in addition to being able to catch displacements in images deals as well in a natural way with local scale deformations and local rotations.

Today's video codecs employ motion compensated prediction in combination with block matching techniques. These techniques, although achieving some level of adaptivity in their latest versions, continue to rely on the decomposition of frames on a set of artificial primitives: blocks. This paper presents a new approach to video coding. A geometrically adaptive image decomposition scheme using an over-complete basis is used to represent the scene. Using matching pursuit (MP), we are able to express local features such as position, anisotropic scale and orientation in terms of a set of spatio-frequential primitives. In order to perform frame prediction, only the changes in the parameters that determine these functions from frame to frame will have to be coded. Such an approach, in addition to being able to catch displacements in images deals as well in a natural way with local scale deformations and local rotations.

An MPEG-7 camera extends the capabilities of conventional cameras by analyzing its scene in order to generate a content-based description according to the recently approved MPEG-7 standard. This gives to the camera a large variety of current and potential applications, such as surveillance, augmented reality, and virtual display. This paper provides an overview of what is meant by an MPEG-7 camera, discusses the above mentioned applications, and provides an implementation example of such a camera using existing hardware products.