Real-time Distributed Media Processing Workflows (DMPW) are popular for online media delivery. Combining distributed media sources and processing can reduce storage costs and increase flexibility. However, high request rates may result in unacceptable latency or even failures in incorrect configurations. Thus, testing DMPW deployments at scale is key, particularly for real-time cases. We propose the new MPEG Network Based Media Processing (NBMP) standard for this and present a testbed implementation that includes all the reference components. In addition, the testbed includes a set of configurable functions for load generation, monitoring, data-collection and visualization. The testbed is used to test Dynamic Adaptive HTTP streaming functions under different workloads in a standardized and reproducible manner. A total of 327 tests with different loads and Real-Time DMPW configurations were completed. The results provide insights in the performance, reliability and time-consistency of each configuration. Based on these tests, we selected the preferred cloud instance type, considering hypervisor options and different function implementation configurations. Further, we analyzed different processing tasks and options for distributed deployments on edge and centralized clouds. Last, a classifier was developed to detect if failures happen under a certain workload. Results also show that, normalized inter-experiment standard deviation of the metric means can be an indicator for unstable or incorrect configurations.

Achieving low-latency is paramount for live streaming scenarios, that are now-days becoming increasingly popular. In this paper, we propose a novel algorithm for bitrate adaptation in HTTP Adaptive Streaming (HAS), based on Online Convex Optimization (OCO). The proposed algorithm, named Learn2Adapt-LowLatency (L2A-LL), is shown to provide a robust adaptation strategy which, unlike most of the state-of-the-art techniques, does not require parameter tuning, channel model assumptions, throughput estimation or application-specific adjustments. These properties make it very suitable for users who typically experience fast variations in channel characteristics. The proposed algorithm has been implemented in DASH-IF's reference video player (dash.js) and has been made publicly available for research purposes at [22]. Real experiments show that L2A-LL reduces latency significantly, while providing a high average streaming bit-rate, without impairing the overall Quality of Experience (QoE); a result that is independent of the channel and application scenarios. The presented optimization framework, is robust due to its design principle; its ability to learn and allows for modular QoE prioritization, while it facilitates easy adjustments to consider applications beyond live streaming and/or multiple user classes.

An open source implementation of the CMAF live media ingest specification developed by the DASH Industry Forum is presented. CMAF live ingest provides a protocol for live uplink streaming of content using the Common Media Application Track Format (CMAF) and HTTP POST. An example source, an example receiver and additional tooling for CMAF metadata conversion are included. The tools can be used to perform live ingest of CMAF track files to selected destinations in real-time. The reference receiver can store and process CMAF tracks into a streaming media presentation. The additional tools for conversion enable creation of timed metadata tracks. The distribution also includes sample CMAF track files with aligned audio, video, timed text and metadata content. The tools can be used for emulating live sources and receivers when investigating workflows based on the CMAF live media ingest protocol. We demonstrate usage of the tools in relevant streaming use cases showing performance benefits compared to ingest using a reverse proxy cache.

HTTP Adaptive Streaming (HAS) is increasingly deployed at large, gradually replacing traditional broadcast. However, testing large-scale deployments remains challenging, costly and error-prone. Especially, testing with realistic streaming loads from massive numbers of users is challenging and costly. To improve this, we introduce an open-source load testing tool that can be deployed in the cloud or on-premise in a distributed manner, for load generation.

Most media streaming services are composed by different virtualized processing functions such as encoding, packaging, encryption, content stitching etc. Deployment of these functions in the cloud is attractive as it enables flexibility in deployment options and resource allocation for the different functions. Yet, most of the time overprovisioning of cloud resources is necessary in order to meet demand variability. This can be costly, especially for large scale deployments. Prior art proposes resource allocation based on analytical models that minimize the costs of cloud deployments under a quality of service (QoS) constraint. However, these models do not sufficiently capture the underlying complexity of services composed of multiple processing functions. Instead, we introduce a novel methodology based on full-stack telemetry and machine learning to profile virtualized or cloud native media processing functions individually. The basis of the approach consists of investigating 4 categories of performance metrics: throughput, anomaly, latency and entropy (TALE) in offline (stress tests) and online setups using cloud telemetry. Machine learning is then used to profile the media processing function in the targeted cloud/NFV environment and to extract the most relevant cloud level Key Performance Indicators (KPIs) that relate to the final perceived quality and known client side performance indicators. The results enable more efficient monitoring, as only KPI related metrics need to be collected, stored and analyzed, reducing the storage and communication footprints by over 85%. In addition a detailed overview of the functions behavior was obtained, enabling optimized initial configuration and deployment, and more fine-grained dynamic online resource allocation reducing overprovisioning and avoiding function collapse. We further highlight the next steps towards cloud native carrier grade virtualized processing functions relevant for future network architectures such as in emerging 5G architectures.

In this article, we have partially benchmarked Hammer, a real-world end-to-end network traffic simulator, and demonstrated some of its capabilities on testing actual platforms deployed in production-like networking environments. Hammer was integrated to a complex testbed and proved capable of identifying the operational breaking point or the target node, as well as obtaining several metrics for its behavior. The tool supports an immense number of use cases for functional, stress or capacity testing and offers inherent acceleration in the data plane, thus delivering high performance levels. Its resource-savvy nature and modular design, offers excellent scalability and performs significantly better with latest generation CPUs, indicating the software's suitability for high-demanding tasks when installed in contemporary servers in bare-metal.

We present Unified Remix, our solution for adaptive bit-rate streaming of video presentations with inserted or edited content. The solution addresses three important challenges encountered when streaming personalized media presentations. First, it reduces vulnerability to ad blocking technologies and client-side playback deviations encountered when using manifest manipulation based methods. Second, it reduces storage and computational costs associated with alternative server side solutions such as brute force re-encoding or duplicate storage towards levels comparable to linear video streaming (VoD or Live). Third, it handles the multi-source, multi-DRM and multi-protocol aspects for modern video streaming natively in the workflow. The solution is based on a combination of existing proven streaming technologies such as Unified Origin and newly designed components such as the Remix MPEG-4 module. The framework uses standardized technologies such as MPEG-4 ISOBMFF, SMIL and MPEG-DASH. The components work together in a micro service architecture enabling flexible deployment using a (container) orchestration framework on premises or in the cloud. The solution is demonstrated in two use cases: content pre-/post/mid roll and Live Archive to VoD conversion. As many use cases can be implemented based upon Unified Remix, we envision it as a key component of professional video streaming platforms.

Compression of dynamic point clouds for streaming and broadcast is a big challenge arising in mixed and virtual reality applications. To advance this area, common performance benchmarking and assessment is key as to compare different technologies consistently. This paper presents a methodology for performance assessment that was established in the MPEG point cloud compression activity in a collaboration of leading companies and academic institutes. It includes selected datasets, objective metrics and target coding rates corresponding to distinguishable subjective qualities and different use cases. The performance assessment methodology is applied to a compression framework on a large dataset developing an anchor performance benchmark. The benchmark results and methodology are used in the recent MPEG Call for proposals on point cloud coding to consistently compare and evaluate technologies.

We present a generic and real-time time-varying point cloud codec for 3D immersive video. This codec is suitable for mixed reality applications in which 3D point clouds are acquired at a fast rate. In this codec, intra frames are coded progressively in an octree subdivision. To further exploit interframe dependencies, we present an inter-prediction algorithm that partitions the octree voxel space in N × N × N macroblocks (N = 8, 16, 32). The algorithm codes points in these blocks in the predictive frame as a rigid transform applied to the points in the intra-coded frame. The rigid transform is computed using the iterative closest point algorithm and compactly represented in a quaternion quantization scheme. To encode the color attributes, we defined a mapping of color per vertex attributes in the traversed octree to an image grid and use legacy image coding method based on JPEG. As a result, a generic compression framework suitable for realtime 3D tele-immersion is developed. This framework has been optimized to run in real time on commodity hardware for both the encoder and decoder. Objective evaluation shows that a higher rate-distortion performance is achieved compared with available point cloud codecs. A subjective study in a state-of-the-art mixed reality system shows that introduced prediction distortions are negligible compared with the original reconstructed point clouds. In addition, it shows the benefit of reconstructed point cloud video as a representation in the 3D virtual world. The codec is available as open source for integration in immersive and augmented communication applications and serves as a base reference software platform in JTC1/SC29/WG11 (MPEG) for the further development of standardized point-cloud compression solutions.

Summary form only given. We present an overview of the MPEG activity on Point Cloud Compression (PCC). This activity aims to satisfy the growing need for lossless and lossy PCC in industry. It follows recent advances in consumer electronics in 3D reconstruction and capturing. We present the preliminary architecture, test datasets and the test model reference software implementation. Contributions beyond the state of the art in PCC include lossy coding of color attributes and lossy inter-predictive coding.

We present MP3DG-PCC, an open source framework for design, implementation and evaluation of point cloud compression algorithms. The framework includes objective quality metrics, lossy and lossless anchor codecs, and a test bench for consistent comparative evaluation. The framework and proposed methodology is in use for the development of an international point cloud compression standard in MPEG. In addition, the library is integrated with the popular point cloud library, making a large number of point cloud processing available and aligning the work with the broader open source community.

In practice, video delivery using multiple protocols is needed to support the wide range of clients in the digital media ecosystem. Naive approaches increase backhaul traffic and cache storage requirements proportional to the number of protocols in use. To reduce this overhead, we present an efficient multi-protocol video delivery architecture. It exploits the fact that media segments in different protocols are often based on the same raw encoded media data. Instead of caching and distributing near duplicate media segments for each protocol throughout the content delivery network, we push their generation towards the edge. We call this Late Trans-Muxing (LTM). We implement LTM in a smart edge cache that can request and cache media byte ranges and generate protocol specific media segments on the fly. Experiments in an emulation testbed show that backhaul traffic and cache storage are reduced up to 85% and 50% respectively compared to CDN based approaches. This shows the benefit of LTM in multi-protocol DASH based video delivery.

Geometry based 3D Tele-Immersion is a novel emerging media application that involves on the fly reconstructed 3D mesh geometry. To enable real-time communication of such live reconstructed mesh geometry over a bandwidth limited link, fast dynamic geometry compression is needed. However, most tools and methods have been developed for compressing synthetically generated graphics content. These methods achieve good compression rates by exploiting topological and geometric properties that typically do not hold for reconstructed mesh geometry. The live reconstructed dynamic geometry is causal and often non-manifold, open, non-oriented and time-inconsistent. Based on our experience developing a prototype for 3D Teleimmersion based on live reconstructed geometry, we discuss currently available tools. We then present our approach for dynamic compression that better exploits the fact that the 3D geometry is reconstructed and achieve a state of art rate-distortion under stringent real-time constraints.

REVERIE (REal and Virtual Engagement in Realistic Immersive Environments [1]) targets novel research to address the demanding challenges involved with developing state-of-the-art technologies for online human interaction. The REVERIE framework enables users to meet, socialise and share experiences online by integrating cutting-edge technologies for 3D data acquisition and processing, networking, autonomy and real-time rendering. In this paper, we describe the innovative research that is showcased through the REVERIE integrated framework through richly defined use-cases which demonstrate the validity and potential for natural interaction in a virtual immersive and safe environment. Previews of the REVERIE demo and its key research components can be viewed at www.youtube.com/user/REVERIEFP7.

3-D tele-immersion (3DTI) enables participants in remote locations to share, in real time, an activity. It offers users interactive and immersive experiences, but it challenges current media-streaming solutions. Work in the past has mainly focused on the efficient delivery of image-based 3-D videos and on realistic rendering and reconstruction of geometry-based 3-D objects. The contribution of this paper is a real-time streaming component for 3DTI with dynamic reconstructed geometry. This component includes both a novel fast compression method and a rateless packet protection scheme specifically designed towards the requirements imposed by real time transmission of live-reconstructed mesh geometry. Tests on a large dataset show an encoding speed-up up to ten times at comparable compression ratio and quality, when compared with the high-end MPEG-4 SC3DMC mesh encoders. The implemented rateless code ensures complete packet loss protection of the triangle mesh object and a delivery delay within interactive bounds. Contrary to most linear fountain codes, the designed codec enables real-time progressive decoding allowing partial decoding each time a packet is received. This approach is compared with transmission over TCP in packet loss rates and latencies, typical in managed WAN and MAN networks, and heavily outperforms it in terms of end-to-end delay. The streaming component has been integrated into a larger 3DTI environment that includes state of the art 3-D reconstruction and rendering modules. This resulted in a prototype that can capture, compress transmit, and render triangle mesh geometry in real-time in realistic internet conditions as shown in experiments. Compared with alternative methods, lower interactive end-to-end delay and frame rates over three times higher are achieved.

3D Tele-immersion enables participants in remote locations to share, in real-time, an activity. It offers users natural interactivity and immersive experiences, but it challenges current networking solutions. Work in the past has mainly focused on the efficient delivery of image-based 3D videos and on the realistic rendering and reconstruction of geometry-based 3D objects. The contribution of this paper is a complete media pipeline that allows for geometry-based 3D tele-immersion. Unlike previous approaches, that stream videos or video plus depth estimate, our streaming module can transmit the live-reconstructed 3D representations (triangle meshes). Based on a set of comparative experiments, this paper details the architecture and describes a novel component that can efficiently stream geometry in real-time. This component includes both a novel fast local compression algorithm and a rateless packet protection scheme geared towards the requirements imposed by real-time transmission of live-capture mesh geometry. Tests on a large dataset show an encoding and decoding speed-up of over 10 times at similar compression and quality rates, when compared to the high-end MPEG-4 SC3DMC mesh encoder. The implemented rateless code ensures complete packet loss protection of the triangle mesh object and avoids delay introduced by retransmissions. This approach is compared to a streaming mechanism over TCP and outperforms it at packet loss rates over 2% and/or latencies over 9 ms in terms of end-to-end transmission delay. As reported in this paper, the component has been successfully integrated into a larger tele-immersive environment that includes beyond state of the art 3D reconstruction and rendering modules. This resulted in a prototype that can capture, compress transmit and render triangle mesh geometry in real-time over the internet.

Synchronization between locations is an important factor for enabling remote shared experiences. Still, experimental data on what is the acceptable synchronization level is scarce. This paper discusses the synchronization requirements for watching online videos together - a popular set of services that recreate the shared experience of watching TV together by offering tools to communicate while watching. It studies the noticeability and annoyance of synchronization differences of the video being watched, as well as the impact on users' feelings of togetherness, both for voice chat and text chat. Results of an experiment with 36 participants show that when using voice chat, users notice synchronization differences sooner, are more annoyed and feel more together than when using text chat. However, users with high text chat activity notice synchronization differences similar to participants using voice chat.