Versatile Video Coding (VVC) is the most recent international video coding standard jointly developed by ITU-T and ISO/IEC, which has been finalized in July 2020. VVC allows for significant bit-rate reductions around 50% for the same subjective video quality compared to its predecessor, High Efficiency Video Coding (HEVC). One year after finalization, VVC support in devices and chipsets is still under development, which is aligned with the typical development cycles of new video coding standards. This paper presents open-source software packages that allow building a complete VVC end-to-end toolchain already one year after its finalization. This includes the Fraunhofer HHI VVenC library for fast and efficient VVC encoding as well as HHI's VVdeC library for live decoding. An experimental integration of VVC in the GPAC software tools and FFmpeg media framework allows packaging VVC bitstreams, e.g. encoded with VVenC, in MP4 file format and using DASH for content creation and streaming. The integration of VVdeC allows playback on the receiver. Given these packages, step-by-step tutorials are provided for two possible application scenarios: VVC file encoding plus playback and adaptive streaming with DASH.

With the massive increase of video traffic over the internet, HTTP adaptive streaming has now become the main technique for infotainment content delivery. In this context, many bandwidth adaptation algorithms have emerged, each aiming to improve the user QoE using different session information e.g. TCP throughput, buffer occupancy, download time... Notwithstanding the difference in their implementation, they mostly use the same inputs to adapt to the varying conditions of the media session. In this paper, we show that it is possible to predict the bitrate decision of any ABR algorithm, thanks to machine learning techniques, and supervised classification in particular. This approach has the benefit of being generic, hence it does not require any knowledge about the player ABR algorithm itself, but assumes that whatever the logic behind, it will use a common set of input features. Then, using machine learning feature selection, it is possible to predict the relevant features and then train the model over real observation. We test our approach using simulations on well-known ABR algorithms, then we verify the results on commercial closed-source players, using different VoD and Live realistic data sets. The results show that both Random Forest and Gradient Boosting achieve a very high prediction accuracy among other ML-classifier.

Modern multimedia frameworks mix a variety of functionalities, such as network inputs and outputs, multiplexing stacks, compression, uncompressed domain effects and scripting, and require realtime processing for live services. They usually end up becoming very difficult to apprehend for end users and/or third-party developers, with complex testing and maintenance. The GPAC open-source media framework is no exception here. After 15 years of development and experiences in interactive media content, the possibilities offered by the framework were heavily restrained by a fixed media pipeline approach, despite the large number of tools available in its code base. In this paper, we discuss the major re-architecture undergone by GPAC to offer developers and end users a completely configurable media pipeline in a simple way, review the core concepts of this new design, their reasoning and the new features they unlock. We show how various complex use cases can now simply be achieved and how the re-architecture improved GPAC stability, making it a first-class candidate for research, commercial and educational projects involving multimedia processing.

As video traffic becomes the dominant part of the global Internet traffic, keeping a good quality of experience (QoE) becomes more challenging. To improve QoE, HTTP adaptive streaming with various adaptive bitrate (ABR) algorithms has been massively deployed for video delivery. Based on their required input information, these algorithms can be classified, into buffer-based, throughput-based or hybrid buffer-throughput algorithms. Nowadays, due to their low cost and high scalability, peer-to-peer (P2P) networks have become an efficient alternative for video delivery over the Internet, and many attempts at merging HTTP adaptive streaming and P2P networks have surfaced. However, the impact of merging these two approaches is still not clear enough, and interestingly, the existing HTTP adaptive streaming algorithms lack testing in a P2P environment. In this paper, we address and analyze the main problems raised by the use of the existing HTTP adaptive streaming algorithms in the context of P2P networks. We propose two methodologies to make these algorithms more efficient in P2P networks regardless of the ABR algorithm used, one favoring overall QoE and one favoring P2P efficiency. Additionally, we propose two new metrics to quantify the P2P efficiency for ABR delivery over P2P.

The following topics are dealt with: learning (artificial intelligence); feature extraction; video coding; object detection; data compression; image classification; image coding; image representation; image reconstruction; optimisation.

Currently on the market there is a plethora of affordable dedicated cameras or smartphones, able to record video and timed geospa-tial data (device location and orientation). This timed metadata can be used to identify relevant (in time and space) recordings. However, there has not been a platform that allows to exploit this information in order to utilize the relevant recordings in an interactive consumption scenario. In this paper we present SWAPUGC, a browser-based platform for building applications that use the accompanying geospatial data to dynamically select the streams for watching an event (or any spatiotemporal reference point). The view selection can be performed either manually, or automatically by a predefined algorithm that switches to the most suitable stream according to the recording characteristics. SWAPUGC is a research tool to test such adaptation algorithms and it is provided as an open-source project, accompanied by an example demo application and references to a compatible dataset and recorder. In this paper, we explain and then demonstrate the capabilities of the platform by an example implementation and examine future prospects and extensions.

Adaptive streaming technologies are used heavily to deliver video content over the Internet, but still face problems in networks with limited bandwidth, for videos with ultra-high definition (UHD) or greater, even when using the latest High-Efficiency Video Coding (HEVC) standard. In this paper, we propose a solution to these problems based on three things: a specific HEVC-compliant motion-constrained tiled encoding allowing tile-based decoding and rendering; a specific derivation of the ISO base media file format enabling a tile-based encapsulation and delivery; and their use in adaptive streaming of tiled video content using MPEG dynamic adaptive streaming over HTTP requiring only a single standard HEVC decoder on the client side. We present an evaluation of constrained and unconstrained HEVC tiling compression, and also of the associated packaging and streaming overheads. The results of this evaluation indicate that our approach incurs a complete overhead of at most 7.3% for UHD content, at 10 Mbps or above, when split in 25 tiles, and only 3.5% for nine tiles, when compared with the streaming of the non-tiled video. Our approach also comes with a small impact on client-side players but provides the interesting benefit of being able to deliver UHD videos over bandwidth-constrained networks adaptively, based on its tiled content.

Interactive spatial navigation within a video is getting attention in the research community, especially in scenarios such as 360° video, video surveillance or region-of-interest viewing. With more content being delivered over the top, the complexity of spatial access to the video is increased by the need to perform bitrate adaptation to react to network variations. In this paper, we describe how spatial access can be performed in an adaptive HTTP streaming context, using tiling of the source content, MPEG-DASH and its SRD extensions. We describe a configurable implementation of these technologies, within the GPAC open-source player, allowing experimentations of different adaptation policies for tiled video content.

While over-the-top video distribution is now widely deployed, it still suffers from much higher latencies than traditional broadcast, typically from a few seconds up to half a minute. In this paper, we demonstrate a novel DASH system with latency close to broadcast channels, and show how such a system can be used to enable combined broadcast and broadband services while keeping the client buffering requirements on the broadcast link low.

Adaptive HTTP Streaming has become a widespread technology for delivering video to the end users. However, this technology usually suffers from various latencies, requiring an initial delay before playback. This is problematic for low delay or interactive applications, such as gaming or live event webcasting. A long initial waiting time is perceived by most users as bad quality, or even non-working content. In this paper, we review the causes of initial delays in MPEG-DASH and common strategies used to reduce this latency. We propose a new method based on HTTP/1.1 and compatible with existing infrastructures for the initial setup of an MPEG-DASH session. We compare our proposal to several existing approaches based on HTTP/1.x and on the HTTP/2 server push technique. Our experimental results show that 2 RTTs can be avoided and that the total downloaded size is reduced by 25% when using HTTP/1.1 and by 20% in HTTP/2 with similar latency.

There are several High Efficiency Video Coding (HEVC) extensions providing new tools on the top of a common HEVC base layer. These HEVC extensions enable higher temporal, spatial or quality of the video, 3-D rendering and range extension. In addition to the conforming HEVC decoder, the end-user need to support the decoding of the HEVC extension to benefits from its tools. In this paper we propose an unified software decoder enabling to decode all HEVC extensions. This solution is based on the open source project OpenHEVC which implements a conforming HEVC decoder. The new tools defined in HEVC extension are implemented and integrated into the OpenHEVC decoder. We show the first end-to-end video streaming demonstration of the HEVC extensions with the OpenHEVC decoder and the GPAC player. The GPAC server streams one HEVC base layer and two enhancement layers. At the client side, GPAC player uses the OpenHEVC to decode the base layer for HD resolution and can also decode whether the first EL for 4K resolution or the second EL for 3-D rendering.

Ultra High Definition video is entering a phase of deployment tests, with first UHD services being launched in late 2013. Most on-air broadcast UHD services will not start until 4K display market penetration is high enough, which will not happen until at least 2018. Therefore, most initial services will be offered via IPTV services or Over-The-Top delivery technologies. On the other hand, the new HEVC video compression standard, with a 50% gain over its predecessor AVC|H.264, is the natural industry choice for compressing UHD content; at the transport level, MPEG-DASH offers a complete set of tools for broadcast and broadband convergence, and is one of the first candidates for HEVC preferred transport layer. In this context, deployment tests of MPEG-DASH and UHD HEVC content are expected to be a very important topic in the next two years. In this paper, we present a Ultra High Definition HEVC DASH dataset ranging from HD to UHD in different bit rates. This data set may be used to simulate UHD DASH services, whether on-demand or live, using real-life professional quality content.

Video streaming has become a very popular application on the web, with rich player interfaces and subtitle integration. Additionally, live streaming solutions are deployed based on HTTP Streaming technologies. However, the integration of video and subtitles in live streaming solutions still poses some problems. This paper describes a demonstration of live streaming of video and subtitle data using the MPEG-DASH technology and its synchronous playback in a web browser. It presents the formats, architecture and technical choices made for this demonstration and shows that it is feasible with upcoming browsers, paving the way for richer web video applications.

The delivery of multimedia content over HTTP and on top of existing Internet infrastructures is becoming the preferred method within heterogeneous environment. The basic design principle is having an intelligent client which selects given and applicable media representations by issuing HTTP requests for individual segments based on the users' context and current conditions. Typically, this client behavior differs between implementations of the same kind and for the objective evaluations thereof appropriate datasets are needed. This paper presents a distributed dataset for the recently published MPEG-DASH standard which is mirrored at different sites across Europe, namely Klagenfurt, Paris, and Prague. A client implementation may choose to request segments from these sites and dynamically switch to a different location, e.g., in case the one currently used causes any issues. Thus, this distributed DASH dataset can be used for real-world evaluations enabling the simulation of switching between different content delivery networks.

Existing broadcast networks can deliver identical content to a large number of users. Broadband networks can deliver personalized content to specific users. Hybrid delivery tries to use the best of both types of networks to provide customized services to many users. This paper studies the theoretical aspects and challenges behind the delivery of multimedia content over hybrid broadcast broadband networks, including bootstrapping, synchronization and resynchronization. It presents a solution based on the use of a global clock which does not require communication across networks and is compatible with existing technologies. This solution is implemented in a multimedia player and evaluated against two real-world scenarios, mixing DVB or FM broadcast networks with a broadband IP network. The results confirm the theoretical approach and show that some fine tuning in the networks is needed for a tighter synchronization.

Multimedia applications are challenging software and require collaboration of very different components such as networking, rendering, or scripting to provide a nice user experience. GPAC is an open source multimedia framework that implements a vast number of components and helps experimenting with different types of multimedia applications. It provides a multimedia packager, some servers and a player for dynamic and interactive multimedia content. In this paper, we present the recent additions to the project which allow experimenting with new applications such as dynamic home networking or rich broadcasting of interactive content on the latest devices, including with 3D displays. Such experiments can be useful in research work and, as illustrated, in academic environment for educational purposes.

The Huawei/3DLife Grand Challenge Dataset provides multimodal recordings of Salsa dancing, consisting of audiovisual streams along with depth maps and inertial measurements. In this paper, we propose a system for augmented reality-based evaluations of Salsa dancer performances. An essential step for such a system is the automatic temporal synchronisation of the multiple modalities captured from different sensors, for which we propose efficient solutions. Furthermore, we contribute modules for the automatic analysis of dance performances and present an original software application, specifically designed for the evaluation scenario considered, which enables an enhanced dance visualisation experience, through the augmentation of the original media with the results of our automatic analyses.

In recent years, audio-visual distribution over Internet has witnessed the growing usage of HTTP based delivery systems. While these systems have their drawbacks for some use-cases, they also have many advantages, the most important one being reusing the existing delivery infrastructure such as HTTP servers, proxies and caches. The MPEG group has started the standardization of the Dynamic Adaptive Streaming over HTTP (DASH) of major transport formats, MPEG-2 TS and ISO Base Media File, and mostly focuses on audio, video and subtitle formats. We believe Rich Media services have a role to play in this landscape, as a presentation layer for the audio-visual content first, but also as a dedicated media in the DASH content for real-time media-synchronized interactive services. In this paper, we present a study on usages of DASH for Rich Media Services.

Widgets are small and focused multimedia applications that can be found on desktop computers, mobile devices or even TV sets. Widgets rely on structured documents to describe their spatial, temporal and interactive behavior but also to communicate with remote data sources. However, these sources have to be known at authoring time and the communication process relies heavily on scripting. In this paper, we describe a mechanism enabling the communication between widgets and their dynamic environment (other widgets, remote data sources). The proposed declarative mechanism is compatible with existing Widgets technologies, usable with script-based widgets as well as with fully declarative widgets. A description of an implementation is also provided.

International audience

The Scalable Vector Graphics (SVG) language allows in its version 1.2 the description of multimedia scenes including audio, video, vector graphics, interactivity and animations. This standard has been selected by the mobile industry as the format for vector graphics and rich media content. For this purpose, additional tools were introduced in the language to solve the problem of the playback of long-running SVG sequences on memory-constrained devices like mobile phones. However, the proposed tools are not entirely sufficient and solutions outside the scope of SVG are needed.

GPAC is a multimedia framework for research and academic purposes in different aspects of multimedia, with a focus on presentation technologies (graphics, animation and interactivity). The project started in 2003 with the initial goal to develop from scratch, in ANSI C, clean software compliant to the MPEG-4 Systems standard, a small and flexible alternative to the MPEG-4 reference software. Since then, the project has evolved into an advanced multimedia player, a multimedia packager and several servers. The project is intended to a wide audience ranging from end-users or content creators with development skills who want to experiment the new standards for interactive technologies or want to convert files for mobile devices, to developers who need players and/or server for multimedia streaming applications.