Versatile Video Coding (VVC) is the most recent international video coding standard jointly developed by ITU-T and ISO/IEC, which has been finalized in July 2020. VVC allows for significant bit-rate reductions around 50% for the same subjective video quality compared to its predecessor, High Efficiency Video Coding (HEVC). One year after finalization, VVC support in devices and chipsets is still under development, which is aligned with the typical development cycles of new video coding standards. This paper presents open-source software packages that allow building a complete VVC end-to-end toolchain already one year after its finalization. This includes the Fraunhofer HHI VVenC library for fast and efficient VVC encoding as well as HHI's VVdeC library for live decoding. An experimental integration of VVC in the GPAC software tools and FFmpeg media framework allows packaging VVC bitstreams, e.g. encoded with VVenC, in MP4 file format and using DASH for content creation and streaming. The integration of VVdeC allows playback on the receiver. Given these packages, step-by-step tutorials are provided for two possible application scenarios: VVC file encoding plus playback and adaptive streaming with DASH.

Modern video compression solutions rely on intra-prediction to exploit spatial redundancy within a frame. In this paper, a new intra-prediction scheme is presented whereby one or more prediction directions are inferred at the decoder side by means of a texture analysis process. The inferred directions are then combined by means of prediction fusion, to compute the final predicted samples. No further information is required in the bitstream, considerably limiting the amount of bits necessary to be transmitted in order to derive the prediction block at the decoder. Experiments in the context of the state-of-the-art Versatile Video Coding standard show the proposed method can improve coding efficiency by -0.63% BD-rates on average, and up to -1.04% on larger resolution sequences, with limited impact on the decoder complexity.

Following the finalization of the state-of-the-art video compression standard High Efficiency Video Coding (HEVC) in 2013, there is a growing need for more compression efficiency to handle various new video services and formats. In order to address this need, the ITU-T and MPEG standardization bodies are working together in a Joint Video Expert Team (JVET) group on the standardization of a new video codec called Versatile Video Coding (VVC), aiming to achieve 50% bitrate reduction over HEVC by 2020. One of the solutions that can be proposed in the intra mode is increasing the number of prediction modes, thus allowing more efficient predictions and less energetic residues. However, this solution causes an increase in the signaling cost of the chosen mode, which counterbalances some of the gains obtained by the newly added prediction modes. To address this problem, we propose to derive prediction modes at the decoder, to avoid sending them in the bitstream. This is done by deriving the best prediction mode at the encoder through a gradient calculation performed on each pixel of a well-determined causal region of the image. This method is identically applied at the decoder, thus avoiding the intra mode signaling in the bitstream. This process defines a new coding mode named Decoder side Intra Mode Derivation (DIMD), which competes with other coding modes normally tested at the encoder, including the classical intra coding mode. The proposed method reduces, on average 0.3%, the bitrate used to encode a sequence configured in All Intra (AI), with limited additional decoding complexity (2%).

The standardization of the next generation video codec, called Versatile Video Coding (VVC), has been progressing fast since 2018. At the same video quality, the VVC Test Model (VTM) currently provides about 27% bitrate saving compared to the preceding High Efficiency Video Coding (HEVC) standard. The ultimate objective is to reach about 50% improvement by the end of 2020. This paper presents an intra coding algorithm for VTM that skips the mode signaling at the encoder side and leaves it to be derived at the decoder side. To this end, a Histogram of Gradient (HoG) is computed from the texture of the previously reconstructed pixels and processed to derive the intra mode. Moreover, the proposed method is tuned to minimize its side-effects on the existing intra coding tools. Experimental results, on top of VTM-3.0, show that the proposed algorithm brings -0.22% BD-rate gain, with a negligible complexity.

Two High Efficiency Video Coding (HEVC) extensions, namely, the scalable HEVC (SHVC) extension and multiview HEVC (MV-HEVC) extension, have been finalized in July 2014 by the Moving Picture Experts Group and Video Coding Experts Group. These two extensions enable additional features not covered in the first version of the HEVC standard such as spatial, fidelity, bitdepth, and color gamut scalability, as well as stereoscopic and multiview representations. In this paper, we propose a software parallel decoder architecture for the HEVC standard and its multilayer extensions, including SHVC and MV-HEVC extensions. The decoder consists of multiple instances of the OpenHEVC decoder, one instance to decode each layer with a communication between dependent layers to perform inter-layer predictions. The proposed multilayer HEVC decoder is parallel friendly and supports both wavefront parallelism to simultaneously process adjacent rows of the frame and frame-based parallelism to decode a set of temporal and spatial frames in parallel. Moreover, the most time-consuming operation introduced in the SHVC extension, namely, the resampling of the inter-layer reference picture in spatial scalability, is optimized in single instruction multiple data for x86 platform. We assess the complexity of the multilayer HEVC decoder with respect to the simulcast configuration. The multilayer decoder decoding two SHVC layers introduces in average 40%-71% additional complexity compared with the single layer HEVC decoder. Moreover, the low level optimizations with a hybrid parallel processing solution enable a real-time decoding of 4Kp60 enhancement layer on a 6-core Intel i7 processor running at 3.4 GHz.

In this paper we investigate a selective video encryption in the scalable HEVC extension (SHVC). The SHVC extension encodes the video in several layers corresponding to different spatial and quality representations of the video. We propose a selective encryption solution using a chaotic-based encryption system. The proposed solution encrypts a set of sensitive parameters with a minimum complexity overhead, at constant bitrate and SHVC format compliant. Experimental results compare the performance of three encryption schemes: encrypt only the lowest layer, all layers, and only the highest layer. The first two schemes achieve a high security level with a drastic degradation in the decoded video, while the last scheme enables a perceptual video encryption by decreasing the quality of the highest layer below the quality of the clear layers.

The scalable high efficiency video coding (SHVC) standard aims to provide features of temporal, spatial and quality scalability. In this paper we investigate a pipeline and parallel software architecture for the SHVC decoder. The proposed architecture is based on the OpenHEVC software which implements the high efficiency video coding (HEVC) decoder. The architecture of the SHVC decoder enables two levels of parallelism. The first level decodes the base layer and the enhancement layers in parallel. The second level of parallelism performs the decoding of both the base layer and enhancement layers in parallel through the HEVC high level parallel processing solutions, including tile and wavefront. Up to the best of our knowledge, it is the first real time and parallel software implementation of the SHVC decoder. On an Intel Xeon processor running at 3.2 GHz, the SHVC decoder reaches the decoding of 1600p enhancement layer at 40 fps for x1.5 spatial scalability with using six concurent threads.

This paper introduces advanced software synthesis techniques that enhance the implementation of dynamic dataflow programs. These techniques have been implemented into open-source tools and demonstrated on well-known video decoders including one based on the new High Efficiency Video Coding (HEVC) standard. The results show an improvement of more than 100% of the frame-rate over previously proposed implementations, and achieve real-time decoding of high definition video sequences.

The Scalable High efficiency Video Coding (SHVC) standard is developed to offer spatial and quality scalability with high coding efficiency. In this paper we investigate a complexity analysis of a real time and parallel SHVC decoder. We first provide details on the implementation of the SHVC decoder including its low level optimizations. Furthermore, we introduce parallelism tools integrated in the SHVC software for parallel decoding. These tools include frame-based parallelism to decode a set of temporal and spatial frames in parallel as well as wavefront parallelism to simultaneously process separated regions of a picture. We assessed through experimental results the complexity of the real time SHVC decoder in different coding configurations. The SHVC decoder with two layers introduces in average an additional complexity of 43 to 80% in respect to a simulcast configuration. The low level optimizations together with a hybrid parallelism solution enables a real time decoding of 1600p40 enhancement layer on an Intel i7 processor.

There are several High Efficiency Video Coding (HEVC) extensions providing new tools on the top of a common HEVC base layer. These HEVC extensions enable higher temporal, spatial or quality of the video, 3-D rendering and range extension. In addition to the conforming HEVC decoder, the end-user need to support the decoding of the HEVC extension to benefits from its tools. In this paper we propose an unified software decoder enabling to decode all HEVC extensions. This solution is based on the open source project OpenHEVC which implements a conforming HEVC decoder. The new tools defined in HEVC extension are implemented and integrated into the OpenHEVC decoder. We show the first end-to-end video streaming demonstration of the HEVC extensions with the OpenHEVC decoder and the GPAC player. The GPAC server streams one HEVC base layer and two enhancement layers. At the client side, GPAC player uses the OpenHEVC to decode the base layer for HD resolution and can also decode whether the first EL for 4K resolution or the second EL for 3-D rendering.

The new Scalable High efficiency Video Coding (SHVC) standard is based on a multi-loop coding structure which requires the total decoding of all intermediate layers. The decoding complexity becomes then a real issue, especially for a real time decoding of ultra high video resolutions. A parallel processing architecture is proposed to reduce both the decoding time and the latency of the SHVC decoder. The proposed solution combines the high level parallel processing solutions defined in the HEVC standard with an extension of the frame-based parallelism. The latter solution enables the decoding of several spatial and temporal SHVC frames in parallel to enhance both decoding frame rate and latency. The wavefront parallel processing solution is used for more coarse level of granularity. The proposed hybrid parallel processing approach achieves a near optimal speedup and provides a good trade-off between decoding time, latency and memory usage. On a 6 cores Xeon processor, the parallel SHVC decoder performs a real time decoding of 1600p60 video resolution.

Ultra High Definition video is entering a phase of deployment tests, with first UHD services being launched in late 2013. Most on-air broadcast UHD services will not start until 4K display market penetration is high enough, which will not happen until at least 2018. Therefore, most initial services will be offered via IPTV services or Over-The-Top delivery technologies. On the other hand, the new HEVC video compression standard, with a 50% gain over its predecessor AVC|H.264, is the natural industry choice for compressing UHD content; at the transport level, MPEG-DASH offers a complete set of tools for broadcast and broadband convergence, and is one of the first candidates for HEVC preferred transport layer. In this context, deployment tests of MPEG-DASH and UHD HEVC content are expected to be a very important topic in the next two years. In this paper, we present a Ultra High Definition HEVC DASH dataset ranging from HD to UHD in different bit rates. This data set may be used to simulate UHD DASH services, whether on-demand or live, using real-life professional quality content.

Summary form only given. In this paper we present efficient parallelization implementations for different stages of the HEVC decoder, which are LCU decoding, deblocking filtering and SAO filtering. Each of the stages are parallelized in separate passes. The LCU decoding is parallelized using Wave front Parallel Processing (WPP). Deblocking and SAO filtering are parallelized by segmenting each picture into separate regions of consecutive LCU rows and processing each of the regions in a concurrent fashion. On a 6 core machine with 6 threads running concurrently, experimental results showed an average accelerating factor of 4.6, 5, 5.35 for the LCU decoding stage and 4.5, 4.9, 5 for deblocking filtering stage and 4, 4.5 and 5 for SAO filtering stages on HD, 1600p and 2160p sequences respectively.

Unlike deblocking filter of H.264/AVC, deblocking filter of HEVC is computationally less complex and offers more parallelization possibilities. In this paper we present comparison of three different parallelization implementations of deblocking filter which operate on picture by picture basis. The first method filters the vertical edges and horizontal edges in separate passes. The other two methods combine the vertical and horizontal edge filtering in a single pass. On a 6 core machine with 6 threads running concurrently, experimental results showed an average accelerating factor of 4.5, 5 and 5 for each of the implementation methods on 1080p, 1600p and 2160p sequences respectively.

In this paper, we present Orcc, an open-source development environment that aims at enhancing multimedia development by offering all the advantages of dataflow programming: flexibility, portability and scalability. To do so, Orcc embeds two rich eclipse-based editors that provide an easy writing of dataflow applications, a simulator that allows quick validation of the written code, and a multi-target compiler that is able to translate any dataflow program, written in the RVC-CAL language, into an equivalent description in both hardware and software languages. Orcc has already been used to successfully write tens of multimedia applications, such as a video decoder supporting the new High Efficiency Video Coding standard, that clearly demonstrates the ability of the environment to develop complex applications. Moreover, results show scalable performances on multi-core platforms and achieve real-time decoding frame-rate on HD sequences.

Data process algorithms are increasing in complexity especially for image and video coding. Therefore, hardware development using directly hardware description languages (HDL) such as VHDL or Verilog is a difficult task. Current research axes in this context are introducing new methodologies to automate the generation of such descriptions. In our work we adopted a high level and target-independent language called CAL (Caltrop Actor Language). This language is associated with a set of tools to easily design dataflow applications and also a hardware compiler to automatically generate the implementation. Before the modifications presented in this paper, the existing CAL hardware back-end did not support some high-level features of the CAL language. Consequently, high-level designed actors had to be manually transformed to be synthesizable. In this paper, we introduce a general automatic transformation of CAL descriptions to make these structures compliant and synthesizable. This transformation analyses the CAL code, detects the target features and makes the required changes to obtain synthesizable code while keeping the same application behavior. This work resolves the main bottleneck of the hardware generation flow from CAL designs.

CAL is a dataflow oriented language for writing high-level specifications of signal processing applications. The language has recently been standardized and selected for the new MPEG Reconfigurable Video Coding standard. Application specifications written in CAL can be transformed into executable implementations through development tools. Unfortunately, the present tools provide no way to schedule the CAL entities efficiently at run-time. This paper proposes an automated approach to analyze specifications written in CAL, and produce run-time schedules that perform on average 1.45× faster than implementations relying on default scheduling. The approach is based on quasi-static scheduling, which reduces conditional execution in the run-time system.

In this paper, we introduce the Just-In-Time Adaptive Decoder Engine (Jade) project, which is shipped as part of the Open RVC-CAL Compiler (Orcc) project. Orcc provides a set of open-source software tools for managing decoders standardized within MPEG by the Reconfigurable Video Coding (RVC) experts. In this framework, Jade acts as a Virtual Machine for any decoder description that uses the MPEG RVC paradigm. Jade dynamically generates a native decoder representation suitable for X86, ARM and CELL platforms with a possibility of exploiting multi-core CPUs. Thus, according to the MPEG RVC decoder description coupled with a video coded stream, Jade can create, configure and re-configure video decompression algorithms adapting to the video content.

Data-flow has proven to be an attractive computation model for programming digital signal processing (DSP) applications. A restricted version of data-flow, termed synchronous data-flow (SDF), offers strong compile-time predictability properties, but has limited expressive power. A new type of hierarchy (Interface-based SDF) has been proposed allowing more expressivity while maintaining its predictability. One of the main problems with this hierarchical SDF model is the lack of trade-off between parallelism and network clustering. This paper presents a systematic method for applying an important class of loop transformation techniques in the context of interface-based SDF semantics. The resulting approach provides novel capabilities for integrating parallelism extraction properties of the targeted loop transformations with the useful modeling, analysis, and code reuse properties provided by SDF.

This paper describes an OMAP based real-time test bench to find the Pareto frontier of an H.264/SVC decoder within a distortion-energy optimization space. A metric to estimate video distortion is introduced. In addition, energy consumption estimates are obtained from real-time measurements of the computational load. Finally, test bench operation is successfully demonstrated with different H.264/SVC-compliant sets of sequences.

The elaboration of new and innovative systems such as MPSoC (Multiprocessor System on Chip) which are made up of multiple processors, memories and IPs lies on the designers to achieve a complex codesign work. Specific tools and methods are needed to cope with the increasing complexity of both algorithms and platforms. Our approach to design such systems is based on the usage of a high level of abstraction language called RVC CAL. This language is dataflow oriented and thus points out the concurrency and parallelism of algorithms. Moreover CAL is supported by the OpenDF simulator and by two code generators called CAL2C (software generator) and CAL2HDL (hardware generator). The MPEG expert group has recently elaborated the Reconfigurable Video Coding (RVC) standard which defines the RVC CAL language as reference for MPEG video decoder descriptions. This paper introduces the opportunities to design an innovative system involving hardware and software IPs, embedded processors and memories from a CAL model. Practical results on a FPGA are provided with a codesign solution of an MPEG4 Simple Profile (SP).

This paper describes the Open SVC Decoder project, an open source library which implements the Scalable Video Coding (SVC) standard, the latest standardized by the Joint Video Team (JVT). This library has been integrated into open source players The Core Pocket Media Player (TCPMP) and mplayer, in order to be deployed over different platforms with different operating systems.

Current video coding standards, and their reference implementations, are architected as large monolithic and sequential algorithms, in spite of the considerable overlap of functionality between standards, and the fact that they are frequently implemented on highly parallel computing platforms. The former leads to unnecessary complexity in the standardization process, while the latter implies that implementations have to be rebuilt from the ground up to reflect the parallel nature of the target.

Dataflow descriptions have been used in a wide range of Digital Signal Processing (DSP) applications, such as multi-media processing, and wireless communications. Among various forms of dataflow modeling, Synchronous Dataflow (SDF) is geared towards static scheduling of computational modules, which improves system performance and predictability. However, many DSP applications do not fully conform to the restrictions of SDF modeling. More general dataflow models, such as CAL, have been developed to describe dynamically-structured DSP applications. Such generalized models can express dynamically changing functionality, but lose the powerful static scheduling capabilities provided by SDF. This paper focuses on detection of SDF-like regions in dynamic dataflow descriptions - in particular, in the generalized specification framework of CAL. This is an important step for applying static scheduling techniques within a dynamic dataflow framework. Our techniques combine the advantages of different dataflow languages and tools, including CAL, DIF and CAL2C. The techniques are demonstrated on the IDCT module of MPEG Reconfigurable Video Coding (RVC).

Video codec applications become more and more complex to design. To ease the description of such applications, MPEG creates a Framework called Reconfigurable Video Coding (RVC). All existing codecs in MPEG have a similar structure, they are based on a hybrid decoding structure and some of their part can be reused on other design. In RVC, the dataflow is expressed using a network of components also called Functional Units (FUs) interconnected by FIFOs. An FU, programmed in CAL Language, includes the processing and the internal states. This paper puts the focus on a parallel dataflow description of the most complex MPEG RVC decoder available called MPEG4-AVC Constrained Baseline Profile (CBP) decoder.

This demonstration presents an integrated environment that translates a CAL-based dataflow specification [1] into a heterogeneous implementation, composed by HDL and C codes. The demonstration focuses on the capability of the co-design environment to automatically build an executable heterogeneous system implementation running on a platform composed of a processor and a FPGA from the annotation of the CAL specification. The possibility of direct synthesis from a high level specification is a crucial issue for enabling efficient re-design cycles that include rapid prototyping and validation of performances of the final implementation. The design approach enabled by such integrated environment is particularly suited for development of complex processing systems such as video codecs. As a case study, the demonstration provides the analysis and validation of different software and hardware partitioning of a MPEG-4 simple profile decoder.

This paper presents an in-depth case study on dataflow-based analysis and exploitation of parallelism in the design and implementation of a MPEG reconfigurable video coding decoder. Dataflow descriptions have been used in a wide range of digital signal processing (DSP) applications, such as applications for multimedia processing and wireless communications. Because dataflow models are effective in exposing concurrency and other important forms of high level application structure, dataflow techniques are promising for implementing complex DSP applications on multicore systems, and other kinds of parallel processing platforms. In this paper, we use the client access license (CAL) language as a concrete framework for representing and demonstrating dataflow design techniques. Furthermore, we also describe our application of the differential item functioning dataflow interchange format package (TDP), a software tool for analyzing dataflow networks, to the systematic exploitation of concurrency in CAL networks that are targeted to multicore platforms. Using TDP, one is able to automatically process regions that are extracted from the original network, and exhibit properties similar to synchronous dataflow (SDF) models. This is important in our context because powerful techniques, based on static scheduling, are available for exploiting concurrency in SDF descriptions. Detection of SDF-like regions is an important step for applying static scheduling techniques within a dynamic dataflow framework. Furthermore, segmenting a system into SDF-like regions also allows us to explore cross-actor concurrency that results from dynamic dependences among different regions. Using SDF-like region detection as a preprocessing step to software synthesis generally provides an efficient way for mapping tasks to multicore systems, and improves the system performance of video processing applications on multicore platforms.

The MPEG reconfigurable video coding (RVC) framework aims to provide a unified specification of all video technology. In this framework, a decoder is modularly built as a configuration of video coding tools taken from the MPEG toolbox library. The elements of the library are specified using the CAL actor language. CAL is a dataflow based language providing computation models that are concurrent and modular. This paper presents a synthesis tool that from a CAL specification generates C code. Indeed, code generators are fundamental supports for the deployment and success of the MPEG RVC framework. This paper focuses on the automatic translation of a CAL actor. This approach has been used to obtain a C implementation of the inverse DCT module which is part of the MPEG-4 Simple Profile decoder, chosen by MPEG experts to validate the RVC approach. The generated code is validated against the original CAL description and simulated using the Open Dataflow environment.

The MPEG reconfigurable video coding (RVC) framework is a new standard under development by MPEG that aims at providing a unified specification of current MPEG video coding technologies. In this framework, a decoder is built as a configuration of video coding modules taken from the standard ldquoMPEG toolbox libraryrdquo. The elements of the library are specified using the CAL actor language (CAL). CAL is a dataflow based language providing computation models that are concurrent and modular. This paper describes a synthesis tool that from a CAL specification automatically generates compilable C-code. Code generators are fundamental supports for the deployment and success of the MPEG RVC framework. This paper focuses on the automatic translation of CAL actions, which is the first step to a complete actor translation. The techniques described here enable to automatically generate C-code according to a finite set of rules. This approach has been used to obtain a C implementation of the IDCT module which is one element of the RVC library. The generated code is validated against the original CAL dataflow program simulated using the open dataflow environment.

Sequential MPEG-4 solutions actually developed for single processors try to integrate the most functionalities as possible in an unique software, and are generally oversized compared with the actual service requirement. Moreover, they can hardly be projected onto multiprocessors targets, leading to an extra load of source code and calculations, but also to a sub-optimal use of the architecture parallelism. This paper introduces a distributed MPEG-4 application, where the system part is hosted by a standard PC, and the video decoder is supported by a multi-DSPs board. In particular, we present our AVSynDEx methodology allowing both an incremental building, an easy update on the video decoder description, and a quasi-automatic implementation onto a multi-C6x platform. We also define a global scheduler managing the parallel execution of the video and system applications.

Sequential Mpeg-4 solutions actually developed for single processors try to integrate the most functionalities as possible in an unique software, and are generally oversized compared with the actual service requirement. Moreover, they can hardly be projected onto multiprocessors targets, leading to an extra load of source code and calculations, but also to a sub-optimal use of the architecture parallelism. This paper introduces a distributed Mpeg-4 application, where the system part is hosted by a standard PC, and the video decoder is supported by a multi-DSPs board. In particular, we present our AVSynDEx methodology allowing both an incremental building, an easy update on the video decoder description, and a quasi-automatic implementation onto a multi-C6x platform. We also define a global scheduler managing the parallel execution of the video and system applications.

Real time signal and image applications have very important time constraints, involving the use of several powerful numerical calculation units. Our aim is to develop a fast prototyping process dedicated to parallel architectures made of several last generation Texas Instruments TMS320C6X DSP. The methodology is based on the use of SynDEx, a CAD software improving the algorithm implementation onto multiprocessor architectures, finding the best matching between an algorithm and an architecture. A SynDEx executive kernel has been developed for the C6X DSP family in order to automatically generate a distributed and optimized static executive of the specified algorithm onto those processors. We have tested the efficiency of our methodology with a complete Mpeg-2 coding application.