We consider the problem of embedding a neural network directly into a video decoder. This requires a design with complexity suitable for implementation on mobile and power constrained devices. To achieve this goal, we explored Multi-scale CNN (MSCNN) design in [1]. In this paper, we improve the design to support super resolution spatial scale factors SF==(1.5×, 2×, 3×, 4×, 6×) by modifying the polyphase filter (Figure 1a) that generates an upsampled output using g(scale) phases and stride of Sscale. When SF= 1.5 ×, g(scale, Sscale) = (9,2); Otherwise it is (scale2,1). gsG, kK, and sS denote channel group size of G, kernel size of K×K, and stride of S. To reduce per-pixel Multiply-Accumulates (MACs), the 3×1 and 1×3 convolutional layers use Canonical Polyadic (CP) decomposition and reduced channel count. These changes reduce MACs/pixel from 1,924 in [1] to 1,192 to 584. Figure 1b, shows the placement of MSCNN in AVM [2]. We code 4K video, using AOMedia's Adaptive Streaming (AS) test conditions and compare MSCNN versus following resampler combinations: Downsampling - [L5: Lanczos(5), L6: Lanczos(6)]; Upsampling - [L5, L6, BL: Bilinear, BC: Bicubic]. We observe MSCNN provides on average 30.4% rate saving.

Incorporating neural networks into a video codec as an in-loop filter has been shown to provide significant improvements in coding efficiency. Unfortunately, the computational complexity associated with the neural network, specifically the number of multiply-accumulate (MAC) operations, makes these approaches intractable in practice. In this paper, we consider using a multiscale approach to reduce complexity while maintaining coding efficiency. Experimental results demonstrate a 5.4× reduction in MAC operations while achieving an average bit rate savings of 6.4% and 6.3% for all intra and random access coding, respectively, when compared to the evolving AV2 standard. Ablation studies are also provided and show that the approach achieves all but 0.2% of the coding efficiency of full resolution processing.

The Joint Video Experts Team recently evaluated technology in response to a Call for Proposals for Video Compression with Capability beyond HEVC. A number of proposed solutions were evaluated, with a sub-set demonstrating the potential to reduce bit-rates by over 40% compared to HEVC. This paper presents the author's contributions to one of these proposals. The proposal emphasized a flexible, rectangular partitioning structure that was combined with new coding tools, including improved motion vector coding and quantization signaling methods. Results show the efficacy of the approach. Using the evaluation procedure defined in the Call, the described approach provides coding gains relative to an HEVC anchor of 41.2% and 35.7% for 4K-SDR and HD-SDR sequences, respectively, using the random access configuration; 29.0% for HD-SDR sequences using a low delay configuration, and gains of 34.3% and 32.2% for PQ-HDR and HLG-HDR sequences, respectively, using a random access configuration.

Recent broadcast standards support Layered Division Multiplexing (LDM) to achieve graceful degradation as signal quality degrades at the receiver. LDM is accomplished by using different constellations within the same Radio Frequency (RF) spectrum. LDM thus enables delivering multiple service tiers in a single broadcast channel. LDM when used in conjunction with scalable source coding codecs such as the Scalable extension of High Efficiency Video Coding (SHVC), further helps improve overall spectrum utilization and efficiency. In this paper we investigate a 2-tier broadcast LDM based service with one service tier aimed at lower video resolution such as 540p, 720p, 1080p for a mobile receiver (smaller/indoor antenna) and the other service tier targeting twice the video resolution of the lower tier, for stationary receivers (larger/outdoor antenna). The primary contribution of this paper is to identify 2-tier transmission configurations of interest to broadcasters and compare the spectrum and bitrate coding efficiency gains of an SHVC-based multi-tier service versus a simulcast (single layer) based multi-tier service for an Advanced Television Systems Committee (ATSC) 3.0 transmission system. Bitrate savings ranging from 38% and 57% is observed for the SHVC based layered system. For large coverage and pedestrian with a receiver test scenarios, channel utilization savings ranging from 23% to 46% is observed. For mobile and tablet in bedroom scenarios a smaller broadcast bandwidth savings ranging from 6% to 9% is observed.