• Content-adaptive video encryption in the compressed domain • Dynamically selects syntax elements based on video content complexity • Maintains full format compliance using Adaptive Syntax Integrity (ASI) • Tunable parameters balance encryption strength and bitrate increase With the ever-increasing amount of digital video content, efficient encryption is crucial to protect visual content across diverse platforms. Existing methods often incur excessive bitrate overhead due to content variability. Furthermore, since most videos are already compressed, encryption in the compressed domain is essential to avoid processing overhead and re-compression quality loss. However, achieving both format compliance and compression efficiency while ensuring that the decoded content remains unrecognizable is challenging in the compressed domain, since only limited information is available without full decoding. This paper proposes an adaptive compressed domain video encryption (ACDC) method that dynamically adjusts the encryption strategy according to content characteristics. Two tunable parameters derived from the bitstream information enable adaptation to various application requirements. An adaptive syntax integrity method is employed to produce format-compliant bitstreams without full decoding. Experimental results show that ACDC reduces bitrate overhead by 48.2% and achieves a 31-fold speedup in encryption time compared to the latest state of the art, while producing visually unrecognizable outputs.

Leveraging the overfitting property of deep neural networks (DNNs) is trending in video delivery systems to enhance video quality within bandwidth limits. Existing approaches transmit overfitted super-resolution (SR) model streams for low-resolution (LR) bitstreams, which are used to reconstruct high-resolution (HR) videos at the decoder. Although these approaches show promising results, the huge computational costs of training a large number of video frames limit their practical applications. To overcome this challenge, we propose an efficient patch sampling method named EPS for video SR network overfitting, which identifies the most valuable training patches from video frames. To this end, we first present two low-complexity Discrete Cosine Transform (DCT)-based spatial-temporal features to measure the complexity score of each patch directly. By analyzing the histogram distribution of these features, we then categorize all possible patches into different clusters and select training patches from the cluster with the highest spatial-temporal information. The number of sampled patches is adaptive based on the video content, addressing the trade-off between training complexity and efficiency. Our method reduces the number of training patches by 75.00% to 91.69%, depending on the resolution and number of clusters, while preserving high video quality and greatly improving training efficiency. Our method speeds up patch sampling by up to 82.1× compared to the state-of-the-art patch sampling technique (EMT).

Existing stereo image quality assessment (SIQA) methods generally have limitations in binocular fusion and fine-grained perception modeling. To address these issues, we propose a Perception-Inspired Network for SIQA that simulates binocular difference-guided fusion, high-frequency sensitivity, and hierarchical perception mechanisms of the human visual system (HVS). First, a difference-guided binocular fusion (DGBF) module is designed to mimic the binocular difference sensitivity mechanism, which exploits difference information at both the feature-level and image-level to optimize binocular fusion. Furthermore, the image distortion primarily affects the high-frequency components, which are critical for perceptual quality. To reflect this, we propose a high-frequency enhancement module (HFEM) to simulate the human eye's sensitivity to edge and texture distortions. Finally, to better achieve fine-grained perception modeling, we propose a hierarchical quality regression strategy that simulates the human perceptual process, from perceiving local details to forming a global quality judgment, thereby achieving a quality prediction more aligned with human subjective evaluation. Experimental results demonstrate that the proposed method outperforms mainstream approaches, achieving a PLCC of 0.9734 on the LIVE I database, and a PLCC of 0.9632 on the LIVE II database.

Scalable video coding encodes a video stream into multiple layers so that it can be decoded at different levels of quality/resolution, depending on the device’s capabilities or the available network bandwidth. Recent advances in implicit neural representation (INR)-based video codecs have shown competitive compression performance to both traditional and other learning-based methods. In INR approaches, a neural network is trained to overfit a video sequence, and its parameters are compressed to create a compact representation of the video content. While they achieve promising results, existing INR-based codecs require training separate networks for each resolution/quality of a video, making them challenging for scalable compression. In this paper, we propose Neural representations for Scalable Video Coding (NSVC) that encodes multi-resolution/-quality videos into a single neural network comprising multiple layers. The base layer (BL) of the neural network encodes video streams with the lowest resolution/quality. Enhancement layers (ELs) encode additional information that can be used to reconstruct a higher resolution/quality video during decoding using the BL as a starting point. This multi-layered structure allows the scalable bitstream to be truncated to adapt to the client’s bandwidth conditions or computational decoding requirements. Experimental results show that NSVC outperforms AVC’s Scalable Video Coding (SVC) extension and surpasses HEVC’s scalable extension (SHVC) in terms of VMAF. Additionally, NSVC achieves comparable decoding speeds at high resolutions/qualities.

HTTP Adaptive Streaming (HAS) is a widely adopted method for delivering video content over the Internet, requiring each video to be encoded at multiple bitrates and resolution pairs, known as representations, to adapt to various network conditions and device capabilities. This multi-bitrate encoding introduces significant challenges due to the computational and time-intensive nature of encoding multiple representations. Conventional approaches often encode these videos independently without leveraging similarities between different representations of the same input video. This paper proposes an accelerated multi-resolution encoding strategy that utilizes representations of lower resolutions as references to speed up the encoding of higher resolutions when using Versatile Video Coding (VVC); specifically in VVenC, an optimized open-source software implementation. For multi-resolution encoding, a mid-bitrate representation serves as the reference, allowing interpolated encoded partition data to efficiently guide the partitioning process in higher resolutions. The proposed approach uses shared encoding information to reduce redundant calculations, optimizing partitioning decisions. Experimental results demonstrate that the proposed technique achieves a reduction of up to 17% compared to medium preset in encoding time across videos of varying complexities with minimal BDBR/BDT of 0.12 compared to the fast preset.

In response to the growing demand for high-quality videos, Versatile Video Coding (VVC) was released in 2020, building on the hybrid coding architecture of its predecessor, HEVC, achieving about 50% bitrate reduction for the same visual quality. It introduces more flexible block partitioning, enhancing compression efficiency at the cost of increased encoding complexity. To make efficient use of VVC in practical applications, optimization is essential. VVenC, an optimized open-source VVC encoder, introduces multiple presets to address the trade-off between compression efficiency and encoder complexity. Although an optimized set of encoding tools has been selected for each preset, the rate-distortion (RD) search space in the encoder presets still poses a challenge for efficient encoder implementations. In this paper, we propose Early Termination using Reference Frames (ETRF), which improves the trade-off between encoding efficiency and time complexity and positions itself as a new preset between medium and fast presets. The CTU partitioning map of the reference frames in lower temporal layers is employed to accelerate the encoding of frames in higher temporal layers. The results show a reduction in the encoding time of around 21% compared to the medium preset. Specifically, for videos with high spatial and temporal complexities, which typically require longer encoding times, the proposed method achieves a better trade-off between bitrate savings and encoding time compared to the fast preset.

Blind dehazed image quality assessment (BDQA), which aims to accurately predict the visual quality of dehazed images without any reference information, is essential for the evaluation, comparison, and optimization of image dehazing algorithms. Existing learning-based BDQA methods have achieved remarkable success, while the small scale of DQA datasets limits their performance. To address this issue, in this paper, we propose to adapt Contrastive Language-Image Pre-Training (CLIP), pre-trained on large-scale image-text pairs, to the BDQA task. Specifically, inspired by the fact that the human visual system understands images based on hierarchical features, we take global and local information of the dehazed image as the input of CLIP. To accurately map the input hierarchical information of dehazed images into the quality score, we tune both the vision branch and language branch of CLIP with prompt learning. Experimental results on two authentic DQA datasets demonstrate that our proposed approach, named CLIP-DQA, achieves more accurate quality predictions over existing BDQA methods. The code is available at https://github.com/JunFu1995/CLIP-DQA.

Recent years have witnessed an unprecedented growth of multimodal data in healthcare, ranging from distributed sensors and medical imaging devices (MRI, CT, X-rays) to digital health platforms that integrate audio, video, 3D geometry, and clinical text. The increasing availability of such data presents significant opportunities for computer-aided diagnosis and intelligent healthcare solutions, yet also poses substantial challenges in multimodal integration, large-scale analysis, and real-world deployment. The 2nd International Workshop on Multimedia Computing for Health and Medicine (MCHM'25), held in conjunction with ACM Multimedia 2025, focuses on advanced multimedia computing techniques, including mobile and hardware solutions, for tackling real-world problems in healthcare. The workshop brings together researchers and practitioners in multimedia computing, artificial intelligence, and medicine to explore emerging methods, applications, and systems that have a direct impact on human health.

The rapid expansion of multimedia services, such as video streaming, video conferencing, virtual reality, and cloud gaming, makes maintaining and evaluating high perceptual visual quality essential for user experience and system competitiveness. However, visual content can degrade at multiple stages, including acquisition, compression, transmission, enhancement, and display, where suboptimal enhancement may also introduce artifacts and reduce perceived quality. The core challenge is to reliably measure and predict this perceived quality so that it can be maintained or improved. Perceptual Visual Quality Assessment (PVQA) addresses this by evaluating visual quality from the perspective of human subjects, through subjective studies and objective prediction models. Beyond humans, recent work also extends PVQA to machines and robots, where the goal is to preserve downstream task performance (e.g., segmentation accuracy and planning success) under distortions or bandwidth constraints. This tutorial provides a concise, practice-oriented overview of PVQA: fundamentals and human vision considerations; image and video quality assessment; methods for immersive/3D media; opportunities and challenges in the era of foundation models and GenAI; perceptual optimization loops that close the gap between assessment and decisions in coding, streaming, and embodied perception; and domain applications. Finally, we summarize the key concepts, toolchains, and future opportunities for PVQA to be used in modern multimedia communication.

Cataract surgery is the most frequently performed surgical procedure worldwide, involving the replacement of a patient's clouded eye lens with a synthetic intraocular lens to restore visual acuity. Although typically brief, the operation consists of distinct phases that demand precision and extensive training, traditionally constrained by the limitations of real-time observation under a microscope. To enhance learning and procedural accuracy, modern advancements in stereoscopic video capture and head-mounted displays (HMDs) offer a promising solution. This paper demonstrates the application of stereoscopic cataract surgery videos, visualized through Apple Vision Pro (AVP) and Meta Quest 3, to provide immersive 3D perspectives that enhance depth perception and spatial awareness. An expert evaluation study with experienced surgeons indicates that stereoscopic visualization significantly improves comprehension of spatial relationships and procedural maneuvers, suggesting its potential to revolutionize surgical education and real-time guidance in ophthalmic surgery.

We present a physics-driven 3D dart-throwing interaction system for Apple Vision Pro (AVP), developed using Unity 6 engine and running in augmented reality (AR) mode on the device. The system utilizes the PolySpatial and Apple's ARKit software development kits (SDKs) to ensure hand input and tracking in order to intuitively spawn, grab, and throw virtual darts similar to real darts. The application benefits from physics simulations alongside the innovative no-controller input system of AVP to manipulate objects realistically in an unbounded spatial volume. By implementing spatial distance measurement, scoring logic, and recording user performance, this project enables user studies on quality of experience in interactive experiences. To evaluate the perceived quality and realism of the interaction, we conducted a subjective study with 10 participants using a structured questionnaire. The study measured various aspects of the user experience, including visual and spatial realism, control fidelity, depth perception, immersiveness, and enjoyment. Results indicate high mean opinion scores (MOS) across key dimensions.

The push toward data-driven video processing, combined with recent advances in video coding and streaming technologies, has fueled the need for diverse, large-scale, and high-quality video datasets. However, the limited availability of such datasets remains a key barrier to the development of next-generation video processing solutions. In this paper, we introduce Nature-1k, a large-scale video dataset consisting of 1000 professionally captured 4K Ultra High Definition (UHD) videos, each recorded at 60fps. The dataset covers a wide range of environments, lighting conditions, texture complexities, and motion patterns. To maintain temporal consistency, which is crucial for spatio-temporal learning applications, the dataset avoids scene cuts within the sequences. We further characterize the dataset using established metrics, including spatial and temporal video complexity metrics, as well as colorfulness, brightness, and contrast distribution. Moreover, Nature-1k includes a compressed version to support rapid prototyping and lightweight testing. The quality of the compressed videos is evaluated using four commonly used video quality metrics: PSNR, SSIM, MS-SSIM, and VMAF. Finally, we compare Nature-1k with existing datasets to demonstrate its superior quality and content diversity. The dataset is suitable for a wide range of applications, including Generative Artificial Intelligence (AI), video super-resolution and enhancement, video interpolation, as well as video coding, and adaptive video streaming optimization. Dataset URL: https://cd-athena.github.io/Nature-1k.

Stereoscopic video has long been the subject of research due to its ability to deliver immersive three-dimensional content to a wide range of applications. The dual-view format inherently provides binocular disparity cues that enhance depth perception and realism, making it indispensable for fields such as telepresence, 3D mapping, and robotic vision. Until recently, however, end-to-end pipelines for capturing, encoding, and viewing high-quality stereoscopic video were neither widely accessible nor optimized for consumer-grade devices. Today's smartphones, such as the iPhone Pro, and modern Head-Mounted Displays (HMDs) like the Apple Vision Pro, offer built-in support for stereoscopic video capture, hardware-accelerated encoding, and seamless playback on devices like the Apple Vision Pro and Meta Quest 3, which require minimal user intervention. Apple refers to this streamlined workflow as spatial Video. Making the full stereoscopic video process available to everyone has made new applications possible. Despite these advances, there remains a notable absence of publicly available datasets that include the complete spatial video pipeline on consumer platforms, hindering reproducibility and comparative evaluation of emerging algorithms.

This paper reviews the VCIP 2025 Grand Challenge on Live Broadcasting Video Quality Assessment. The competition aims to foster innovation in both subjective and objective VQA techniques tailored to live broadcasting videos, addressing the unique challenges posed by live streaming impairments while emphasizing the evaluation of QoE. The grand challenge used live broadcasting database LBVD which consists of 1013 videos focusing on distortion in live broadcasting videos. The competition had 14 participants and 5 teams submitted valid solutions for the final testing phase. The proposed solutions have shown significant progress in areas such as combining traditional feature engineering with deep learning models, achieved state-of-the-art performances for LBVD. Team ATHENA-Live-QoE and Team HZX Force tied for the first position. The dataset can be found at https://github.com/cpf0079/LBVD.

This paper presents a block stacking simulation developed for Apple Vision Pro (AVP) using Unity’s PolySpatial framework, designed to study both depth perception in spatial computing and physics comprehension of user-driven kinetic controls in augmented reality (AR). The simulation offers two interactive modes: a tower assembly mode and a removal mode. Each game session includes four stages with the virtual table positioned at various distances to observe user adaptation across varying virtual depths. User input is captured through eye tracking and hand tracking, and block behavior is handled by real-time physics simulation, which includes collision response, gravity, and mass-based interactions. The system supports two physics configurations: raw Unity physics and a modified variant with adjusted material and rigidbody parameters for improved stability and realism. It utilizes spatial computing features such as world anchoring to preserve spatial consistency and depth perception through stereoscopic rendering and dynamic shadows, so that users can better judge the spatial coordinates between virtual blocks and their physical surroundings. The simulation is intended to evaluate how 3D spatial rendering and physically realistic interactions contribute to immersion and task performance in AR environments. To assess user performance, the system records key interaction metrics to support analysis of learning progression, control accuracy, and adaptability across varying distances and physics configurations. This work contributes to the understanding of spatial and physics-based interaction design in AR and may inform future applications in education, simulation, and spatial gaming.

This paper presents ISV-Demo, a novel system for delivering interactive stereoscopic video experiences on head-mounted displays in spatial environments. Unlike traditional flat or real-time VR media, our approach leverages pre-rendered, high-fidelity 3D video to enable immersive, cinematic storytelling enhanced by user agency. Viewers influence narrative direction by making real-time decisions at branching points within the stereoscopic scene. We propose two interaction models: a timeline-based model with embedded prompts, and a loop-based segmented model offering flexible timing and decision persistence. These models define a new paradigm for authored cinematic interaction in extended reality, addressing the gap between passive 3D video and dynamic VR content.

Just Noticeable Difference (JND) thresholds, particularly for quality metrics such as Video Multimethod Assessment Fusion (VMAF), are critical in streaming, helping identify when quality changes become perceptible and reducing redundant bitrate representations. The Satisfied User Ratio (SUR) complements JND by quantifying the percentage of users who do not perceive a difference, offering practical guidance for perceptually optimized streaming. This paper proposes a novel two-branch deep neural network (DNN) for predicting the 75% SUR for VMAF, the encoding level where 75% of viewers cannot perceive degradation. The framework combines handcrafted features (e.g., spatial and temporal indicators such as SI, TI, etc.) and deep learning-based (DL-based) representations extracted via a convolutional neural network (CNN) backbone. The DL-based branch employs a spatio-temporal attention mechanism and a Long Short-Term Memory (LSTM) to capture temporal dynamics, while the handcrafted branch encodes interpretable indicators through a fully connected layer. Both outputs are fused and passed through a lightweight Multilayer Perceptron (MLP) to predict 75% SUR. To improve robustness to noise and label uncertainty, the model is trained using the Smooth-L1 loss. Experiments on the VideoSet dataset show our method outperforms SOTA across all metrics, achieving a notably higher R2score (0.46 vs. 0.36), indicating improved prediction reliability and low computational complexity, making it suitable for real-time video streaming.

Evaluating perceived video quality is essential for ensuring high Quality of Experience (QoE) in modern streaming applications. While existing subjective datasets and Video Quality Metrics (VQMs) cover a broad quality range, many practical use cases—especially for premium users—focus on high-quality scenarios requiring finer granularity. Just Noticeable Difference (JND) has emerged as a key concept for modeling perceptual thresholds in these high-end regions and plays an important role in perceptual bitrate ladder construction. However, the relationship between JND and the more widely used Mean Opinion Score (MOS) remains unclear. In this paper, we conduct a Degradation Category Rating (DCR) subjective study based on an existing JND dataset to examine how MOS corresponds to the 75% Satisfied User Ratio (SUR) points of the 1stand 2ndJNDs. We find that while MOS values at JND points generally align with theoretical expectations (e.g., 4.75 for the 75% SUR of the 1stJND), the reverse mapping—from MOS to JND—is ambiguous due to overlapping confidence intervals across PVS indices. Statistical significance analysis further shows that DCR studies with limited participants may not detect meaningful differences between reference and JND videos.

InIn HTTP Adaptive Streaming (HAS), each video is divided into smaller segments, and each segment is encoded at multiple pre-defined bitrates to construct abitrate ladder. To optimize bitrate ladders, per-title encoding approaches encode each segment at various bitrates and resolutions to determine the convex hull. From the convex hull, an optimized bitrate ladder is constructed, resulting in an increased Quality of Experience (QoE) for end-users. With the ever-increasing efficiency of deep learning-based video enhancement approaches, they are more and more employed at the client-side to increase the QoE, specifically when GPU capabilities are available. Therefore, scalable approaches are needed to support end-user devices with both CPU and GPU capabilities (denoted as CPU-only and GPU-available end-users, respectively) as a new dimension of a bitrate ladder. To address this need, we proposeDeepStream, ascalable content-awareper-title encoding approach to support both CPU-only and GPU-available end-users. (i) To supportbackward compatibility,DeepStreamconstructs a bitrate ladder based on any existing per-title encoding approach. Therefore, the video content will be provided for legacy end-user devices with CPU-only capabilities as a base layer (BL). (ii) For high-end end-user devices with GPU capabilities, an enhancement layer (EL) is added on top of the base layer comprising lightweight video super-resolution deep neural networks (DNNs) for each bitrate-resolution pair of the bitrate ladder. A content-aware video super-resolution approach leads to higher video quality, however, at the cost of bitrate overhead. To reduce the bitrate overhead for streaming content-aware video super-resolution DNNs,DeepCABAC, context-adaptive binary arithmetic coding for DNN compression, is used. Furthermore, the similarity among (i) segments within a scene and (ii) frames within a segment are used to reduce the training costs of DNNs. Experimental results show bitrate savings of 34% and 36% to maintain the same PSNR and VMAF, respectively, for GPU-available end-users, while the CPU-only users get the desired video content as usual.

In HTTP Adaptive Streaming (HAS), a video is encoded at various bitrate-resolution pairs, collectively known as the bitrate ladder, allowing users to select the most suitable representation based on their network conditions. Optimizing this set of pairs to enhance the Quality of Experience (QoE) requires accurately measuring the quality of these representations. VMAF and ITU-T's P.1204.3 are highly reliable metrics for assessing the quality of representations in HAS. However, in practice, using these metrics for optimization is often impractical for live streaming applications due to their high computational costs and the large number of bitrate-resolution pairs in the bitrate ladder that need to be evaluated. To address their high complexity, our paper introduces a new method calledVQM4HAS, which extractslow-complexityfeatures, including ($i$) video complexity features, ($ii$) frame-level encoding statistics logged during the encoding process, and ($iii$) lightweight video quality metrics. These extracted features are then fed into a regression model to predict VMAF or P.1204.3. TheVQM4HASmodel is designed to operate on a per bitrate-resolution pair, per-resolution, and cross-representation basis, optimizing quality predictions across different scenarios. Our experimental results demonstrate thatVQM4HASachieves a high correlation with VMAF and P.1204.3, with Pearson correlation coefficients (PCC) ranging from 0.95 to 0.96 for VMAF and 0.97 to 0.99 for P.1204.3, depending on the resolution. Despite achieving a high correlation with VMAF and P.1204.3,VQM4HASexhibits significantly less complexity than both metrics, with 98% and 99% less complexity for VMAF and P.1204.3, respectively, making it suitable for live streaming scenarios. We also conduct a feature importance analysis to further reduce the complexity of the proposed method. Furthermore, we evaluate the effectiveness of our method by using it to predict subjective quality scores. The results show thatVQM4HASachieves a higher correlation with subjective scores at various resolutions despite its minimal complexity. The source code is available athttps://github.com/cd-athena/VQM4HAS.

Existing JPEG encryption approaches pose a security risk due to the difficulty in changing all block-feature values while considering format compatibility and file size expansion. To address these concerns, this paper introduces a novel JPEG image encryption scheme. First, the security of sketch information against chosen-plaintext attacks is improved by increasing the change rate of block-feature values. Second, a classification global permutation approach is designed to encrypt the undivided run/size, value (RSV)-based AC groups to achieve larger changes in the block-feature values. Third, to reduce file size expansion while maintaining format compatibility, the DC coefficients are rotated based on the mapped DC differences in the same category, and the nonzero AC coefficients are mapped in the same category. Extensive experiments demonstrate that the proposed algorithm is superior to existing schemes in terms of security. Notably, the average change rate of block-feature values is increased by at least 20%. Furthermore, the proposed scheme reduces the file size by an average of 2.036% compared to existing JPEG image encryption methods.

Reversible data hiding in encrypted images (RDH-EI) has gained widespread attention due to its potential applications in secure cloud storage. However, the security challenges of RDH-EI in cloud storage scenarios remain largely unexplored. In this article, we present a counterfeiting attack on RDH-EI schemes that utilize block-permutation and Co-XOR (BPCX) encryption. We demonstrate that ciphertext images generated by BPCX-based RDH-EI are easily tampered with to produce a counterfeit decrypted image with different contents imperceptible to the human eye. This vulnerability is mainly because the block permutation key information of BPCX is susceptible to known-plaintext attacks (KPAs). Taking ciphertext images in telemedicine scenarios as an example, we describe two potential counterfeiting attacks, namely fixed-area and optimal-area attacks. We show that the quality of forged decrypted images depends on the accuracy of the estimated block-permutation key under KPA conditions. To improve the invisibility of counterfeit decrypted images, we analyze the limitations of existing KPA methods against BPCX encryption for \(2\times 2\) block sizes and propose a novel diagonal inversion rule specifically designed for image blocks. This rule further enhances the accuracy of the estimated block-permutation key. The experiments show that, compared to existing KPA methods, the accuracy of the estimated block-permutation key in the UCID dataset increases by an average of 11.5%. In the counterfeiting attack experiments on Camera’s encrypted image, we successfully tampered with over 80% of the pixels in the target area under the fixed-region attack. Additionally, we achieved a tampering success rate exceeding 90% in the optimal-region attack.

HTTP adaptive streaming (HAS) has emerged as a prevalent approach for over-the-top (OTT) video streaming services due to its ability to deliver a seamless user experience. A fundamental component of HAS is the bitrate ladder, which comprises a set of encoding parameters (e.g., bitrate-resolution pairs) used to encode the source video into multiple representations. This adaptive bitrate ladder enables the client’s video player to dynamically adjust the quality of the video stream in real-time based on fluctuations in network conditions, ensuring uninterrupted playback by selecting the most suitable representation for the available bandwidth. The most straightforward approach involves using a fixed bitrate ladder for all videos, consisting of pre-determined bitrate-resolution pairs known as one-size-fits-all . Conversely, the most reliable technique relies on intensively encoding all resolutions over a wide range of bitrates to build the convex hull , thereby optimizing the bitrate ladder by selecting the representations from the convex hull for each specific video. Several techniques have been proposed to predict content-based ladders without performing a costly, exhaustive search encoding. This article provides a comprehensive review of various convex hull prediction methods, including both conventional and learning-based approaches. Furthermore, we conduct a benchmark study of several handcrafted- and deep learning (DL)-based approaches for predicting content-optimized convex hulls across multiple codec settings. The considered methods are evaluated on our proposed large-scale dataset, which includes 300 UHD video shots encoded with software and hardware encoders using three state-of-the-art video standards, including AVC/H.264, HEVC/H.265, and VVC/H.266, at various bitrate points. Our analysis provides valuable insights and establishes baseline performance for future research in this field ( Dataset URL : https://nasext-vaader.insa-rennes.fr/ietr-vaader/datasets/br_ladder ).

Video streaming has evolved from push-based, broad-/multicasting approaches with dedicated hard-/software infrastructures to pull-based unicast schemes utilizing existing Web-based infrastructure to allow for better scalability. In this article, we provide an overview of the foundational principles of HTTP Adaptive Streaming (HAS), from video encoding to end user consumption, while focusing on the key advancements in adaptive bitrate algorithms, Quality of Experience (QoE), and energy efficiency. Furthermore, the article highlights the ongoing challenges of optimizing network infrastructure, minimizing latency, and managing the environmental impact of video streaming. Finally, future directions for HAS, including immersive media streaming and neural network-based video codecs, are discussed, positioning HAS at the forefront of next-generation video delivery technologies.

The evolving landscape of the delivery of multimedia content requires a deep understanding of how the design of the bitrate ladder (for HTTP adaptive streaming) influences cost and quality. This paper explores the use of Just Noticeable Differences (JND) to select bitrate-resolution pairs for constructing a bitrate ladder with respect to the proportion of satisfied user ratio (SUR). To expand the investigation to various codecs, first, a method is explained that transfers the JND points obtained through subjective testing from one codec (e.g., , AVC) to other codecs (e.g., , HEVC, VVC). This approach helps avoid the additional costs associated with conducting subjective tests to obtain JND points for a wide range of different codecs. To achieve this objective, we investigate the codec-agnostic nature of various video quality metrics, followed by the transfer of JND between two codecs, taking into account the most suitable codec-agnostic video quality metric. Secondly, we delve into the analysis of the bitrate cost of a given bitrate ladder from a JND perspective, i.e., , as a function of the SUR. Among others, our experimental results demonstrate that increasing SUR leads to an exponential increase in bitrate. For example, to raise the SUR from 75% to 90%, it is necessary to double the video bitrate.

In HTTP Adaptive Streaming (HAS), a video is encoded at multiple bitrate-resolution pairs, referred to as representations, which enables users to choose the most suitable representation based on their network connection. To optimize the set of bitrate-resolution pairs and improve the Quality of Experience (QoE) for users, it is of utmost importance to measure the quality of the representations. VMAF is a highly reliable metric used in HAS to assess the quality of representations. However, in practice, using it for optimization can be a very time-consuming process, and it is infeasible for live streaming applications. To tackle its high complexity, our paper introduces a new method called VQM4HAS, which extracts low-complexity features, including (i) video complexity features, (ii) bitstream features logged during the encoding process, and (iii) basic video quality metrics. These extracted features are then fed into a regression model to predict VMAF. Our experimental results demonstrate that VQM4HAS achieves a high Pearson Correlation Coefficient (PCC) with VMAF, ranging from 0.95 to 0.96 depending on the resolution. However, it exhibits significantly lower complexity, making it suitable for live streaming scenarios.

With the ubiquity of video streaming, optimizing the delivery of video content while reducing energy consumption has become increasingly critical. Traditional adaptive streaming relies on a fixed set of bitrate-resolution pairs, known as bitrate ladders, for encoding. However, this "one-size-fits-all" approach is suboptimal for diverse video content. As a result, per-title encoding approaches dynamically select the bitrate ladder for each content. In this paper, we address the pressing issue of increasing energy consumption in video streaming by introducing GreenRes, a novel approach that goes beyond the traditional selection of quality-centric resolutions. Instead, GreenRes considers both video quality and energy consumption to construct an optimal bitrate ladder tailored to the unique characteristics of each video content. To achieve this, GreenRes, similar to per-title encoding, encodes each video content at various resolutions, each with a set of bitrates. It then sets a maximum acceptable quality drop threshold and selects resolutions that maintain video quality above this threshold while minimizing energy consumption. Our experimental results demonstrate an average reduction in energy consumption of 30.82%, while ensuring a maximum quality drop of only 0.53 Video Multimethod Assessment Fusion (VMAF) points.

Omnidirectional or 360-degree video is being increasingly deployed, largely due to the latest advancements in immersive virtual reality (VR) and extended reality (XR) technology. However, the adoption of these videos in streaming encounters challenges related to bandwidth and latency, particularly in mobility conditions such as with unmanned aerial vehicles (UAVs). Adaptive resolution and compression aim to preserve quality while maintaining low latency under these constraints, yet downscaling and encoding can still degrade quality and introduce artifacts. Machine learning (ML)-based super-resolution (SR) and quality enhancement techniques offer a promising solution by enhancing detail recovery and reducing compression artifacts. However, current publicly available 360-degree video SR datasets lack compression artifacts, which limit research in this field. To bridge this gap, this paper introduces omnidirectional video streaming dataset (ODVista), which comprises 200 high-resolution and high-quality videos downscaled and encoded at four bitrate ranges using the high-efficiency video coding (HEVC)/H.265 standard. Evaluations show that the dataset not only features a wide variety of scenes but also spans different levels of content complexity, which is crucial for robust solutions that perform well in real-world scenarios and generalize across diverse visual environments. Additionally, we evaluate the performance, considering both quality enhancement and runtime, of two handcrafted and two ML-based SR models on the validation and testing sets of ODVista. Dataset URL: https://github.com/Omnidirectional-video-group/ODVista

The optimization of video compression and streaming workflows critically relies on understanding the video complexity, including both spatial and temporal features. These features play a vital role in guiding rate control, predicting video encoding parameters (such as resolution and frame rate), and selecting test videos for subjective analysis. Traditional methods primarily utilize Spatial Information (SI) and Temporal Information (TI) to measure these spatial and temporal complexity features, respectively. Moreover, the Video Complexity Analyzer (VCA) has been introduced as a tool employing Discrete Cosine Transform (DCT)-based functions, namely E and h, to evaluate the spatial and temporal complexity features, respectively. In this paper, we introduce the Enhanced Video Complexity Analyzer (EVCA), an advanced tool that integrates the functionalities of both VCA and the SITI approach. Developed in Python to ensure compatibility with GPU processing, EVCA enhances the definition of temporal complexity originally used in VCA. This refinement significantly improves the detection of temporal complexity features in VCA (i.e., h), raising its Pearson Correlation Coefficient (PCC) from 0.6 to 0.77. Furthermore, EVCA demonstrates exceptional performance on Graphics Processing Unit (GPU) devices, achieving feature extraction speeds exceeding 1200 fps for 1080p resolution videos.

Video streaming constitutes 65 % of global internet traffic, prompting an investigation into its energy consumption and CO2 emissions. Video encoding, a computationally intensive part of streaming, has moved to cloud computing for its scalability and flexibility. However, cloud data centers' energy consumption, especially video encoding, poses environmental challenges. This paper presents VEED, a FAIR Video Encoding Energy and CO2 Emissions Dataset for Amazon Web Services (AWS) EC2 instances. Additionally, the dataset also contains the duration, CPU utilization, and cost of the encoding. To prepare this dataset, we introduce a model and conduct a benchmark to estimate the energy and CO2 emissions of different Amazon EC2 instances during the encoding of 500 video segments with various complexities and resolutions using Advanced Video Coding (AVC) and High-Efficiency Video Coding (HEVC). VEED and its analysis can provide valuable insights for video researchers and engineers to model energy consumption, manage energy resources, and distribute workloads, contributing to the sustainability of cloud-based video encoding and making them cost-effective. VEED is available at https://github.com/cd-athena/VEED-dataset.

HTTP Adaptive Streaming (HAS) has emerged as the predominant solution for delivering video content on the Internet. The urgency of the climate crisis has accentuated the demand for investigations into the environmental impact of HAS techniques. In HAS, clients rely on adaptive bitrate (ABR) algorithms to drive the quality selection for video segments. Focusing on maximizing video quality, these algorithms often prioritize maximizing video quality under favorable network conditions, disregarding the impact of energy consumption. To thoroughly investigate the effects of energy consumption, including the impact of bitrate and other video parameters such as resolution and codec, further research is still needed. In this paper, we propose COCONUT, a COntent COnsumption eNergy measUrement daTaset for adaptive video streaming collected through a digital multimeter on various types of client devices, such as laptop and smartphone, streaming MPEG-DASH segments. Furthermore, we analyze the dataset and find insights into the influence of multiple codecs, various video encoding parameters, such as segment length, framerate, bitrates, and resolutions, and decoding type, i.e., hardware or software, on energy consumption. We gather and categorize these measurements based on segment retrieval through the network interface card (NIC), decoding, and rendering. Additionally, we compare the impact of different HAS players on energy consumption. This research offers valuable perspectives on the energy usage of streaming devices, which could contribute to creating a media consumption experience that is both more sustainable and resource-efficient. Dataset URL: https://athena.itec.aau.at/coconut/.

Just Noticeable Difference (JND) establishes the threshold between two images or videos wherein differences in quality remain imperceptible to an individual. This threshold, collectively known as the Satisfied User Ratio (SUR), holds significant importance in image and video compression applications, ensuring that differences in quality are imperceptible to the majority (p%) of users, known as p%SUR. While substantial efforts have been dedicated to predicting the p%SUR for various encoding parameters (e.g., QP) and quality metrics (e.g., VMAF), referred to as proxies, systematic consideration of the prediction uncertainties associated with these proxies has hitherto remained unexplored. In this paper, we analyze the uncertainty of p%SUR through Confidence Interval (CI) estimation and assess the consistency of various Video Quality Metrics (VQMs) as proxies for SUR. The analysis reveals challenges in directly using p%SUR as ground truth for training models and highlights the need for uncertainty estimation for SUR with different proxies.

In the field of video streaming, the optimization of video encoding and decoding processes is crucial for delivering high-quality video content. Given the growing concern about carbon dioxide emissions, it is equally necessary to consider the energy consumption associated with video streaming. Therefore, to take advantage of machine learning techniques for optimizing video delivery, a dataset encompassing the energy consumption of the encoding and decoding process is needed. This paper introduces a comprehensive dataset featuring diverse video content, encoded and decoded using various codecs and spanning different devices. The dataset includes 1000 videos encoded with four resolutions (2160p, 1080p, 720p, and 540p) at two frame rates (30fps and 60fps), resulting in eight unique encodings for each video. Each video is further encoded with four different codecs — AVC (libx264), HEVC (libx265), AV1 (libsvtav1), and VVC (VVenC) — at four quality levels defined by QPs of 22, 27, 32 and 37. In addition, for AV1, three additional QPs of 35, 46 and 55 are considered. We measure both encoding and decoding time and energy consumption on various devices to provide a comprehensive evaluation, employing various metrics and tools. Additionally, we assess encoding bitrate and quality using quality metrics such as PSNR, SSIM, MS-SSIM, and VMAF. All data and the reproduction commands and scripts have been made publicly available as part of the dataset, which can be used for various applications such as rate and quality control, resource allocation, and energy-efficient streaming.Dataset URL: https://github.com/cd-athena/MVCD.

As video dimensions – including resolution, frame rate, and bit depth – increase, a larger bitrate is required to maintain a higher Quality of Experience (QoE). While videos are often optimized for resolution and frame rate to improve compression and energy efficiency, the impact of color space is often overlooked. Larger color spaces are essential for avoiding color banding and delivering High Dynamic Range (HDR) content with richer, more accurate colors, although this comes at the cost of higher processing energy. This paper investigates the effects of bit depth and color subsampling on video compression efficiency and energy consumption. By analyzing different bit depths and subsampling schemes, we aim to determine optimized settings that balance compression efficiency with energy consumption, ultimately contributing to more sustainable and high-quality video delivery. We evaluate both encoding and decoding energy consumption and assess the quality of videos using various metrics including PSNR, VMAF, ColorVideoVDP, and CAMBI. Our findings offer valuable insights for video codec developers and content providers aiming to improve the performance and environmental footprint of their video streaming services.

Measuring the complexity of visual content is crucial in various applications, such as selecting sources to test processing algorithms, designing subjective studies, and efficiently determining the appropriate encoding parameters and bandwidth allocation for streaming. While spatial and temporal complexity measures exist for 2D videos, a geometric complexity measure for 3D content is still lacking. In this paper, we present the first study to characterize the geometric complexity of 3D point clouds. Inspired by existing complexity measures, we propose several compression-based definitions of geometric complexity derived from the rate-distortion curves obtained by compressing a dataset of point clouds using G-PCC. Additionally, we introduce density-based and geometry-based descriptors to predict complexity. Our initial results show that even simple density measures can accurately predict the geometric complexity of point clouds.

With the development of Internet of Things (IoT) and cloud services, many images generated from IoT devices are stored in the cloud, calling for efficient data encryption methods. To balance the security and usability, the thumbnail preserving encryption (TPE) has emerged. However, existing JPEG image-based TPE (JPEG-TPE) schemes face challenges in achieving low file extension, lossless decryption and better privacy protect of detailed information. To solve these challenges, we propose a novel JPEG-TPE scheme. Firstly, to achieve a smaller file size expansion and preserve the thumbnail, we reallocate the values, maintaining the sum for the DC difference instead of the DC coefficient. To ensure that the coefficients do not overflow, the valid range of reallocated difference is constrained not only by the sum but also by the neighborhood difference. Secondly, to preserve file size of AC encryption while improve the security of detailed information, the AC coefficient groups with undivided RSV are permuted adaptively. Besides, the intra TPE block swapping of DC difference, quantization table modification, non-zero AC coefficients mapping, and block permutation are used to further encrypt the image. The experimental results show that the proposed JPEG-TPE scheme achieves lossless decryption, reducing the file size expansion of encrypted images from 15.41% to 0.64% compared to the state-of-the-art scheme. Additionally, it is observed that the proposed method can effectively resist against various attacks, including the deep-learning based super-resolution attack.

Abstract HTTP Adaptive Streaming (HAS) plays a key role in over-the-top video streaming with the ability to reduce the video stall duration by adapting the quality of transmitted video segments to the network conditions. However, HAS still suffers from two problems. First, it incurs variations in video quality because of throughput fluctuation. Adaptive bitrate (ABR) algorithms at the HAS client usually select a low-quality segment when the throughput drops to avoid stall events, which impairs the Quality of Experience (QoE) of the end-users. Second, many ABR algorithms choose the lowest-quality segments at the beginning of a video streaming session to ramp up the playout buffer early on. Although this strategy decreases the startup time, clients can be annoyed as they have to watch a low-quality video initially. To address these issues, we introduced the H2BR technique (H TTP/2-B ased R etransmission) (Nguyen et al. 33) that utilizes certain features of HTTP/2 (including server push, multiplexing, stream priority, and stream termination) for late transmissions of higher-quality versions of video segments already in the client buffer, in order to improve video quality. Although H2BR was shown to enhance the QoE, limited streaming scenarios were considered resulting in a lack of general conclusions on H2BR’s performance. Thus, this article provides a profound evaluation to answer three open questions: (i) how H2BR’s performance is impacted by parameters at the server side (i.e., various encoding specifications), at the network side (i.e., packet loss rate), and at the client side (i.e., buffer size) on the performance of H2BR; (ii) how H2BR outperforms other state-of-the-art approaches in different configurations of the parameters above; (iii) how to effectively utilize H2BR on top of ABR algorithms in various streaming scenarios. The experimental results show that H2BR’s performance increases with the buffer size and decreases with increasing packet loss rates and/or video segment duration. The number of quality levels can negatively or positively impact on H2BR’s performance, depending on the ABR algorithm deployed. In general, H2BR is able to enhance the video quality by up to 17% and 14% in scalable video streaming and in non-scalable video streaming, respectively. Compared with an existing retransmission technique (i.e., SQUAD Wang et al., ACM Trans Multimed Comput Commun Applic (TOMM) 13(3s): 45, 49), H2BR shows better results with more than 10% in QoE and 9% in the average video quality.

Video streaming and its applications are growing rapidly, making video optimization a primary target for content providers looking to enhance their services. Enhancing the quality of videos requires the adjustment of different encoding parameters such as bitrate, resolution, and frame rate. To avoid brute force approaches for predicting optimal encoding parameters, video complexity features are typically extracted and utilized. To predict optimal encoding parameters effectively, content providers traditionally use unsupervised feature extraction methods, such as ITU-T’s Spatial Information (SI) and Temporal Information (TI) to represent the spatial and temporal complexity of video sequences. Recently, Video Complexity Analyzer (VCA) was introduced to extract DCT-based features to represent the complexity of a video sequence (or parts thereof). These unsupervised features, however, cannot accurately predict video encoding parameters. To address this issue, this paper introduces a novel supervised feature extraction method named DeepVCA, which extracts the spatial and temporal complexity of video sequences using deep neural networks. In this approach, the encoding bits required to encode each frame in intra-mode and inter-mode are used as labels for spatial and temporal complexity, respectively. Initially, we benchmark various deep neural network structures to predict spatial complexity. We then leverage the similarity of features used to predict the spatial complexity of the current frame and its previous frame to rapidly predict temporal complexity. This approach is particularly useful as the temporal complexity may depend not only on the differences between two consecutive frames but also on their spatial complexity. Our proposed approach demonstrates significant improvement over unsupervised methods, especially for temporal complexity. As an example application, we verify the effectiveness of these features in predicting the encoding bitrate and encoding time of video sequences, which are crucial tasks in video streaming. The source code and dataset is available at https://github.com/cd-athena/ DeepVCA.

HTTP Adaptive Streaming (HAS) methods divide a video into smaller segments, encoded at multiple pre-defined bitrates to construct a bitrate ladder. Bitrate ladders are usually optimized per title over several dimensions, such as bitrate, resolution, and framerate. This paper adds a new dimension to the bitrate ladder by considering the energy consumption of the encoding process. Video encoders often have multiple pre-defined presets to balance the trade-off between encoding time, energy consumption, and compression efficiency. Faster presets disable certain coding tools defined by the codec to reduce the encoding time at the cost of reduced compression efficiency. Firstly, this paper evaluates the energy consumption and compression efficiency of different x265 presets for 500 video sequences. Secondly, optimized presets are selected for various representations in a bitrate ladder based on the results to guarantee a minimal drop in video quality while saving energy. Finally, a new per title model, which optimizes the trade-off between compression efficiency and energy consumption, is proposed. The experimental results show that decreasing the VMAF score by 0.15 and 0.39 while choosing an optimized preset results in encoding energy savings of 70% and 83%, respectively.

In video streaming applications, a fixed set of bitrate-resolution pairs (known as a bitrate ladder) is typically used during the entire streaming session. However, an optimized bitrate ladder per scene may result in (i) decreased storage or delivery costs or/and (ii) increased Quality of Experience. This paper introduces a Just Noticeable Difference (JND)-aware perscene bitrate ladder prediction scheme (JASLA) for adaptive video-on-demand streaming applications. JASLA predicts jointly optimized resolutions and corresponding constant rate factors (CRFs) using spatial and temporal complexity features for a given set of target bitrates for every scene, which yields an efficient constrained Variable Bitrate encoding. Moreover, bitrate-resolution pairs that yield distortion lower than one JND are eliminated. Experimental results show that, on average, JASLA yields bitrate savings of 34.42% and 42.67% to maintain the same PSNR and VMAF, respectively, compared to the reference HTTP Live Streaming (HLS) bitrate ladder Constant Bitrate encoding using x265 HEVC encoder, where the maximum resolution of streaming is Full HD (1080p). Moreover, a 54.34% average cumulative decrease in storage space is observed.

Despite remarkable advances, current Extended Reality (XR) applications are in their majority local and individual experiences. A plethora of interactive applications, such as teleconferencing, telesurgery, interconnection in new buildings project chain, Cultural Heritage, and Museum contents communication, are well on their way to integrating immersive technologies. However, interconnected, and interactive XR, where participants can virtually interact across vast distances, remains a distant dream. In fact, three great barriers stand between current technology and remote immersive interactive life-like experiences, namely (i) content realism, (ii) motion-to-photon latency, and accurate (iii) human-centric quality assessment and control. Overcoming these barriers will require novel solutions at all elements of the end-to-end transmission chain. This workshop focuses on the challenges, applications, and major advancements in multimedia, networks, and end-user infrastructures to enable the next generation of interactive XR applications and services.

Video content in Live HTTP Adaptive Streaming (HAS) is typically encoded using a pre-defined, fixed set of bitrate-resolution pairs (termed Bitrate Ladder), allowing play-back devices to adapt to changing network conditions using an adaptive bitrate (ABR) algorithm. However, using a fixed one-size-fits-all solution when faced with various content complexities, heterogeneous network conditions, viewer device resolutions and locations, does not result in an overall maximal viewer quality of experience (QoE). Here, we consider these factors and design LALISA, an efficient framework for dynamic bitrate ladder optimization in live HAS. LALISA dynamically changes a live video session’s bitrate ladder, allowing improvements in viewer QoE and savings in encoding, storage, and bandwidth costs. LALISA is independent of ABR algorithms and codecs, and is deployed along the path between viewers and the origin server. In particular, it leverages the latest developments in video analytics to collect statistics from video players, content delivery networks and video encoders, to perform bitrate ladder tuning. We evaluate the performance of LALISA against existing solutions in various video streaming scenarios using a trace-driven testbed. Evaluation results demonstrate significant improvements in encoding computation (24.4%) and bandwidth (18.2%) costs with an acceptable QoE.

The integration of advanced video codecs into the streaming pipeline is growing in response to the increasing demand for high quality video content. However, the significant computational demand for advanced codecs like Versatile Video Coding (VVC) poses challenges for service providers, including longer encoding time and higher encoding cost. This challenge becomes even more pronounced in streaming, as the same content needs to be encoded at multiple bitrates (also known as representations) to accommodate different network conditions. To accelerate the encoding process of multiple representations of the same content in VVC, we employ the encoding map of a single representation, known as the reference representation, and utilize its partitioning structure to accelerate the encoding of the remaining representations, referred to as dependent representations. To ensure compatibility with parallel processing, we designate the lowest bitrate representation as the reference representation. The experimental results indicate a substantial improvement in the encoding time for the dependent representations, achieving an average reduction of 40%, while maintaining a minimal average quality drop of only 0.43 in Video Multi-method Assessment Fusion (VMAF). This improvement is observed when utilizing Versatile Video Encoder (VVenC), an open and optimized VVC encoder implementation.

In this paper, we propose a novel attack model called the Got Plaintext Attack (GPA), where the attacker only requires one plaintext and the ciphertext image set stored in the cloud to attack the content of the ciphertext image. Using this model, we examine the security of the Improved Redundant Space Transfer (IRST) encryption method. To this end, we define an ordered characteristic matrix based on the properties of the three keys used in IRST. By comparing the histogram distance of the ordered characteristic matrix, we are able to obtain a plain-ciphertext pair. Furthermore, by leveraging the invariant properties of the ordered characteristic matrix of image blocks in the plain-ciphertext pair, we estimate the block permutation Π2and the bit-plane permutation sequence Π1. Our experiments show that the accuracy of estimating Π2is higher than 70% for block sizes of 3x3 pixels or larger. Despite a 40% accuracy in estimating Π1, the content information of the ciphertext image can still be exposed.

In adaptive video streaming, optimizing the selection of representations for the encoding bitrate ladder has a significant impact on the quality and economics of media delivery. An efficient way to select representations for the bitrate ladder of a given clip is to consider the Satisfied User Ratio (SUR) of the perceived quality of consecutive representations. This ensures that only representations with one Just Noticeable Difference (JND) are encoded and streamed by avoiding encoding similar-quality representations. VMAF (Video Multi-method Assessment Fusion) presently stands as the most commonly utilized quality metric for constructing bitrate ladders. Hence, the precise determination of JND-optimal encoding step-sizes for the VMAF proxy holds paramount importance; nevertheless, this task is intricate and can present considerable challenges.In this paper, we evaluate the effectiveness of different Video Quality Metrics (VQMs) in predicting SUR for the VMAF proxy to better capture content-specific characteristics. Our experimental results provide evidence that incorporating VQMs can improve the precision of the SUR prediction for the VMAF proxy. Compared to a state-of-the-art approach that utilizes video complexity metrics, our proposed approach, which incorporates two quality metrics—specifically, VMAF and SSIM calculated at an optimized quantization parameter (QP)—achieves a substantially reduced Mean Absolute Error (MAE) of 1.67. In contrast, the state-of-the-art approach yields an MAE of 2.01. Hence, we recommend using the above quality metrics to improve the accuracy of the SUR prediction for the VMAF proxy.

In general, manipulated videos will eventually undergo recompression. Video transcoding will occur when the standard of recompression is different from the prior standard. Therefore, as a special sign of recompression, video transcoding can also be considered evidence of forgery in video forensics. In this paper, we focus on the detection and localization of video transcoding from AVC to HEVC (AVC-HEVC). There are two probable cases of AVC-HEVC transcoding — whole video transcoding and partial frame transcoding. However, the existing forensic methods only consider the detection of whole video transcoding, and they do not consider partial frame transcoding localization. In view of this, we propose a framewise scheme based on a convolutional neural network. First, we analyze that the essential difference between AVC-HEVC and HEVC is reflected in the high-frequency components of decoded frames. Then, the partition and location information of prediction units (PUs) are introduced to generate frame-level PU maps to make full use of the local artifacts of PUs. Finally, taking the decoded frames and PU maps as inputs, a dual-path network including specific convolutional modules and an adaptive fusion module is proposed. Through it, the artifacts on a single frame can be better extracted, and the transcoded frames can be detected and localized. Coupled with a simple voting strategy, the results of whole transcoding detection can be easily obtained. A large number of experiments are conducted to verify the performances. The results show that the proposed scheme outperforms or rivals the state-of-the-art methods in AVC-HEVC transcoding detection and localization.

The edge computing paradigm brings cloud capabilities close to the clients. Leveraging the edge’s capabilities can improve video streaming services by employing the storage capacity and processing power at the edge for caching and transcoding tasks, respectively, resulting in video streaming services with higher quality and lower latency. In this paper, we propose, a Cost-and Delay-aware Light-weight Transcoding approach at the Edge, in the context of HTTP Adaptive Streaming (HAS). The encoding of a video segment requires computationally intensive search processes. The main idea of is to store the optimal search results as metadata for each bitrate of video segments and reuse it at the edge servers to reduce the required time and computational resources for transcoding. Aiming at minimizing the cost and delay of Video-on-Demand (VoD) services, we formulate the problem of selecting an optimal policy for serving segment requests at the edge server, including (i) storing at the edge server, (ii) transcoding from a higher bitrate at the edge server, and (iii) fetching from the origin or a CDN server, as a Binary Linear Programming (BLP) model. As a result, stores the popular video segments at the edge and serves the unpopular ones by transcoding using metadata or fetching from the origin/CDN server. In this way, in addition to the significant reduction in bandwidth and storage costs, the transcoding time of a requested segment is remarkably decreased by utilizing its corresponding metadata. Moreover, we prove the proposed BLP model is an NP-hard problem and propose two heuristic algorithms to mitigate the time complexity of . We investigate the performance of in comprehensive scenarios with various video contents, encoding software, encoding settings, and available resources at the edge. The experimental results show that our approach (i) reduces the transcoding time by up to 97%, (ii) decreases the streaming cost, including storage, computation, and bandwidth costs, by up to 75%, and (iii) reduces delay by up to 48% compared to approaches.

In HTTP Adaptive Streaming (HAS), videos are encoded at multiple bitrates and spatial resolutions ( i.e. , representations ) to adapt to the heterogeneity of network conditions, device attributes, and end-user preferences. Encoding the same video segment at multiple representations increases costs for content providers. State-of-the-art multi-encoding schemes improve the encoding process by utilizing encoder analysis information from already encoded representation(s) to reduce the encoding time of the remaining representations. These schemes typically use the highest bitrate representation as the reference to accelerate the encoding of the remaining representations. Nowadays, most streaming services utilize cloud-based encoding techniques, enabling a fully parallel encoding process to reduce the overall encoding time. The highest bitrate representation has the highest encoding time than the other representations. Thus, utilizing it as the reference encoding is unfavorable in a parallel encoding setup as the overall encoding time is bound by its encoding time. This paper provides a comprehensive study of various multi-rate and multi-encoding schemes in both serial and parallel encoding scenarios. Furthermore, it introduces novel heuristics to limit the Rate Distortion Optimization (RDO) process across various representations. Based on these heuristics, three multi-encoding schemes are proposed, which rely on encoder analysis sharing across different representations: (i) optimized for the highest compression efficiency , (ii) optimized for the best compression efficiency-encoding time savings trade-off , and (iii) optimized for the best encoding time savings . Experimental results demonstrate that the proposed multi-encoding schemes (i) , (ii) , and (iii) reduce the overall serial encoding time by 34.71%, 45.27%, and 68.76% with a 2.3%, 3.1%, and 4.5% bitrate increase to maintain the same VMAF, respectively compared to stand-alone encodings. The overall parallel encoding time is reduced by 22.03%, 20.72%, and 76.82% compared to stand-alone encodings for schemes (i) , (ii) , and (iii) , respectively.

Ultra High Definition Television (UHDTV) offers a better immersive audiovisual experience than HDTV by improving the aesthetic sense of the content [1]. How-ever, it may lead to an increase of both encoding time complexity and compression artifacts at lower bitrates. To address this challenge, a low-latency pre-processing algorithm named COntent-aware frame Dropping Algorithm (CODA) is proposed to predict the optimized framerate per video segment in streaming scenarios. The optimized framerate$(\hat{f})$for every video segment at each target bitrate is modelled as an exponential decay (increasing) function whose decay rate is directly proportional to the temporal characteristics$(h)$[2] [3] of the video and the target bitrate$(b)$, and inversely proportional to the spatial characteristics$(E)$of the video. The encoding is carried out with the predicted framerate, saving encoding time and improving visual quality at lower bitrates. At the decoder side, the video is upscaled in the temporal domain to the original framerate$(f_{max})$for display.

Current per-title encoding schemes encode the same video content at various bitrates and spatial resolutions to find an optimized bitrate ladder for each video content in Video on Demand (VoD) applications. However, in live streaming applications, a bitrate ladder with fixed bitrate-resolution pairs is used to avoid the additional latency caused to find optimum bitrate-resolution pairs for every video content. This paper introduces an online per-title encoding scheme (OPTE) for live video streaming applications. In this scheme, each target bitrate’s optimal resolution is predicted from any pre-defined set of resolutions using Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for each video segment. Experimental results show that, on average, OPTE yields bitrate savings of 20.45% and 28.45% to maintain the same PSNR and VMAF, respectively, compared to a fixed bitrate ladder scheme (as adopted in current live streaming deployments) without any noticeable additional latency in streaming.

In two-pass encoding, also known as multi-pass encoding, the input video content is analyzed in the first-pass to help the second-pass encoding utilize better encoding decisions and improve overall compression efficiency. In live streaming applications, a single-pass encoding scheme is mainly used to avoid the additional first-pass encoding run-time to analyze the complexity of every video content. This paper introduces an Efficient low-latency Two-Pass encoding Scheme (ETPS) for live video streaming applications. In this scheme, Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for every video segment are extracted in the first-pass to predict each target bitrate’s optimal constant rate factor (CRF) for the second-pass constrained variable bitrate (cVBR) encoding. Experimental results show that, on average, ETPS compared to a traditional two-pass average bitrate encoding scheme yields encoding time savings of 43.78% without any noticeable drop in compression efficiency. Additionally, compared to a single-pass constant bitrate (CBR) encoding, it yields bitrate savings of 10.89% and 8.60% to maintain the same PSNR and VMAF, respectively.

In live streaming applications, a fixed set of bitrate-resolution pairs (known as bitrate ladder) is used for simplicity and efficiency to avoid the additional encoding run-time required to find optimum resolution-bitrate pairs for every video content. However, an optimized bitrate ladder may result in (i) decreased storage or delivery costs or/and (ii) increased Quality of Experience (QoE). This paper introduces a perceptually-aware per-title encoding (PPTE) scheme for video streaming applications. In this scheme, optimized bitrate-resolution pairs are predicted online based on Just Noticeable Difference (JND) in quality perception to avoid adding perceptually similar representations in the bitrate ladder. To this end, Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for each video segment are used. Experimental results show that, on average, PPTE yields bitrate savings of 16.47% and 27.02% to maintain the same PSNR and VMAF, respectively, compared to the reference HTTP Live Streaming (HLS) bitrate ladder without any noticeable additional latency in streaming accompanied by a 30.69% cumulative decrease in storage space for various representations.

Despite remarkable advances, current Extended Reality (XR) applications are in their majority local and individual experiences. A plethora of interactive applications, such as teleconferencing, tele-surgery, interconnection in new buildings project chain, Cultural Heritage and Museum contents communication, are well on their way to integrate immersive technologies. However, interconnected, and interactive XR, where participants can virtually interact across vast distances, remains a distant dream. In fact, three great barriers stand between current technology and remote immersive interactive life-like experiences, namely the (i) content realism, (ii) motion-to-photon latency, and accurate (iii) human centric quality assessment and control. Overcoming these barriers will require novel solutions at all elements of the end-to-end transmission chain. This workshop focuses on the challenges, applications, and major advancements in multimedia, networks and end-user infrastructures to enable the next generation of interactive XR applications and services. The complete IXR'22 workshop proceedings are available at: https://dl.acm.org/doi/proceedings/10.1145/3552483

The increasing demand for high-quality and low-cost video streaming services calls for the prediction of video encoding complexity. The prior prediction of video encoding complexity including encoding time and bitrate predictions are used to allocate resources and set optimized parameters for video encoding effectively. In this paper, a light-weight video encoding complexity prediction (VECP) scheme that predicts the encoding bitrate and the encoding time of video with high accuracy is proposed. Firstly, low-complexity Discrete Cosine Transform (DCT)-energy-based features, namely spatial complexity, temporal complexity, and brightness of videos are extracted, which can efficiently represent the encoding complexity of videos. The latent vectors are also extracted from a Convolutional Neural Network (CNN) with MobileNet as the backend to obtain additional features from representative frames of each video to assist the prediction process. The extreme gradient boosting (XGBoost) regression algorithm is deployed to predict video encoding complexity using the extracted features. The experimental results demonstrate that VECP predicts the encoding bitrate with an error percentage of up to 3.47% and encoding time with an error percentage of up to 2.89%, but with a significantly low overall latency of 3.5 milliseconds per frame which makes it suitable for both Video on Demand (VoD) and live streaming applications.

This paper introduces an algorithm to improve the security, efficiency, and embedding capacity of reversible data hiding in encrypted images (RDH-EI). It is based on classification encryption difference and adaptive fixed-length coding. Firstly, the prediction error image is obtained, the difference with a bin value greater than the encryption threshold in the difference histogram is found, and it is further modified to obtain the embedding threshold range. Then, under the condition of ensuring that the difference inside and outside the embedding threshold range is not confused, the difference within the threshold is only scrambled, and the difference outside the threshold is scrambled and mod encrypted. After obtaining the encrypted image, an adaptive difference fixed-length coding method is proposed to encode and compress the differences within the threshold. The secret data is embedded in the multiple most significant bits of the encoded difference. Experimental results show that the embedding capacity of the proposed algorithm is improved compared with the state-of-the-art algorithm.

This paper provides an overview of the open Video Complexity Dataset (VCD) which comprises 500 Ultra High Definition (UHD) resolution test video sequences. These sequences are provided at 24 frames per second (fps) and stored online in losslessly encoded 8-bit 4:2:0 format. In this paper, all sequences are characterized by spatial and temporal complexities, rate-distortion complexity, and encoding complexity with the x264 AVC/H.264 and x265 HEVC/H.265 video encoders. The dataset is tailor-made for cutting-edge multimedia applications such as video streaming, two-pass encoding, per-title encoding, scene-cut detection, etc. Evaluations show that the dataset includes diversity in video complexities. Hence, using this dataset is recommended for training and testing video coding applications. All data have been made publicly available as part of the dataset, which can be used for various applications.

For online analysis of the video content complexity in live streaming applications, selecting low-complexity features is critical to ensure low-latency video streaming without disruptions. To this light, for each video (segment), two features, i.e., the average texture energy and the average gradient of the texture energy, are determined. A DCT-based energy function is introduced to determine the block-wise texture of each frame. The spatial and temporal features of the video (segment) are derived from this DCT-based energy function. The Video Complexity Analyzer (VCA) project aims to provide an efficient spatial and temporal complexity analysis of each video (segment) which can be used in various applications to find the optimal encoding decisions. VCA leverages some of the x86 Single Instruction Multiple Data (SIMD) optimizations for Intel CPUs and multi-threading optimizations to achieve increased performance. VCA is an open-source library published under the GNU GPLv3 license.

Many applications and online services produce and deliver multimedia traffic over the Internet. Video streaming services with a rapidly growing desire for more resources to provide better quality, such as Ultra High Definition (UHD) 8K content, are on the list. The HTTP Adaptive Streaming (HAS) technique defines standard baselines for audio-visual content streaming to balance the delivered media quality and minimize defects in streaming sessions. On the other hand, video codecs development and standardization help the progress toward improving such services by introducing efficient algorithms and technologies. Versatile Video Coding (VVC) is one of the latest advancements in video encoding technology that is still not fully optimized and not supported on all available platforms. Mentioned optimization of the video codecs and supporting more platforms require years of research and development. This paper provides multiple test assets in the form of a dataset that facilitates the research and development of the stated technologies. Our open-source dataset comprises Dynamic Adaptive Streaming over HTTP (MPEG-DASH) packaged multimedia assets, encoded with Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), AOMedia Video 1 (AV1), and VVC. We provide our dataset with resolutions of up to 7680x4320 or 8K. Our dataset has a maximum media duration of 322 seconds, and we offer our MPEG-DASH packaged content with two segments lengths, 4 and 8 seconds.

In live streaming applications, a fixed set of bitrate-resolution pairs (known as bitrate ladder) is generally used to avoid additional pre-processing run-time to analyze the complexity of every video content and determine the optimized bitrate ladder. Furthermore, live encoders use the fastest available preset for encoding to ensure the minimum possible latency in streaming. For live encoders, it is expected that the encoding speed is equal to the video framerate. An optimized encoding preset may result in (i) increased Quality of Experience (QoE) and (ii) improved CPU utilization while encoding. In this light, this paper introduces a Content-Adaptive encoder Preset prediction Scheme (CAPS) for adaptive live video streaming applications. In this scheme, the encoder preset is determined using Discrete Cosine Transform (DCT)-energy-based low-complexity spatial and temporal features for every video segment, the number of CPU threads allocated for each encoding instance, and the target encoding speed. Experimental results show that ChPS yields an overall quality improvement of 0.83 dB PSNR and 3.81 VMAF with the same bitrate, compared to the fastest preset encoding of the HTTP Live Streaming (HLS) bitrate ladder using $\times265$ HEVC open-source encoder. This is achieved by maintaining the desired encoding speed and reducing CPU idle time.

We currently witness the rapidly growing importance of intelligent video streaming quality optimization and reduction of video delivery costs. Per-Title encoding, in contrast to a fixed bitrate ladder, shows significant promise to deliver higher quality video streams by addressing the trade-off between compression efficiency and video characteristics such as resolution and frame rate. Selecting encodings with noticeable quality differences in between prevents the construction of an inefficient bitrate ladder that suffers from too similar quality representations. In this respect, the VMAF metric represents a promising foundation for bitrate laddering, as it currently yields the highest video quality prediction performance. However, the minimum noticeable quality difference, referred as to just-noticeable-difference (JND), has not been properly validated for VMAF yet, with existing sources proposing highly diverse ΔVMAF step sizes ranging from two [1] to six [2]. This paper investigates JND for VMAF to facilitate the efficient construction of content-specific bitrate ladders. Using a publicly available JND multimedia dataset, we invalidate existing step size rules of thumb by revealing the large errors they introduce when applied to real-world video content. We investigate different factors that influence the actual ΔVMAF step sizes required to obtain correct JNDs and develop a model using content features that more accurately determines the optimal quality step sizes required for properly laddering individual clips.

Recently, a larger interest in the different plenoptic formats, including digital holograms, has emerged. Aside from other challenges that several steps of the holographic pipeline, from digital acquisition to display, have to face, visual quality assessment of compressed holograms is particularly demanding due to the distinct nature of this 3D image modality when compared to regular 2D imaging. There are few studies on holographic data quality assessment, particularly with respect to perceptual effects of lossy compression. This work aims to study the quality evaluation of digital hologram reconstructions, presented on regular 2D displays, in the presence of compression distortions. As there is no established or generally agreed compression methodology for digital holograms compression on the hologram plane with available implementations, a set of state-of-the-art compression codecs, namely HEVC, AV1, and JPEG2000, were used for compression of the digital holograms on the object plane. Both computer generated and optically generated holograms were considered. Two subjective tests were conducted to evaluate distortions caused by compression. The first subjective test was conducted on the reconstructed amplitude images of central views, while the second test was conducted on pseudo-videos generated from the reconstructed amplitudes of different views. The subjective quality assessment was based on mean opinion scores. A selection of objective quality metrics was evaluated, and their correlations with mean opinion scores were computed. The VIFp metrics appeared to have the highest correlation.

In light field image compression, facilitating random access to individual views plays a significant role in decoding views quickly, reducing memory footprint, and decreasing the bandwidth requirement for transmission. Highly efficient light field image compression methods mainly use inter view prediction. Therefore, they typically do not provide random access to individual views. On the other hand, methods that provide full random access usually reduce compression efficiency. To address this trade-off, a light field image encoding method that favors random access is proposed in this paper. Light field image views are grouped into independent (3× 3) views, which are called Macro View Images (MVIs) . To encode MVIs, the central view is used as a reference to compress its adjacent neighboring views using a hierarchical reference structure. To encode the central view of each MVI, the most central view along with the center of a maximum of three MVIs, are used as reference images for the disparity estimation. In addition, the proposed method allows the use of parallel processing to reduce the maximum encoding/decoding time-complexity in multi-core processors. Tile partitioning can also be used to randomly access different regions of the light field images. The simulation results show that the proposed method outperforms other state-of-the-art methods in terms of compression efficiency while providing random access to both views and regions of interest.

Light field imaging enables some post-processing capabilities like refocusing, changing view perspective, and depth estimation. As light field images are represented by multiple views, they contain a huge amount of data that makes compression inevitable. Although there are some proposals to efficiently compress light field images, their main focus is on encoding efficiency. However, some important functionalities such as viewpoint and quality scalabil- ities, random access, and uniform quality distribution have not been addressed adequately. In this paper, an efficient light field image compression method based on a deep neural network is proposed, which classifies multiple views into various layers. In each layer, the target view is synthesized from the available views of previously encoded/decoded layers using a deep neural network. This synthesized view is then used as a virtual reference for the target view inter-coding. In this way, random access to an arbitrary view is provided. Moreover, uniform quality distribution among multiple views is addressed. In higher bitrates where random access to an arbitrary view is more crucial, the required bitrate to access the requested view is minimized.

Video delivery over the Internet has been becoming a commodity in recent years, owing to the widespread use of Dynamic Adaptive Streaming over HTTP (DASH). The DASH specification defines a hierarchical data model for Media Presentation Descriptions (MPDs) in terms of segments. This paper focuses on segmenting video into multiple shots for encoding in Video on Demand (VoD) HTTP Adaptive Streaming (HAS) applications. Therefore, we propose a novel Discrete Cosine Transform (DCT) feature-based shot detection and successive elimination algorithm for shot detection and compare it against the default shot detection algorithm of the x265 implementation of the High Efficiency Video Coding (HEVC) standard. Our experimental results demonstrate that our proposed feature-based pre-processor has a recall rate of 25% and an F-measure of 20% greater than the benchmark algorithm for shot detection.

Current per-title encoding schemes encode the same video content (or snippets/subsets thereof) at various bitrates and spatial resolutions to find an optimal bitrate ladder for each video content. Compared to traditional approaches, in which a predefined, content-agnostic ("fit-to-all") encoding ladder is applied to all video contents, per-title encoding can result in (i) a significant decrease of storage and delivery costs and (ii) an increase in the Quality of Experience (QoE). In the current per-title encoding schemes, the bitrate ladder is optimized using only spatial resolutions, while we argue that with the emergence of high framerate videos, this principle can be extended to temporal resolutions as well. In this paper, we improve the per-title encoding for each content using spatio-temporal resolutions. Experimental results show that our proposed approach doubles the performance of bitrate saving by considering both temporal and spatial resolutions compared to considering only spatial resolutions.

Recent studies have revealed that network-assisted techniques, by providing a comprehensive view of the network, improve HTTP Adaptive Streaming (HAS) system performance significantly. This paper leverages the capability of Software-Defined Networking, Network Function Virtualization, and edge computing to introduce a CDN-Aware QoE Optimization in SDN-Assisted Adaptive Video Streaming (CSDN) framework. We employ virtualized edge entities to collect various information items and run an optimization model with a new server/segment selection approach in a time-slotted fashion to serve the clients’ requests by selecting optimal cache servers. In case of a cache miss, a client’s request is served by an optimal replacement quality from a cache server, by a quality transcoded from an optimal replacement quality at the edge, or by the originally requested quality from the origin server. Comprehensive experiments conducted on a large-scale testbed demonstrate that CSDN outperforms other approaches in terms of the users’ QoE and network utilization.

High Efficiency Video Coding (HEVC) improves the encoding efficiency by utilizing sophisticated tools such as flexible Coding Tree Units (CTUs) partitioning. The Coding Units (CUs) can be split recursively into four equally sized CUs ranging from 64×64 to 8×8 pixels. At each depth level (or CU size), intra prediction via exhaustive mode search was exploited in HEVC to improve the encoding efficiency and result in a very high encoding time complexity. This paper proposes an Intra CU Depth Prediction (INCEPT) algorithm, which limits Rate-Distortion Optimization (RDO) for each CTU in HEVC by utilizing the spatial correlation with the neighboring CTUs, which is computed using a DCT energy-based feature. Thus, INCEPT reduces the number of candidate CU sizes required to be considered for each CTU in HEVC intra coding. Experimental results show that the INCEPT algorithm achieves a better trade-off between the encoding efficiency and encoding time saving (i.e., BDR/∆T) than the benchmark algorithms. While BDR/∆T is 12.35% and 9.03% for the benchmark algorithms, it is 5.49% for the proposed algorithm. As a result, INCEPT achieves a 23.34% reduction in encoding time on average while incurring only a 1.67% increase in bitrate than the original coding in the x265 HEVC open-source encoder.

MU-MIMO is a high-speed technique in IEEE 802.11ac and upcoming 802.11ax technologies that improves spectral efficiency by allowing concurrent communication between one Access Point and multiple users. In this paper, we present MuVIS, a novel framework that proposes MU-MIMO-aware optimization for multi-user multimedia applications over IEEE 802.11ac/ax. Taking a cross-layer approach, MuVIS first optimizes the MU-MIMO user group selection for the users with the same characteristics in the PHY/MAC layer. It then optimizes the video bitrate for each group accordingly. We present our design and its evaluation on smartphones and laptops over 802.11ac WiFi.

Since video accounts for the majority of today's internet traffic, the popularity of HTTP Adaptive Streaming (HAS) is increasing steadily. In HAS, each video is encoded at multiple bitrates and spatial resolutions (i.e., representations) to adapt to a heterogeneity of network conditions, device characteristics, and end-user preferences. Most of the streaming services utilize cloud-based encoding techniques which enable a fully parallel encoding process to speed up the encoding and consequently to reduce the overall time complexity. State-of-the-art approaches further improve the encoding process by utilizing encoder analysis information from already encoded representation(s) to improve the encoding time complexity of the remaining representations. In this paper, we investigate various multi-encoding algorithms (i.e., multi-rate and multi-resolution) and propose novel multi-encoding algorithms for large-scale HTTP Adaptive Streaming deployments. Experimental results demonstrate that the proposed multi-encoding algorithm optimized for the highest compression efficiency reduces the overall encoding time by 39% with a 1.5% bitrate increase compared to stand-alone encodings. Its optimized version for the highest time savings reduces the overall encoding time by 50% with a 2.6% bitrate increase compared to standalone encodings.

In per-title encoding, to optimize a bitrate ladder over spatial resolution, each video segment is downscaled to a set of spatial resolutions, and they are all encoded at a given set of bitrates. To find the highest quality resolution for each bitrate, the low-resolution encoded videos are upscaled to the original resolution, and a convex hull is formed based on the scaled qualities. Deep learning-based video super-resolution (VSR) approaches show a significant gain over traditional upscaling approaches, and they are becoming more and more efficient over time. This paper improves the per-title encoding over the upscaling methods by using deep neural network-based VSR algorithms. Utilizing a VSR algorithm by improving the quality of low-resolution encodings can improve the convex hull. As a result, it will lead to an improved bitrate ladder. To avoid bandwidth wastage at perceptually lossless bitrates, a maximum threshold for the quality is set, and encodings beyond it are eliminated from the bitrate ladder. Similarly, a minimum threshold is set to avoid low-quality video delivery. The encodings between the maximum and minimum thresholds are selected based on one Just Noticeable Difference. Our experimental results show that the proposed per-title encoding results in a 24% bitrate reduction and 53% storage reduction compared to the state-of-the-art method.

Due to the growing importance of optimizing the quality and efficiency of video streaming delivery, accurate assessment of user-perceived video quality becomes increasingly important. However, due to the wide range of viewing distances encountered in real-world viewing settings, the perceived video quality can vary significantly in everyday viewing situations. In this paper, we investigate and quantify the influence of viewing distance on perceived video quality. A subjective experiment was conducted with full HD sequences at three different fixed viewing distances, with each video sequence being encoded at three different quality levels. Our study results confirm that the viewing distance has a significant influence on the quality assessment. In particular, they show that an increased viewing distance generally leads to increased perceived video quality, especially at low media encoding quality levels. In this context, we also provide an estimation of potential bitrate savings that knowledge of actual viewing distance would enable in practice. Since current objective video quality metrics do not systematically take into account viewing distance, we also analyze and quantify the influence of viewing distance on the correlation between objective and subjective metrics. Our results confirm the need for distance-aware objective metrics when the accurate prediction of perceived video quality in real-world environments is required.

Adaptive HTTP streaming provides multiple representations of the same content at different bit-rates and resolutions and allows the client to request segments from the available representations in a dynamic, adaptive way depending on its context. The growing number of representations in adaptive HTTP streaming makes encoding of one video segment at different representations a challenging task in terms of encoding time-complexity. In this paper, information of both highest and lowest quality representations are used to limit Rate Distortion Optimization (RDO) process for each Coding Unit Tree (CTU) in High Efficiency Video Coding. Our proposed method first encodes the highest quality representation and consequently uses its information to encode the lowest quality representation. Thereafter, information from both the highest and the lowest quality representations are used to predict features of intermediate quality representations. In particular, the block structure and the selected reference frame of both highest and lowest quality representations are used to predict and shorten the RDO process of each CTU for intermediate quality representations. Our proposed method introduces a delay of two CTUs if parallel encoding is used. Experimental results show significant reduction in time-complexity over the reference software (38%) and the state-of-the-art (10%) while quality degradation is negligible.

Recently, there has been a growing interest in the various formats of plenoptic information, including digital holograms. However, the reduced number of studies on holographic data quality assessment results in a difficult analysis of the effects caused by any type of post-processing procedures, like lossy compression. In this work, several state of the art compression codecs, namely HEVC, AV1, and JPEG2000, are applied to the digital holographic data complex representation on the reconstruction plane. Both computer generated and experimental holograms are considered. The perceptual quality of the amplitude is evaluated to measure the quality degradation at four different bit rates. The performance of a selection of quality metrics is also tested.

Recorded cataract surgery videos play a prominent role in training and investigating the surgery, and enhancing the surgical outcomes. Due to storage limitations in hospitals, however, the recorded cataract surgeries are deleted after a short time and this precious source of information cannot be fully utilized. Lowering the quality to reduce the required storage space is not advisable since the degraded visual quality results in the loss of relevant information that limits the usage of these videos. To address this problem, we propose a relevance-based compression technique consisting of two modules: (i) relevance detection, which uses neural networks for semantic segmentation and classification of the videos to detect relevant spatio-temporal information, and (ii) content-adaptive compression, which restricts the amount of distortion applied to the relevant content while allocating less bitrate to irrelevant content. The proposed relevance-based compression framework is implemented considering five scenarios based on the definition of relevant information from the target audience's perspective. Experimental results demonstrate the capability of the proposed approach in relevance detection. We further show that the proposed approach can achieve high compression efficiency by abstracting substantial redundant information while retaining the high quality of the relevant content.

HTTP Adaptive Streaming (HAS) is the most common approach for delivering video content over the Internet. The requirement to encode the same content at different quality levels (i.e., representations) in HAS is a challenging problem for content providers. Fast multirate encoding approaches try to accelerate this process by reusing information from previously encoded representations. In this paper, we propose to use convolutional neural networks (CNNs) to speed up the encoding of multiple representations with a specific focus on parallel encoding. In parallel encoding, the overall time-complexity is limited to the maximum time-complexity of one of the representations that are encoded in parallel. Therefore, instead of reducing the time-complexity for all representations, the highest time-complexities are reduced. Experimental results show that FaME-ML achieves significant time-complexity savings in parallel encoding scenarios (41% in average) with a slight increase in bitrate and quality degradation compared to the HEVC reference software.

Pseudo-sequence based light field compression methods are a highly efficient solution to compress light field images. They use state-of-the-art video encoders like HEVC to encode the image views. HEVC exploits Coding Tree Unit (CTU) structure which is flexible and highly efficient but it is computationally demanding. Each CTU is examined in various depths, prediction and transformation modes to find an optimal coding structure. Efficiently predicting depth of the coding units can reduce complexity significantly. In this paper, a new depth decision method is introduced which exploits the minimum and maximum of previously encoded co-located coding units in spatially closer reference images. Minimum and maximum depths of these co-located CTUs are computed for each coding unit and are used to limit the depth of the current coding unit. Experimental results show up to 55% and 85% encoding time reduction with serial and parallel processing respectively, at negligible degradations.

In light field compression, besides coding efficiency, providing random access to individual views is also a very significant factor. Highly efficient compression methods usually lack random access. Similarly, random access methods usually reduce the compression efficiency. To address this trade-off, a light field image encoding method is proposed in this paper which favors random access. In the proposed scheme 15×15 view images are divided into 25 independent 3×3 view images which are called Macro View Image (MVI). To encode MVIs, the central view image is used to compress its immediate neighboring view images using a hierarchical reference structure. To encode the central view of each MVI, the most central view image, along with the center of at most three MVIs, are used as the reference images for the disparity estimation. In addition, the proposed method enables the use of parallel computation to improve encoding/decoding time complexity. To reduce memory footprint in case a Region of Interest (ROI) is required, HEVC tile partitioning is used.

Light field imaging is a promising technology for 3D computational photography. As Light Field images are represented for multiple views, their subjective evaluation is a very demanding task. Hence, identifying reliable objective quality assessment methodologies plays a very important role. In this paper six objective quality metrics; PSNR-Y, PSNR-YUV, SSIM-Y, MSSSIM-Y, FSIM-Y and HDRVDP2-Y are assessed for five state-of-the-art codecs at various bit-rates. Moreover, the metrics are computed in the linear, perceptually uniform and perceptual quantizer spaces. The results are compared against those of a subjective study and is concluded that the average FSIM-Y is the most reliable metric. The paper also introduces maps of the objective metrics to evaluate the quality dispersion among the different light field image views.

Light fields capture a large number of samples of light rays in both intensity and direction terms, which allow post-processing applications such as refocusing, shifting view-point and depth estimation. However, they are represented by huge amount of data and require a high-efficient coding scheme for its compression. In this paper, light field raw image data is decomposed into multi-views and used as a pseudo-sequence input for state-of-the-art codecs such as High Efficiency Video Coding (HEVC). In order to better exploit redundancy between neighboring views and decrease distances between current view and its references instead of using conventional orders, views are divided into four smaller regions and each region is scanned by a snake order. Furthermore, according to this ordering, an appropriate referencing structure is defined that only selects adjacent views as references. Simulation results show that Rate-Distortion performance of proposed method has higher gain than the other state-of-the-art light field compression methods.