Distributed storage systems, such as Hadoop distributed file system (HDFS), are widely used for video storage due to their outstanding scalability. However, they frequently face challenges related to data unavailability arising from issues like network disconnection, server downtimes, and storage failures. This necessitates file replication, leading to significant storage requirements. To address this, we propose a novel deep reinforcement learning (DRL) algorithm that relies on the tradeoff between video quality and storage demands. We first formulate an optimization problem with the objective of maximizing video quality while constraining storage requirements necessary for replication. The video storage system is then modeled with time-varying video streaming workloads as the DRL environment, where the agent determines the placement of replica files without foreknowledge of future storage availability and video popularity. To address this uncertainty, we use a deep double-Q network (D3QN), which includes an action space that finds the number of replicas for each file, an observation space featuring storage utilization and file placement, and a reward model calculating the expected video quality under various data unavailability probabilities. The implementation of our method is examined within the HDFS. Experimental results show that our method improves video quality by up to 39% compared to benchmarks, achieves quality comparable to Oracle when all bitrates are accessible, and even surpasses HDFS’s triple redundancy method while using only 20% of the storage space.

With the advent of new large-capacity solid-state disks (SSDs) such as quad-level-cells (QLC), SSD arrays can be effectively used in video storage systems that require large-capacity storage space. Typically, SSD manufacturers specify a drive-writes-per-day (DWPD) metric, which is the ratio of bytes written per day to the total capacity in bytes, to ensure an SSD’s specified lifetime; it is important to limit the number of write operations by considering the DWPD for each SSD. We propose a new video file allocation technique to effectively manage the heterogeneous DWPD characteristics of SSDs in distributed storage systems. To express the degree of wear-leveling for heterogeneous SSDs, we first introduce the concept of ADWD, which is the actual number of bytes written per day compared to DWPD. We then propose two algorithms for file placement and migration. The file placement algorithm places files greedily based on the bandwidth-to-space ratio (BSR) of each file and SSD to balance the bandwidth usage and storage of the SSD. The file migration algorithm moves files from overloaded to underloaded SSDs to meet bandwidth limit requirements while minimizing the overall ADWD as a result of migration, and then migrates additional popular files to improve SSD bandwidth utilization. To use these algorithms in actual distributed file systems, we implemented a suite of tools for file placement and migration in the Hadoop distributed file system (HDFS). Experimental results show that the proposed algorithm reduces the mean of ADWD by 35.44% and its standard deviation by 69.78% compared to the benchmark methods on average.

Live-streaming video requires a lot of CPU-intensive transcoding so that viewers can receive video at bitrates appropriate to their devices and network conditions, which is necessary for a good quality of experience (QoE). We allocate transcoding tasks to edge-servers in a multiple-access edge-computing (MEC) architecture, taking into account server capacity, wireless network coverage, and the cost budget of broadcasters, as well as QoE. Our algorithm first chooses candidate transcoding tasks by giving higher priority to the tasks that make the most cost-effective contribution to popularity-weighted video quality (PWQ). It assigns these tasks to edge-servers in a greedy manner, taking network coverage and computational load into account. Subsequently, it meets a cost budget by reassigning some tasks and removing other assignments altogether, while trying to minimize the effect of these alterations on total PWQ. Simulation results show that our scheme achieves 0.06% to 94.62% (average 25.3%) more PWQ than alternative schemes under the same cost budget.

Dynamically adaptive streaming over HTTP requires a large-scale server to transcode various bitrate versions in which different preset parameters can be used to provide different video qualities at each resolution. When transcoding servers contain a heterogeneous mix of CPUs and GPUs, the task scheduler must choose a processor and preset parameter for each transcoding task to meet the transcoding deadlines while achieving the best possible video quality. We apply regression analysis to sample variable-bit-rate videos to provide accurate (mean absolute percentage error values from 1.3% to 13.9%) model for predicting bitrate, transcoding time and video quality at each resolution on different processors. We build this into a greedy allocation and scheduling algorithm which first satisfies deadlines with low video quality, and then redistributes the workload to improve that quality while continuing to meet the deadlines. This scheme was both simulated and implemented on a testbed server. It satisfies all deadlines while outperforming standard algorithms by between 3.12% and 15.59% in terms of popularity-weighted video quality divided by bitrate.

In multiaccess edge computing (MEC), tasks are offloaded from mobile devices to servers at the edge of the network. This speeds up task processing without incurring the latency required to reach central servers. However, the power used by edge servers is significant and needs to be cost-effective. We propose a scheme in which tasks are offloaded to servers with the aim of maximizing a reward within a limited power budget, server processing capacities, and wireless network coverage. Our algorithm determines the maximum utilization of each server while favoring the offloading of tasks with high ratios of reward to power requirement. We model the task allocation problem using a minimum-cost-maximum-flow graph, and propose two edge allocation algorithms, one of which is extended to allow task splitting, which offload tasks subject to server capacity by searching for the highest reward. In simulations, our scheme achieved between 8% and 80% higher rewards than alternative schemes, under the same power constraints.

Transcoding is essential for live video streaming systems, but video streams in many channels may not be transcoded due to the processing capacity limitations of transcoding servers with multiple CPU nodes. To address this, we propose an algorithm that determines the transcodeable bitrate versions for each channel and allocates the transcoding tasks, by taking into account video quality, popularity, and workload balancing. Simulation results show that it can improve popularity-weighted video quality by up to 10.7% over other popularity-based alternatives.

Dynamically adaptive streaming over HTTP (DASH), the most common streaming technique, requires a video server to store all the transcoded versions, resulting in a lot of storage space, thereby consuming a significant disk power. A disk array can be divided into hot and cold zones to allow cold disks to be spun down, but this poses several questions such as (1) which video segments can be stored on the hot disks, (2) how to allocate video segments among the hot disks, and (3) how to handle requests to the cold disks. To address this, we propose three new algorithms; (1) a hot data classification algorithm to determine which segments should be stored on the hot disks, by taking segment popularity and quality-of-experience (QoE) into account, (2) a video segment allocation algorithm to balance workloads among the hot disks, and (3) a disk bandwidth allocation algorithm which determines the bit-rate of each segment with the aim of maximizing overall QoE. Experimental results show that our scheme can reduce the power consumption between 29% and 46% compared with the method of storing all the transcoded versions at the cost of 1.5% QoE degradation.

Dynamic adaptive streaming over HTTP (DASH) requires a video server to transcode each original video object to all the possible bit-rate versions, resulting in high CPU power consumption. To address this, we propose a new scheme that balances quality-of-experience (QoE) against transcoding energy. We start by introducing the concept of transcoding gain to express QoE achieved as a result of transcoding operation, by taking version popularity and QoE into account. We then propose two algorithms that determine which versions should be transcoded with the aim of maximizing the overall QoE. Experimental results show that the proposed scheme can effectively limit the amount of transcoding energy consumption.

Blocking artifacts, commonly introduced during video encoding, are one of the major causes of reduced perceptual video quality. The trade-off between these artifacts and bitrate can be improved by adaptively selecting frames from a set of video copies encoded at different bitrates, prior to actual video encoding. We propose a new direction of constructing mixed bitrate video based on content-based image analysis on each video frame, which was posed as a problem of pre-analysis for the final video encoding step. The proposed method consists of the following steps: First, we define a simple and fast impact metric in order to identify the blocking artifacts in each frame of multiple videos, encoded at different bitrates. Based on the impact metric, we generate a blocking artifact density functions for the available bitrates, on the whole video. Finally, we define and optimize our objective function from the blocking artifact density functions in order to select a bitrate with minimum perceptual blockiness and file size for each frame. We validated our method throughout multiple types of videos, showing improved visual quality for the same file size based on commonly used quality assessment measures, such as MSU blocking, MSU blurring, SSIM, 3SSIM, and stSSIM. The reduction rates of average file size and average blocking artifact were about 4.9% and 8.3% over maximum bitrate encoding, respectively.

Recent popular streaming services such as TV Everywhere, N-Screen, and dynamic adaptive streaming over HTTP (DASH) need to deliver content to the wide range of devices, requiring video content to be transcoded into different versions. Transcoding tasks require a lot of computation, and each task typically has its own real-time constraint. These make it difficult to manage transcoding, but the more efficient use of energy in servers is an imperative. We characterize transcoding workloads in terms of deadlines and computation times, and propose a new dynamic voltage and frequency scaling (DVFS) scheme that allocates a frequency and a workload to each CPU with the aim of minimizing power consumption while meeting all transcoding deadlines. This scheme has been simulated, and also implemented in a Linux transcoding server, in which a frontend node distributes transcoding requests to heterogeneous backend nodes. This required a new protocol for communication between nodes, a DVFS management scheme to reduce power consumption and thread management and scheduling schemes which ensure that transcoding deadlines are met. Power measurements show that this approach can reduce system-wide energy consumption by 17% to 31%, compared with the Linux Ondemand governor.

To support the diverse requirements of mobile clients and fluctuating network bandwidth, video content for wireless streaming is usually transcoded into several versions. However, transcoding requires a lot of computation, resulting in high power consumption by the server. We examine the characteristics of transcoding workloads and characterize them in terms of deadline and computation time. We then propose a new dynamic voltage and frequency scaling (DVFS) scheme that allocates a frequency and a workload to each CPU with the aim of minimizing power consumption while meeting all transcoding deadlines. Experimental results show that our scheme can reduce system-wide energy consumption by 10% to 16% compared with the Linux On-demand governor while ensuring all the transcoding tasks meet their deadlines.

Maintenance and upgrades to the significant storage infrastructure in a video server often create a heterogenous disk array. We show how to manage the energy consumption of such an array by combining caching and prefetching techniques. We first examine how seek operations affect disk energy consumption, and then analyze the relationship between the amount of prefetched data and the number of seeks, and the effect of the size of the prefetching buffer on energy consumption. Based on this, we propose a new data prefetching scheme in which the amount of data prefetched for each video stream is dynamically adjusted to allow for the bit-rates of streams and the power characteristics of different disks. We next examine the impact of caching on disk power consumption and propose a new caching scheme that prioritizes each stream based on the ratio of the amount of energy that can be saved to its cache requirement, so as to make effective use of limited caching space. We address the trade-off between caching and prefetching and propose an algorithm that dynamically divides the entire buffer space into prefetching and caching regions, with the aim of minimizing overall disk energy consumption. Experimental results show that our scheme can reduce disk energy consumption between 26% and 31%, compared to a server without prefetching and caching.

As the storage requirements of a video server increase, disks of different types are often added. This makes it essential to manage the energy requirements of heterogeneous disk arrays. We examine how seek operations affect disk energy consumption, and then analyze the relationship between the amount of prefetched data and the number of seeks, and the effect of the size of the prefetching buffer on energy consumption. Based on this, we propose a new data prefetching scheme in which the amount of data prefetched for each video stream is dynamically adjusted to allow for the bit-rates of streams and the power characteristics of different disks, with the aim of minimizing overall disk energy consumption. Simulations show that our scheme can reduce disk energy by 26%, compared to a server in which prefetching is not used.