Low latency live streaming (LLLS) leverages chunked transfer encoding (CTE) to substantially reduce end-to-end latency. However, this paradigm introduces a cascade of challenges for adaptive bitrate (ABR) algorithms: (1) the sending idle periods between chunks in CTE render bandwidth measurement difficult and prone to error; (2) bandwidth prediction in LLLS is an irregular time series forecasting with uncertain future segment size, leading to a circular prediction dependency; (3) stochastic uncertainty within LLLS, such as fluctuating idle time, leads to imprecise buffer evolution and ABR degradation. In this paper, we tackle the issues and present AAR, a novel LLLS framework that comprises 3 key modules: (1) accurate bandwidth measurement that leverages a server-side Flag to identify burst transmission and isolate chunks. We further propose to fuse our two learning and heuristic-based algorithms via confidence estimation; (2) bandwidth prediction via conditional normalizing flow to simultaneously learn joint variable distributions. We further propose a bitrate-aware transformer to capture the intrinsic circular relationships as backbone flow condition; (3) an LLLS tailored ABR with a novel and robust objective to maximize the minimum Quality of Experience (QoE) under uncertainty. We propose two theorems to derive the min solution via download time bounds, and we maximize the QoE via Model Predictive Controller (MPC) with LLLS tailored state evolution. Extensive experiments on real-world network traces demonstrate that AAR significantly outperforms baselines with absolute error reduction by 11%-83% for measurement and up to 17% for prediction. We also improve QoE by up to 102% across all tested network conditions.

Mobile Personal Livestreaming (MPL), one of the most popular mobile applications recently, has been experiencing explosive growth thanks to ubiquitous smartphones and high-speed wireless connections nowadays, e.g., 4G-LTE/5G and WiFi 802.11ac/ad. Unlike traditional livestreaming services that are either limited to specific content providers or fixed content, MPL enables end-to-end live-stream communication of anything, at anytime and anywhere, with anybody in mobile networks. This paper presents our design and experience of Inke, a large-scale MPL system with 15 million daily active users since 2015, which provides a range of functionalities, including mobile livestreaming, real time communication, augmented reality (AR), and real time face recognition. We characterize its system design from three key aspects: push-streaming protocol, “Join Channel“ livestreaming, and AR features. For the push-streaming protocol, we introduce the bandwidth-aware design that incorporates buffer length control and frame-drop strategy to ensure upload consistency and reduce delay. We also design a mixed-stream upload model that is able to balance the delay and bandwidth consumption during “Join Channel” livestreaming. In addition, we show how the virtual gift value distribution can be used to im-prove the design of animation playback queues. We demonstrate the effectiveness of our system design that works across time, geo-location, and heterogeneous wireless networks for large-scale (more than 130 million registered users) applications. It achieves 126.5736ms average upload delay, 1.093 average rebuffering times during each broadcast at peak time.

Federated Domain Generalization is aimed at learning a global model that generalizes well to new clients with domain shift in a privacy-preserving manner. Inspired by the powerful generation capability of Large language models (LLMs), we propose a novel method for this problem, named Text-guided Federated Domain Generalization (FedTG), which utilizes text generated by LLMs to guide the training of visual models. In the client, we align the image embeddings to the text embeddings, which provides strong generalization knowledge. In the server, we generate abundant textual descriptions about different classes, subsequently converting them into embeddings for fine-tuning the global classifier, which helps the global classifier not overfit to a particular domain. Moreover, we utilize the powerful image generation capabilities of LLMs to generate a cross-domain validation dataset to help select the most generalized model. We conduct experiments on several datasets, and the results prove that FedTG outperforms relevant baselines.

In a vehicular network, multiple vehicle-to-vehicle (V2V) links could support efficient and reliable V2V communications among vehicles by reusing the frequency spectrum preoccupied by vehicle-to-infrastructure (V2I) links. However, it is difficult and impractical to collect accurate instantaneous channel state information (CSI) in a fast-changing network environment under centralized resource management. To address these complex circumstances involving highly dynamic channel fading and variations of interference, we propose Multiple V2V communications on Twin-Delayed DDPG (MV2V-TD3) for allocating spectrum resources in a decentralized manner. Each V2V link or each vehicle acts as an agent to choose V2I sub-band and transmission power with partial CSI of the vehicular network. The simulation results shows MV2V-TD3 has rapid adaptability to the dynamic vehicular environment and better training performance than other Multi-agent Reinforcement Learning (MARL) methods.The experiments are simulated with the setting of four V2V agents on a map featuring four-lane roads covering nine block streets. Extensive experimental results indicate that V2V agents could effectively and quickly learn to cooperate with each other. Compared to other baseline algorithms, MV2V-TD3 achieves a higher success ratio (SR) for payload delivery along with the improved transmission rate of V2V communications.

Large language models (LLMs) have significantly influenced advancements in sequential recommendation. Nevertheless, the integration and alignment of LLMs with sequence recommenders is often underexploited in current research. Existing LLM-based sequential recommenders mostly rely on textual descriptions, neglecting user visual preferences and suffering from LLM hallucination, which can result in suboptimal recommendations. To address these challenges, we propose AlmostRec, a novel framework that incorporates multimodal information, including historical interaction IDs, textual descriptions, and images of items into large foundation models, to facilitate controllable predictions. Instead of employing a textual LLM, AlmostRec utilizes a large multimodal model (LMM) as a backbone, complemented by a content-behavior guidance module to align multimodal information. The framework's ID prediction objective, enhanced via the parameter-efficient LoRA approach, ensures a principal alignment with the sequential recommendation and is not swayed by hallucinations. AlmostRec effectively bridges the gap between large vision-language models with sequential recommenders, offering contextually relevant predictions in multimodal scenarios. Experimental results on real-world datasets demonstrate the superior performance of AlmostRec compared to both traditional and recent LLM-based recommendation approaches.

Federated Learning (FL) has become a powerful technique for collaborative model training across decentralized entities while preserving data privacy. Despite its potential, FL faces significant challenges, including communication overhead, resource heterogeneity, and data heterogeneity. Existing solutions fall short in addressing disparities in client resources and the errors introduced by direct model aggregation across heterogeneous clients. To tackle these issues, we propose DynFed, a novel federated learning framework that incorporates dynamic quantization bit-width allocation and multi-teacher knowledge distillation for model aggregation. DynFed dynamically adjusts quantization bit-widths to clients based on their resource heterogeneity, adapting these allocations according to variations in the local loss function during training. This adaptive quantization strategy optimizes resource utilization while preserving model performance. For model aggregation, DynFed utilizes a dynamic multi-teacher knowledge distillation approach, assigning the most suitable teacher model to each data sample based on a comprehensive evaluation score, thereby ensuring effective knowledge transfer even in the presence of quantization-induced errors. This method not only mitigates the negative effects of heterogeneous bit-widths but also leverages client model diversity to enhance the robustness of the global model. Extensive experimental results demonstrate the superiority of DynFed over state-of-the-art methods.

Over the past decade, adaptive video streaming technology has witnessed significant advancements, particularly driven by the rapid evolution of deep learning techniques. However, the black-box nature of deep learning algorithms presents challenges for developers in understanding decision-making processes and optimizing for specific application scenarios. Although existing research has enhanced algorithm interpretability through decision tree conversion, interpretability does not directly equate to developers' subjective comprehensibility. To address this challenge, we introduce ComTree, the first bitrate adaptation algorithm generation framework that considers comprehensibility. The framework initially generates the complete set of decision trees that meet performance requirements, then leverages large language models to evaluate these trees for developer comprehensibility, ultimately selecting solutions that best facilitate human understanding and enhancement. Experimental results demonstrate that ComTree significantly improves comprehensibility while maintaining competitive performance, showing potential for further advancement. The source code and appendix are available at https://github.com/thu-media/ComTree.

Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce \texttt{Crucible}, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate \texttt{Crucible}'s effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that \texttt{Crucible} systematically quantifies the tunable space across different algorithms. Furthermore, \texttt{Crucible} provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.

Traditional optimization methods based on system-wide Quality of Service (QoS) metrics have approached their performance limitations in modern large-scale streaming systems. However, aligning user-level Quality of Experience (QoE) with algorithmic optimization objectives remains an unresolved challenge. Therefore, we propose LingXi, the first large-scale deployed system for personalized adaptive video streaming based on user-level experience. LingXi dynamically optimizes the objectives of adaptive video streaming algorithms by analyzing user engagement. Utilizing exit rate as a key metric, we investigate the correlation between QoS indicators and exit rates based on production environment logs, subsequently developing a personalized exit rate predictor. Through Monte Carlo sampling and online Bayesian optimization, we iteratively determine optimal parameters. Large-scale A/B testing utilizing 8% of traffic on Kuaishou, one of the largest short video platforms, demonstrates LingXi's superior performance. LingXi achieves a 0.15% increase in total viewing time, a 0.1% improvement in bitrate, and a 1.3% reduction in stall time across all users, with particularly significant improvements for low-bandwidth users who experience a 15% reduction in stall time.

Sequential recommendation aims to recommend the next items that a target user may have interest in based on the user’s sequence of past behaviors, which has become a hot research topic in both academia and industry. In the literature, sequential recommendation adopts a Sequence-to-Item or Sequence-to-Sequence training strategy, which supervises a sequential model with a user’s next one or more behaviors as the labels and the sequence of the past behaviors as the input. However, existing powerful sequential recommendation approaches employ more and more complex deep structures such as Transformer in order to accurately capture the sequential patterns, which heavily rely on hand-crafted designs on key attention mechanism to achieve state-of-the-art performance, thus failing to automatically obtain the optimal design of attention representation architectures in various scenarios with different data. Other works on classic automated deep recommender systems only focus on traditional settings, ignoring the problem of sequential scenarios. In this article, we study the problem of automated sequential recommendation, which faces two main challenges: (1) How can we design a proper search space tailored for attention automation in sequential recommendation, and (2) How can we accurately search effective attention representation architectures considering multiple user interests reflected in the sequential behavior. To tackle these challenges, we propose an automated disentangled sequential recommendation (AutoDisenSeq) model. In particular, we employ neural architecture search (NAS) and design a search space tailored for automated attention representation in attentive intention-disentangled sequential recommendation with an expressive and efficient space complexity of \(O(n^{2})\) given \(n\) as the number of layers. We further propose a context-aware parameter sharing mechanism taking characteristics of each sub-architecture into account to enable accurate architecture performance estimations and great flexibility for disentanglement of latent intention representation. Moreover, we propose AutoDisenSeq-large language model (LLM), which utilizes the textual understanding power of LLM as a guidance to refine the candidate list for recommendation from AutoDisenSeq. We conduct extensive experiments to show that our proposed AutoDisenSeq model and AutoDisenSeq-LLM model outperform existing baseline methods on four real-world datasets in both overall recommendation and cold-start recommendation scenarios.

Domain Generalization (DG) in the setting of federated learning (i.e. Federated Domain Generalization, FDG) is gaining increasing attention. FDG aims to learn a global model generalizing well to clients with unseen domains in a privacy-preserving manner. Existing works mainly focus on how to learn invariant features or transmit data distributions in a privacy-preserving manner, the correctness of aggregation weights has not been extensively discussed. In this paper, we discuss the rationality of aggregation weights and propose a novel method in FDG, namely FedAWA. We replace traditional weighted aggregation based on data volume with arithmetic mean. Then we propose confidence adjustment to adjust aggregation weights to ensure the rationality of aggregation weights due to the uncertainty of local model training. Moreover, we propose fairness adjustment to adjust local training epochs to prevent local models from contributing excessively or insufficiently to the global model. Our method can serve as a plug-and-play plugina. Experiments on several benchmarks have shown the effectiveness of our method, our method enhances the generalization improvement of federated learning algorithms.

Adaptive video streaming has been studied for over 10 years and has demonstrated remarkable performance. However, adaptive video streaming is not an independent algorithm but relies on other components of the video system. Consequently, as other components undergo optimization, the gap between the traditional simulator and the real-world system continues to grow which makes the adaptive video streaming algorithm must adapt to these variations.In order to address the challenges facing industrial adaptive video streaming, we introduce a novel offline reinforcement learning framework called Backwave. This framework leverages history logs to reduce the sim-real gap. We propose new metrics based on counterfactual reasoning to evaluate its performance and we integrate expert knowledge to generate valuable data to mitigate the issue of data override. Furthermore, we employ curriculum learning to minimize additional errors.We deployed Backwave on a mainstream commercial short video platform, Kuaishou. In a series of A/B tests conducted nearly one month with over 400M daily watch times, Backwave consistently outperforms prior algorithms. Specifically, Backwave reduces stall time by 0.45% to 8.52% while maintaining comparable video quality and Backwave demonstrates improvements in average play duration by 0.12% to 0.16%, and overall play duration by 0.12% to 0.26%.

Congestion control has been a fundamental research focus in web transmission for over 30 years. However, with diverse network scenarios like cellular networks and WiFi, traditional models might no longer accurately describe current network conditions -- we empirically observe that the minimum round-trip time (RTTmin) still varies under different network conditions, challenging the assumption of its constancy in traditional models. In this paper, we model it as a normal distribution based on our measurements and propose a novel congestion control algorithm LingBo. LingBo consists of two phases: an offline trained decision model to achieve goals under different RTTmin distributions, and an online perception scheme to detect the current RTTmin distribution. We evaluate LingBo in various network environments and find it consistently performs well in terms of power metric and throughput compared to recent state-of-the-art baselines. Our code is available at https://github.com/thumedia/LingBo.

Video analytics play a pivotal role in enhancing the safety of intelligent surveillance and autonomous driving. However, the transmission of vast video data and the computational demands of video analytics present challenges within traditional cloud computing paradigms. To address latency concerns, dynamic video analytics often leverage edge deployments. Nevertheless, the efficient allocation of resources at the edge, balancing cost-effectiveness and accuracy, becomes crucial, especially when multiple video analytics services concurrently operate within the system. This paper introduces an edge-centric incentive mechanism designed to encourage greater participation from edge nodes in offloading tasks. The key focus is on addressing the dynamic nature of edge resources and optimizing system returns through a rational pricing mechanism. We propose a decentralized Soft Actor-Critic algorithm grounded in game theory (DSACG) to autonomously learn the optimal pricing strategy. A comprehensive theoretical analysis, supported by extensive simulations, substantiates the effectiveness of our proposed solution.

Streamers are the core competency of the crowd-sourced live streaming (CLS) platform. However, little work has explored how different factors relate to their popularity evolution patterns. In this paper, we will investigate a critical problem, i.e., how to discover the promising streamers in their early stage? To tackle this problem, we first conduct large-scale measurement on a real-world CLS dataset. We find that streamers can indeed be clustered into two evolution types (i.e., rising type and normal type), and these two types of streamers will show differences in some inherent properties. Traditional time-sequential models cannot handle this problem, because they are unable to capture the complicated interactivity and extensive heterogeneity in CLS scenarios. To address their shortcomings, we further propose Niffler, a novel heterogeneous attention temporal graph framework (HATG) for predicting the evolution types of CLS streamers. Specifically, through the graph neural network (GNN) and gated-recurrent-unit (GRU) structure, Niffler can capture both the interactive features and the evolutionary dynamics. Moreover, by integrating the attention mechanism in the model design, Niffler can intelligently preserve the heterogeneity when learning different levels of node representations. We systematically compare Niffler against multiple baselines from different categories, and the experimental results show that our proposed model can achieve the best prediction performance.

The low-light environment is an integral surrounding in real-world video analytic applications. Conventional wisdom claims that in order to adapt to the extensive computation requirement of the analytics model and achieve high inference accuracy, the overall pipeline should leverage a client-to-cloud framework that designs a cloud-based inference with on-demand video streaming. However, we show that due to the amplified noise, directly streaming the video in low-light scenarios can introduce significant bandwidth inefficiency.In this paper, we propose Owl, an intelligent framework to optimize the bandwidth utilization and inference accuracy for the low-light video analytic pipeline. The core idea of Owl is two-fold: on the one hand, we will deploy a light-weighted pre-processing module before transmission, through which we will get the denoised video and significantly reduce the transmitted data; on the other hand, we recover the information from the denoised video via an enhancement module in the server-side. Specifically, through well-designed training mechanism and content representation technique, Owl can dynamically select the best configuration for time-varying videos. Experiments with a variety of datasets and tasks show that Owl achieves significant bandwidth benefits, while consistently optimizing the inference accuracy.

Quality of Experience (QoE)-driven adaptive bitrate (ABR) algorithms are typically optimized using QoE models that are based on the mean opinion score (MOS), while such principles may not account for user heterogeneity on rating scales, resulting in unexpected behaviors. In this paper, we propose Jade, which leverages reinforcement learning with human feedback(RLHF) technologies to better align the users' opinion scores. Jade's rank-based QoE model considers relative values of user ratings to interpret the subjective perception of video sessions. We implement linear-based and Deep Neural Network (DNN)-based architectures for satisfying both accuracy and generalization ability. We further propose entropy-aware reinforced mechanisms for training policies with the integration of the proposed QoE models. Experimental results demonstrate that Jade performs favorably on conventional metrics, such as quality and stall ratio, and improves QoE by 8.09%-38.13% in different network conditions, emphasizing the importance of user heterogeneity in QoE modeling and the potential of combining linear-based and DNN-based models for performance improvement.

The delay to obtain analysis results is an important metric in video analytics. Previous work has focused on reducing frame transmission and inference delay to optimize total delay. However, network fluctuations can cause frames to arrive at the backend simultaneously, leading to backend queuing delays. To address this issue, we propose Concerto, a joint front-and backend video analytics pipeline that optimizes both network transmission and backend queuing delays. The backend controls the frame queue, accelerating or skipping inference as needed to mitigate backend queuing delay. The frontend considers both delays when configuring frames to send, resulting in better total delay. Experiments show that Concerto significantly reduces backend queuing delay with minimal loss of accuracy.

Sequential recommender systems often suffer from performance drops due to the data-sparsity issue in real-world scenarios. To address this issue, we bravely take advantage of the strength in diffusion model to conduct data augmentation for sequential recommendation in this paper. However, there remain two critical challenges for this scarcely-explored topic: (i) previous diffusion models are mostly designed for image generation aiming to capture pixel patterns, which can hardly be applied in data augmentation for sequential recommendation aiming to capture the user-item relations; (ii) given a specific diffusion model capable of user-item interaction augmentation, it is non-trivial to guarantee that the diffusion-generated data can always bring benefits towards the sequential recommendation model. To tackle these challenges, we propose Diff4Rec, a curriculum-scheduled diffusion augmentation framework for sequential recommendation. Specifically, a diffusion model is pre-trained on recommendation data via corrupting and reconstructing the user-item interactions in the latent space, and the generated predictions are leveraged to produce diversified augmentations for the sparse user-item interactions. Subsequently, a curriculum scheduling strategy is designed to progressively feed the diffusion-generated samples into the sequential recommenders, with respect to two levels, i.e., interaction augmentation and objective augmentation, to jointly optimize the data and model. Extensive experiments demonstrate that our proposed Diff4Rec framework is able to effectively achieve superior performance over several strong baselines, capable of making high-quality and robust sequential recommendations. We believe the proposed Diff4Rec has the promising potential to bring paradigm shift in multimedia recommendation.

Even though conventional wisdom claims that in order to improve viewer engagement, the cloud-edge providers should serve the viewers with the nearest edge nodes, however, we show that doing this for crowdsourced live streaming (CLS) services can introduce significant costs inefficiency. In this paper, we first carry out large-scale measurement analysis by using the real-world service data from Huawei Cloud, a representative cloud-edge provider in China. We observe that the massive number of channels has proposed great burdens to the operating expenditure of the cloud-edge providers, and most importantly, unbalanced viewer distribution makes the edge nodes suffer significant costs inefficiency. To tackle the above concerns, we proposeAggCast, a novel CLS scheduling framework to optimize the edge node utilization for the cloud-edge provider. The core idea ofAggCastis to aggregate some viewers that are initially scattered on different regions, and assign them to fewer pre-selected nodes, thereby reducing bandwidth costs. In particular, by integrating the useful insights obtained from our large-scale measurement,AggCastcan not only ensure that quality of experience (QoS) does not suffer degradation, but also satisfy the systematic requirements of CLS services.AggCasthas been A/B tested and fully deployed. The online and trace-driven experiments show that, compared to the most prevalent method,AggCastsaves over 16.3%back-to-source(BTS) bandwidth costs while significantly improving QoS (startup latency, stall frequency and stall time are reduced over 12.3%, 4.57% and 3.91%, respectively).

Congestion control (CC) algorithms based on deep reinforcement learning (DRL) have shown their great potential to adapt themselves to a variety of network conditions. However, as the real-world network conditions are diverse and the optimization goals for CC are made up of some contradicted metrics, it is hard to balance these contradicted absolute valves, which makes it difficult to faithfully reflect the algorithm performance if only considering transient status as the reward. In this work, we propose a novel DRL-based CC approach ZiXia, which considers the adjusted ranking reward, the long-term relative performance reward adjusted by the transient reward. In detail, we design a virtual algorithm arena including the DRL-agents and other classic algorithms as competitors in the same environment. After these algorithms end, we rank their delay and throughput respectively and combine the two relative rankings as the ranking reward using the special preference. The ranking reward gives us a more flexible and interpretable long-term evaluation method compared with the absolute value of the transient status, which gives us a more intuitive perspective to support multi-objection. To get more fine-grained action rewards, we adjust the ranking reward using a linear combination of transient status. Through various experiments, in simulated environments, ZiXia achieves the highest throughput and reduces 87% delay compared with BBR, and in global real-world environments, ZiXia improves 13% throughput and reduces 3% delay than BBR.

Short video uploading service has become increasingly important, as at least 30 million videos are uploaded per day. However, we find that existing congestion control (CC) algorithms, either heuristics or learning-based, are not applicable for video uploading -- i.e., lacking in the design of the fundamental mechanism and being short of leveraging network modeling. We present DuGu, a novel learning-based CC algorithm designed by considering the unique proprieties of video uploading via the probing phase and internet networking via the control phase. During the probing phase, DuGu leverages the transmission gap of uploading short videos to actively detect the network metrics to better understand network dynamics. DuGu uses a neural network~(NN) to avoid congestion during the control phase. Here, instead of using handcrafted reward functions, the NN is learned by imitating the expert policy given by the optimal solver, improving both performance and learning efficiency. To build this system, we construct an omniscient-like network emulator, implement an optimal solver and collect a large corpus of real-world network traces to learn expert strategies. Trace-driven and real-world A/B tests reveal that DuGu supports multi-objective and rivals or outperforms existing CC algorithms across all considered scenarios.

Conventional wisdom claims that in order to improve viewer engagement, the cloud-edge providers should serve the viewers with the nearest edge nodes, however, we show that doing this for crowdsourced live streaming (CLS) services can introduce significant costs inefficiency. We observe that the massive number of channels has greatly burdened the operating expenditure of the cloud-edge providers, and most importantly, unbalanced viewer distribution makes the edge nodes suffer significant costs inefficiency. To tackle the above concerns, we propose AggCast, a novel CLS scheduling framework to optimize the edge node utilization for the cloud-edge provider. The core idea of AggCast is to aggregate some viewers who are initially scattered on different regions, and assign them to fewer pre-selected nodes, thereby reducing bandwidth costs. In particular, by leveraging the insights obtained from our large-scale measurement, AggCast can not only ensure quality of experience (QoS), but also satisfy the systematic requirements of CLS services. AggCast has been A/B tested and fully deployed in a top cloud-edge provider in China for over eight months. The online and trace-driven experiments show that, compared to the common practice, AggCast can save over 15% back-to-source (BTS) bandwidth costs while having no negative impacts on QoS.

Federated learning (FL) is a privacy-preserving distributed learning paradigm that enables collaborative training of machine learning models among multiple participants. However, despite recent progress, existing federated learning systems can still not handle heterogeneous models. For instance, candidate clients with heterogeneous models are inaccessible to the established federated system. And within the federated system, local models are forbidden to be updated to become heterogeneous models, even though the updated models work better.

Internet adaptive video streaming is a typical form of video delivery that leverages adaptive bitrate (ABR) algorithms to provide video services with high quality of experience (QoE) for various users in diverse and unique network conditions. Such heterogeneous network environments, which can be viewed as exogenous input processes, often lead to the unstable performance of ABR algorithms. Unfortunately, learning-based ABR algorithm which generated by state-of-the-art reinforcement learning (RL) technologies achievesgood average performancebut fails to perform well in all kinds of network conditions. In this work, considering the video playback process as the Input-driven Markov Decision Process (IMDP), we propose$\text{A}^{2}$BR (Adaptation of ABR), a novel meta-RL ABR approach.$\text{A}^{2}$BR is mainly composed of an online stage and an offline stage. It leverages meta-RL to learn an initial meta-policy with various network conditions at the offline stage and makes decisions in personalized network conditions at the online stage. At the same time, we continually optimize the meta-policy to the tailor-made ABR policy for varying the current network environment within few shots. Moreover, in order to improve the learning efficiency, we fully utilize domain knowledge for implementing a virtual player to replay the previously experienced network. Using trace-driven experiments on various scenarios including different vehicles, users, network types, and heterogeneous user-preferences, we show that$\text{A}^{2}$BR outperforming recent ABR approaches with rapidly adapting to the personalized QoE metrics and specific network conditions. Testbed experimental results also illustrate the superiority of$\text{A}^{2}$BR in adapting to the unseen environments.

Video transmission services adopt adaptive algorithms to ensure users’ demands. Existing techniques are often optimized and evaluated by a function that linearly combines several weighted metrics. Nevertheless, we observe that the given function often fails to describe the requirement accurately, resulting in the violation of generating the required methods. We proposeZwei, a self-play reinforcement learning framework that updates the policy by straightforwardly utilizing the actual requirement. Technically, Zwei effectively rolls out the trajectories from the same initial state, and instantly estimate the win rate w.r.t the competition outcome, where the outcome represents which trajectory is closer to the assigned requirement. We evaluate Zwei with different requirements on various video transmission tasks, including adaptive bitrate streaming, crowd-sourced live streaming scheduling, and real-time communication. Results indicate that Zwei optimizes itself according to the assigned requirement faithfully, outperforming the state-of-the-art methods under all considered scenarios. Moreover, we further proposeZwei$^+$, which enables Zwei to learn the policies in the vanilla no-regret reinforcement learning scenario. We validate Zwei$^+$in the adaptive bitrate streaming task and show the superiority of the proposed method over existing state-of-the-art approaches.

In this paper, we propose a novel mechanism for alleviating the negative transfer in multi-task learning. Multi-task learning aims to learn the general meta-knowledge via sharing inductive bias for improving generalization ability. However, there exists a negative transfer problem in MTL, in which the performance improvement of a specific task leads to performance degradation of other tasks. Multi-task learning is essentially a multi-objective problem because of the task competition, necessitating a trade-off in individual optimality. Inspired by knowledge distillation, a set of task-specific teacher models can transfer the knowledge to a student multi-task model, without signicifantly loss in performance. In other words, the individual optimality can be achieved via multi-teacher knowledge distillation, where each teacher model is optimal and contains sufficient general meta-knowledge. Therefore, we propose multi-task distillation to couple with the negative transfer, turning the multi-objective optimization problem into a multi-teacher knowledge distillation problem. In detail, we firstly collect the optimal task-specific teacher models, and then achieve the individual optimality of the student model by knowledge distillation. Extensive experiment results on different benchmark datasets demonstrate the effectiveness of our method over current state-of-the-art multi-task learning formulations and single-task training.

360 video virtual cinematography attempts to direct a virtual camera and capture the most salient regions of 360 videos. In this paper, we propose a data-drive solution to achieve high-quality and diversified 360 cinematography based on crowd-sourced viewing histories. Specifically, we try to address two problems: 1) how to locate the semantically important regions of interest (RoI) from raw data, 2) how to generate virtual camera paths that follow chronological narratives. We first design a dynamic spherical mixture model based algorithm to locate variable number of RoIs on each video frame. We then model the camera transition and chronological orders with a Bayesian network and conditional probabilities. With the above two designs, we can generate “optimal” cinematography paths based on a dynamic programming algorithm. By modeling the RoIs as spherical mixture model, we are also able to provide diversified cinematography results. We show its effectiveness through extensive experiments.

Federated Learning (FL) is a learning paradigm, which allows the model to directly use a large amount of data in edge devices for training without heavy communication costs and privacy leakage. An important problem that FL faced is the heterogeneity of data at different edge nodes, resulting in a lack of convergence efficiency. In this paper, we propose Federated Trace (FedTrace) to address this problem. In FedTrace, we define the time series of some performance metrics of the global model on the edge node as the training trace of this node, which can reflect the data distribution of the edge node. By clustering the training traces, we can know which nodes have similar data distribution, which can guide the selection of nodes in each round of training. Here, we use a simple but effective method, that is, randomly selecting nodes from each cluster evenly. Experiments on various settings demonstrate that our method significantly reduces the number of communication rounds required in FL.

In the typical transcoding pipeline for adaptive video streaming, raw videos are pre-chunked and pre-encoded according to a set of resolution-bitrate or resolution-quality pairs on the server-side, where the pair is often named as bitrate ladder. Different from existing heuristics, we argue that a good bitrate ladder should be optimized by considering video content features, network capacity, and storage costs on the cloud. We propose DeepLadder, a per-chunk optimization scheme which adopts state-of-the-art deep reinforcement learning (DRL) method to optimize the bitrate ladder w.r.t the above concerns. Technically, DeepLadder selects the proper setting for each video resolution autoregressively. We use over 8,000 video chunks, measure over 1,000,000 perceptual video qualities, collect real-world network traces for more than 50 hours, and invent faithful virtual environments to help train DeepLadder efficiently. Across a series of comprehensive experiments on both Constant Bitrate (CBR) and Variable Bitrate (VBR)-encoded videos, we demonstrate significant improvements in average video quality bandwidth utilization, and storage overhead in comparison to prior work as well as the ability to be deployed in the real-world transcoding framework.

Conventional tile-based 360° video streaming methods, including deep reinforcement learning (DRL) based, ignore the interactive nature of 360° video streaming and download tiles following fixed sequential orders, thus failing to respond to the user's head motion changes. We show that these existing solutions suffer from either the prefetch accuracy or the playback stability drop. Furthermore, these methods are constrained to serve only one fixed streaming preference, causing extra training overhead and the lack of generalization on unseen preferences. In this paper, we propose a dual-queue streaming framework, with accuracy and stability purposes respectively, to enable the DRL agent to determine and change the tile download order without incurring overhead. We also design a preference-aware DRL algorithm to incentivize the agent to learn preference-dependent ABR decisions efficiently. Compared with state-of-the-art DRL baselines, our method not only significantly improves the streaming quality, e.g., increasing the average streaming quality by 13.6% on a public dataset, but also demonstrates better performance and generalization under dynamic preferences, e.g., an average quality improvement of 19.9% on unseen preferences.

Modeling powerful interactions is a critical challenge in Click-through rate (CTR) prediction, which is one of the most typical machine learning tasks in personalized advertising and recommender systems. Although developing hand-crafted interactions is effective for a small number of datasets, it generally requires laborious and tedious architecture engineering for extensive scenarios. In recent years, several neural architecture search (NAS) methods have been proposed for designing interactions automatically. However, existing methods only explore limited types and connections of operators for interaction generation, leading to low generalization ability. To address these problems, we propose a more general automated method for building powerful interactions named AutoPI. The main contributions of this paper are as follows: AutoPI adopts a more general search space in which the computational graph is generalized from existing network connections, and the interactive operators in the edges of the graph are extracted from representative hand-crafted works. It allows searching for various powerful feature interactions to produce higher AUC and lower Logloss in a wide variety of applications. Besides, AutoPI utilizes a gradient-based search strategy for exploration with a significantly low computational cost. Experimentally, we evaluate AutoPI on a diverse suite of benchmark datasets, demonstrating the generalizability and efficiency of AutoPI over hand-crafted architectures and state-of-the-art NAS algorithms.

Owing to the unprecedented growth of video demands, video caching has been a basic network functionality in today's network architectures to offload backbone traffics, as well as provide users with lower access delay. Although abundant cache replacement algorithms have been proposed recently, they all suffer from a critical limitation: due to their immature rules, inaccurate feature engineering or unresponsive model update, they cannot strike a balance between the long-term history and short-term sudden events. To tackle this problem, we propose LA-E2, a long-short-term fusion cache replacement approach, which is based on a learning-aided exploration-exploitation process. Specifically, by effectively combining the deep neural network (DNN) based prediction with the online exploitation exploration through a top-k method, LA-E2 can both make use of the historical information and adapt to the constantly changing popularity responsively. Through the extensive experiments in two real-world datasets, LA-E2 is demonstrated to achieve state-of-the-art performance and generalize well. Especially when the cache size is small, LA-E2 outperforms the baselines by 17.5% 68.7% higher in total hit rate.

Federated learning (FL) refers to the learning paradigm that trains machine learning models directly in the decentralized systems consisting of smart edge devices without transmitting the raw data, which avoids the heavy communication costs and privacy concerns. Given the typical heterogeneous data distributions in such situations, the popular FL algorithm Federated Averaging (FedAvg) suffers from weight divergence and thus cannot achieve a competitive performance for the global model (denoted as the initial performance in FL) compared to centralized methods. In this paper, we propose the local continual training strategy to address this problem. Importance weights are evaluated on a small proxy dataset on the central server and then used to constrain the local training. With this additional term, we alleviate the weight divergence and continually integrate the knowledge on different local clients into the global model, which ensures a better generalization ability. Experiments on various FL settings demonstrate that our method significantly improves the initial performance of federated models with few extra communication costs.

The leading adaptive bitrate (ABR) algorithm leverages model predictive control (MPC) method to determine next chunks' video bitrate, while it heavily relies on the accuracy of throughput prediction, which thereby fails to perform well in all considered network scenarios. In this paper, we propose DeepMPC, which enhances MPC via two deep learning-based modules, i.e., DL-based Throughput Predictor (DTP), which can precisely predict future bandwidth, and Discounted Factor Optimizer (DFO), which estimates the prediction error. Using trace-driven experiments, we illustrate that DeepMPC outperforms existing ABR schemes in all considered network conditions, with the improvements on average QoE of 5.91% - 56.1%. Moreover, we implement DeepMPC in real-world network environments and extensive experimental results demonstrate the superiority of DeepMPC against existing state-of-the-art approaches.

Recently, data-driven prediction strategies have shown the potential of shepherding the optimization strategies for end viewer's Quality-of-Experience in practical streaming applications. The current prediction-based designs have largely focused on optimizing the last-mile, i.e., viewer-side, which 1) need the real-time feedback from viewers to improve the prediction accuracy; and 2) need quick responses to guarantee the effectiveness of optimization strategies in the future. Thanks to the emerged crowdsourced livecast services, e.g., Twitch.tv, we for the first time exploit the opportunity to realize the long-term prediction and optimization with the assistance derived from the first-mile, i.e., source broadcasters.In this paper, we propose a novel framework CastFlag, which analyzes the broadcasters' operations and interactions, predicts the key events (i.e., highlights), and optimizes the transcoding stage in the corresponding live streams, even before the encoding stage. Taking the most popular eSports gamecast as an example, we illustrate the effectiveness of this framework in the game highlight prediction and transcoding workload allocation. The trace-driven evaluation shows the superiority of CastFlag as it: (1) improves the prediction accuracy over other learning-based approaches by up to 30%; (2) achieves an average of 10% saving of the transcoding latency at less cost.

Off-the-shelf buffer-based approaches leverage a simple yet effective buffer-bound to control the adaptive bitrate (ABR) streaming system. Nevertheless, such approaches in standard parameters fail to always provide high quality of experience (QoE) video streaming services under all considered network conditions. Meanwhile, state-of-the-art learning-based ABR approach Pensieve outperforms existing schemes but is impractical to deploy. Therefore, how to harmoniously fuse the buffer-based and learning-based approach has become a key challenge for further enhancing ABR methods. In this paper, we propose Stick, an ABR algorithm that fuses the deep learning method and traditional buffer-based method. Stick utilizes the deep reinforcement learning (DRL) method to train the neural network, which outputs the buffer-bound to control the buffer-based approach for maximizing the QoE metric with different parameters. Trace-driven emulation illustrates that Stick betters Pensieve by 3.5% - 9.41% with an overhead reduction of 88%. Moreover, aiming to further reduce the computational costs while preserving the performances, we propose Trigger, a light-weighted neural network that determines whether the buffer-bound should be adjusted. Experimental results show that Stick+Trigger rivals or outperforms existing schemes in average QoE by 1.7%-28%, and significantly reduces the Stick's computational overhead by 24%-61%. Meanwhile, we show that Trigger also helps other ABR schemes mitigate the overhead. Extensive results on real-world evaluation demonstrate the superiority of Stick over existing state-of-the-art approaches.

Today's explosively growing Internet video traffics and viewers' ever-increasing quality of experience (QoE) demands for video streaming bring tremendous pressures to the backbone network. As a new network paradigm, mobile edge caching provides a promising alternative by pushing video content closer at the network edge rather than the remote CDN servers so as to reduce both content access latency and redundant network traffic. However, our large-scale trace analysis shows that different from CDN based caching, edge caching environment is much more complicated with massively dynamic and diverse request patterns, which renders that existing rule-based and model-based caching solutions may not well fit such complicated edge environments. Moreover, although cooperative caching has been proposed to better afford limited storage on each individual edge server, our trace analysis also shows that the request similarity among neighboring edges can be highly dynamic and diverse, which is drastically different from CDN based caching environment, and can easily compromise the benefits from traditional cooperative caching mostly designed based on CDN environment. In this paper, we propose MacoCache, an intelligent edge caching framework that is carefully designed to afford the massively diversified and distributed caching environment to minimize both content access latency and traffic cost. Specifically, MacoCache leverages a multi-agent deep reinforcement learning (MADRL) based solution, where each edge is able to adaptively learn its own best policy in conjunction with other edges for intelligent caching. The real trace-driven evaluation further demonstrates that MacoCache is able to reduce an average of 21% latency and 26% cost compared with the state-of-the-art caching solution.

Livecast streaming has received great success in recent years. Although many prior efforts have suggested that dynamic viewer scheduling according to the quality of service (QoS) can improve user engagement, they may suffer inefficiency due to their ignorance of viewer heterogeneity in how the QoS impact quality of experience (QoE).

Automatic text summarization has been a hot research topic for years. Though most of the existing studies only use the content itself to generate the summaries, researchers believe that an individual's reading behaviors have much to do with the summaries s/he generates, usually regarded as the ground truth. However, such research is limited by the lack of a dataset that provides the connection between people's reading behaviors and the summaries provided by them. This paper fills in this gap by providing a dataset covering 50 individuals' gaze behaviors collected by a high-accurate eye tracking device (that generates 100 gaze points per second) when they are reading 100 articles (from 10 popular categories) and composing the corresponding summaries for each article. Collected in a controlled environment, our dataset with 157 million gaze points in total, provides not only the basic gaze behaviors when different people read an article and compose its corresponding summary, but also the connections between different behavior patterns and the summaries they will provide. We believe such a dataset will be valuable for a wide range of studies, and we also provide sample use cases of the dataset.

Video transmission services adopt adaptive algorithms to ensure users' demands. Existing techniques are often optimized and evaluated by a function that linearly combines several weighted metrics. Nevertheless, we observe that the given function fails to describe the requirement accurately. Thus, such proposed methods might eventually violate the original needs. To eliminate this concern, we propose Zwei, a self-play reinforcement learning algorithm for video transmission tasks. Zwei aims to update the policy by straightforwardly utilizing the actual requirement. Technically, Zwei samples a number of trajectories from the same starting point, and instantly estimates the win rate w.r.t the competition outcome. Here the competition result represents which trajectory is closer to the assigned requirement. Subsequently, Zwei optimizes the strategy by maximizing the win rate. To build Zwei, we develop simulation environments, design adequate neural network models, and invent training methods for dealing with different requirements on various video transmission scenarios. Trace-driven analysis over two representative tasks demonstrates that Zwei optimizes itself according to the assigned requirement faithfully, outperforming the state-of-the-art methods under all considered scenarios.

Existing Adaptive Bitrate (ABR) algorithms pick future video chunks' bitrates via fixed rules or offline trained models to ensure good quality of experience (QoE) for Internet video. Nevertheless, data analysis demonstrates that a good ABR algorithm is required to continually and fast update for adapting itself to time-varying network conditions. Therefore, we propose Comyco, a video quality-aware learning-based ABR approach that enormously improves recent schemes by i) picking the chunk with higher perceptual video qualities rather than video bitrates; ii) training the policy via imitating expert trajectories given by the expert strategy; iii) employing the lifelong learning method to continually train the model w.r.t the fresh trace collected by the users. To achieve this, we develop a complete quality-aware lifelong imitation learning-based ABR system, construct quality-based neural network architecture, collect a quality-driven video dataset, and estimate QoE metrics with video quality features. Using trace-driven and real-world experiments, we demonstrate Comyco reaches 1700-fold improvements in the number of samples required and 16-fold speedup in the training time compared with the prior work. Meanwhile, Comyco outperforms existing methods, with the improvements on average QoE of 7.5%-16.79%. Moreover, experimental results on continual training also illustrate that lifelong learning helps Comyco further improve the average QoE of 1.07%-9.81% in comparison to the offline trained model.

Scheduling viewers effectively among different Content Delivery Network (CDN) providers is challenging owing to the extreme diversity in the crowdsourced live streaming (CLS) scenarios. Abundant algorithms have been proposed in recent years, which, however, suffer from a critical limitation: Due to their inaccurate feature engineering or naive rules, they cannot optimally schedule viewers. To address this concern, we put forward LTS (Learn to Schedule), a novel scheduling algorithm that can adapt to the dynamics from both viewer traffics and CDN performance. In detail, we first propose LTS-RL, an approach that schedules CLS viewers based on deep reinforcement learning (DRL). Since LTS-RL is trained in an end-to-end way, it can automatically learn scheduling algorithms without any pre-programmed models or assumptions about the environment dynamics. At the same time, to practically deploy LTS-RL, we then use the decision tree and imitation learning to convert LTS-RL into a more light-weighted and interpretable model, which is denoted as Fast-LTS. After the extensive evaluation of the real data from a leading CLS platform in China, we demonstrate that our proposed model (both LTS-RL and Fast-LTS) can improve the average quality of experience (QoE) over state-of-the-art approaches by 8.71--15.63%. At the same time, we also demonstrate that Fast-LTS can faithfully convert the complicated LTS-RL with slight performance degradation (< 2%), while significantly reducing the decision time (×7--10).

Today’s anywhere and anytime broadband connection and audio/video capture have boosted the deployment of crowdsourced livecast services (orcrowdcast). Bridging a massive amount of geo-distributed broadcasters and their fellow viewers, such representatives as Twitch.tv, Youtube Gaming, and Inke.tv, have greatly changed the generation and distribution landscape of streaming content. They also enable rich online interactions among the crowd, and strive to offer personalized Quality-of-Experience (QoE) for individual viewers. Given the ultra-large scale and the dynamics of the crowd, personalizing QoE however is much more challenging than in early generation streaming services. The rich interactions among the broadcasters, viewers, and the network system, on the other hand, also offer invaluable data that could be utilized towards informed management. This paper presentsDeepCast, an edge-assisted crowdcast framework that explores the sheer amount of viewing data towards intelligent decisions for personalized QoE demands. DeepCast seamlessly integrates cloud, CDN, and edge servers for crowdcast content distribution, and advocates a data-driven design that extracts the hidden information from the complex interactions among the system components. Through deep reinforcement learning (DRL), it automatically identifies the most suitable strategies for viewer assignment and transcoding at edges. We collect multiple real-world datasets and evaluate the performance of DeepCast with trace-driven experiments. The results demonstrate its flexibility and effectiveness towards better personalized QoE and lower cost for crowdcast systems.

Federated learning enables on-device training over distributed networks consisting of a massive amount of modern smart devices, such as smartphones and IoT devices. However, the leading optimization algorithm in such settings, i.e., federated averaging, suffers from heavy communication cost and inevitable performance drop, especially when the local data is distributed in a Non-IID way. In this paper, we propose a feature fusion method to address this problem. By aggregating the features from both the local and global models, we achieve a higher accuracy at less communication cost. Furthermore, the feature fusion modules offer better initialization for newly incoming clients and thus speed up the process of convergence. Experiments in popular federated learning scenarios show that our federated learning algorithm with feature fusion mechanism outperforms baselines in both accuracy and generalization ability while reducing the number of communication rounds by more than 60%.

Existing reinforcement learning (RL)-based adaptive bitrate (ABR) approaches outperform the previous fixed control rules based methods by improving the Quality of Experience (QoE) score, as the QoE metric can hardly provide clear guidance for optimization, finally resulting in the unexpected strategies. In this paper, we propose Tiyuntsong, a self-play reinforcement learning approach with generative adversarial network (GAN)-based method for ABR video streaming. Tiyuntsong learns strategies automatically by training two agents who are competing against each other. Note that the competition results are determined by a set of rules rather than a numerical QoE score that allows clearer optimization objectives. Meanwhile, we propose GAN Enhancement Module to extract hidden features from the past status for preserving the information without the limitations of sequence lengths. Using testbed experiments, we show that the utilization of GAN significantly improves the Tiyuntsong's performance. By comparing the performance of ABRs, we observe that Tiyuntsong also betters existing ABR algorithms in the underlying metrics.

Video transcoding is widely adopted in live streaming services to bridge the format and resolution gap between content producers and consumers (i.e., broadcasters and viewers). Meanwhile, the cloud has been recognized as one of the most reliable and cost-effective ways for video transcoding. However, due to the dynamic and uncertainty of the transcoding workloads in live streaming, it is very challenging for cloud service providers to provision computing resources and schedule transcoding tasks while guaranteeing the Service Level Agreement (SLA). To this end, we propose a joint resource provisioning and task scheduling approach for transcoding live streams in the cloud. We adopt Deep Reinforcement Learning (DRL) to train a neural network model for resource provisioning under dynamic workloads. Moreover, we design a QoS-aware task scheduling algorithm that maps transcoding tasks to Virtual Machines (VMs) by considering the real-time QoS requirement. We evaluate our approach with trace-driven experiments and the results demonstrate that our approach outperforms heuristic baselines by up to 89% improvements on average QoS with 4% extra resource overhead at most.

Recently, the fusion of 360° video and multi-viewpoint video, called multi-viewpoint (MVP) 360° interactive video, has emerged and created much more immersive and interactive user experience, but calls for a low latency solution to request the high-definition contents. Such viewing-related features as head movement have been recently studied, but several key issues still need to be addressed. On the viewer side, it is not clear how to effectively integrate different types of viewing-related features. At the session level, questions such as how to optimize the video quality under dynamic networking conditions and how to build an end-to-end mapping between these features and the quality selection remain to be answered. The solutions to these questions are further complicated given the many practical challenges, e.g., incomplete feature extraction and inaccurate prediction.This paper presents an architecture, called iView, to address the aforementioned issues in an MVP 360° interactive video scenario. To fully understand the viewing-related features and provide a one-step solution, we advocate multimodal learning and deep reinforcement learning in the design. iView intelligently determines video quality and reduces the latency without pre-programmed models or assumptions. We have evaluated iView with multiple real-world video and network datasets. The results showed that our solution effectively utilizes the features of video frames, networking throughput, head movements, and viewpoint selections, achieving at least 27.2%, 15.4%, and 2.8% improvements on the three video datasets, respectively, compared with several state-of-the-art methods.

Recent years have seen booming development and great success in interactive crowdsourced livecast (i.e., crowdcast). Different from traditional livecast services, crowdcast is featured with tremendous video contents at the broadcaster side, highly diverse viewer side content watching environments/preferences as well as viewers' personalized quality of experience (QoE) demands (e.g., individual preferences for streaming delays, channel switching latencies and bitrates). This imposes unprecedented key challenges on how to flexibly and cost-effectively accommodate the heterogeneous and personalized QoE demands for the mass of viewers. In this paper, we propose DeepCast, an edge-assisted crowdcast framework, which makes intelligent decisions at edges based on the massive amount of real-time information from the network and viewers to accommodate personalized QoE with minimized system cost. Given the excessive computation complexity in this context, we propose a data-driven deep reinforcement learning (DRL) based solution that can automatically learn the best suitable strategies for viewer scheduling and transcoding selection. To our best knowledge, DeepCast is the first edge-assisted framework that applies the advance of DRL to explicitly accommodate personalized QoE optimization for crowdcast services. We collect multiple real-world datasets and evaluate the performance of DeepCast using trace-driven experiments. The results demonstrate the superiority of our DeepCast framework and its DRL-based solution.

Learning-based Adaptive Bit Rate~(ABR) method, aiming to learn outstanding strategies without any presumptions, has become one of the research hotspots for adaptive streaming. However, it is still suffering from several issues, i.e., low sample efficiency and lack of awareness of the video quality information. In this paper, we propose Comyco, a video quality-aware ABR approach that enormously improves the learning-based methods by tackling the above issues. Comyco trains the policy via imitating expert trajectories given by the instant solver, which can not only avoid redundant exploration but also make better use of the collected samples. Meanwhile, Comyco attempts to pick the chunk with higher perceptual video qualities rather than video bitrates. To achieve this, we construct Comyco's neural network architecture, video datasets and QoE metrics with video quality features. Using trace-driven and real world experiments, we demonstrate significant improvements of Comyco's sample efficiency in comparison to prior work, with 1700x improvements in terms of the number of samples required and 16x improvements on training time required. Moreover, results illustrate that Comyco outperforms previously proposed methods, with the improvements on average QoE of 7.5% - 16.79%. Especially, Comyco also surpasses state-of-the-art approach Pensieve by 7.37% on average video quality under the same rebuffering time.

Viewer scheduling among different CDN providers in crowdsourced live streaming (CLS) service is especially challenging due to the large-scale dynamic viewers as well as the time-variant performance of the content delivery network. A practical scheduling method should tackle the following challenges: 1) accurate modeling of viewer patterns and CDN performance; 2) intelligent workload offloading to save costs while guaranteeing the quality of service (QoS); 3) and ease of integration with practical CDN infrastructure in CLS platforms.

The leading learning-based rate control method, i.e., QARC, achieves state-of-the-art performances but fails to interpret the fundamental principles, and thus lacks the abilities to further improve itself efficiently. In this paper, we propose EQARC (Explainable QARC) via reconstructing QARC's modules, aiming to demystify how QARC works. In details, we first utilize a novel hybrid attention-based CNN+GRU model to re-characterize the original quality prediction network and reasonably replace the QARC's 1D-CNN layers with 2D-CNN layers. Using trace-driven experiment, we demonstrate the superiority of EQARC over existing state-of-the-art approaches. Next, we collect several useful information from each interpretable modules and learn the insight of EQARC. Following this step, we further propose AQARC (Advanced QARC), which is the light-weighted version of QARC. Experimental results show that AQARC achieves the same performances as the QARC with an overhead reduction of 90%. In short, through learning from deep learning, we generalize a rate control method which can both reach high performance and reduce computation cost.

With the growing demand for crowdsourced live streaming (CLS), how to schedule the large-scale dynamic viewers effectively among different Content Delivery Network (CDN) providers has become one of the most significant challenges for CLS platforms. Although abundant algorithms have been proposed in recent years, they suffer from a critical limitation: due to their inaccurate feature engineering or naive rules, they cannot optimally schedule viewers. To address this concern, we propose LTS (Learn to schedule), a deep reinforcement learning (DRL) based scheduling approach that can dynamically adapt to the variation of both viewer traffics and CDN performance. After the extensive evaluation the real data from a leading CLS platform in China, we demonstrate that LTS improves the average quality of experience (QoE) over state-of-the-art approach by 8.71%-15.63%.

, is one of the major roadblocks that prevent deep neural networks from achieving human-level artificial intelligence. Several research efforts (e.g., lifelong or continual learning algorithms) have proposed to tackle this problem. However, they either suffer from an accumulating drop in performance as the task sequence grows longer, or require storing an excessive number of model parameters for historical memory, or cannot obtain competitive performance on the new tasks. In this letter, we focus on the incremental multitask image classification scenario. Inspired by the learning process of students, who usually decompose complex tasks into easier goals, we propose an adversarial feature alignment method to avoid catastrophic forgetting. In our design, both the low-level visual features and high-level semantic features serve as soft targets and guide the training process in multiple stages, which provide sufficient supervised information of the old tasks and help to reduce forgetting. Due to the knowledge distillation and regularization phenomena, the proposed method gains even better performance than fine-tuning on the new tasks, which makes it stand out from other methods. Extensive experiments in several typical lifelong learning scenarios demonstrate that our method outperforms the state-of-the-art methods in both accuracy on new tasks and performance preservation on old tasks.

Crowdsourced live streaming (CLS), such as Twitch.tv and Inke.tv, has emerged as an important multimedia application in recent years. Video delivery in such CLS service involves two steps: 1) video uploading-video streaming (i.e., a live channel) generated from a broadcaster is uploaded to the server, which we call the “first mile” network and 2) video distribution-the video streaming is then delivered to viewers in the channel. Today's CLS services usually use conventional content delivery network solutions to address the video distribution problem, while little attention has been paid to improve the video uploading quality. Our measurement study shows that the first mile network causes 17% viewer rebuffers, and some viewers quit the channel once encountering rebuffer. In this paper, we propose a content harvest network (CHN) architecture to address the uploading problem in the CLS service. Specifically, the CHN architecture employs edge devices in the network as relays to receive the streaming uploaded by broadcasters and then forward to the central servers. On one hand, we need to reduce the latency since it is live streaming; on the other hand, we must provide sustainable upload bandwidth. It is challenging to achieve both at the same time, especially in such high dynamic system as CLS. In order to provide global optimal and real-time assignment, we propose a hybrid solution, i.e., centralized and distributed assignment. Specifically, we formulate the centralized relay assignment problem as an optimization problem to achieve both low latency and sustainable bandwidth. To cope with the frequent channel establishments, we use a multi-armed bandit method to characterize the time-variant network condition. Experiment results on a large-scale trace provided by Inke.tv show that our solution can reduce the overall viewer cost by 40% compared to state-of-the-art solutions. The viewers' rebuffer can also be reduced by 50%.

Adaptive bitrate streaming enables video users toadapttheir playing bitrates to the real-time network conditions, hence achieving the desirable quality-of-experience (QoE). In a multi-user wireless scenario, however, existing single-user based bitrate adaptation methods may fail to provide the desirable QoE, due to lack of consideration of multi-user interactions (such as the multi-user interferences and network congestion). In this work, we propose a novel user cooperation framework based onuser-provided networkingfor multi-user mobile video streaming over wireless cellular networks. The framework enables nearby mobile video users to crowdsource their cellular links and resources for cooperative video streaming. We first analyze the social welfare performance bound of the proposed cooperative streaming system by introducing a virtual time-slotted system. Then, we design a low complexity Lyapunov-based online algorithm, which can be implemented in an online and distributed manner without the complete future and global network information. Numerical results show that the proposed online algorithm achieves an average 97 percent of the theoretical maximum social welfare. We further conduct experiments with real data traces, to compare our proposed online algorithm with the existing online algorithms in the literature. Experiment results show that our algorithm outperforms the existing algorithms in terms of both the achievable bitrate (with an average gain of 20$\sim$30 percent) and social welfare (with an average gain of 10$\sim$50 percent).

Edge caching is a key technology to accelerate the content delivery and to mitigate the network congestion. As a real-world example, Microsoft has been providing the edge caching service with a usage-based pricing mechanism since 2009. However, the challenge of designing a better pricing mechanism lies in balancing the finer grain size of the mechanism and the complexity of business operations it required. Our work proposes a novel location dependent pricing mechanism for the edge caching market which both improves the Internet service provider's (ISP) revenue and the social welfare. First, we introduce the location dependent pricing mechanism (LDP) that offers location-specific price, considering the popularity differences between the edge nodes(e.g. base stations, WiFi access points) at different locations. We capture the interactions between the content providers (CPs) and the ISP as a two-stage Stackelberg game and derive the CP's best responses and the ISP's optimal pricing strategy. Then we provide a partial location dependent pricing scheme (PLDP) which limits the number of price tiers to provide a balance between the optimality and economic viability. Finally, through extensive numerical simulation, we find that the LDP (or PLDP) improves the revenue of the ISP by 31%↑50% (23%↑%35) and also improves the social welfare by 46% (%32) on average, compared to the uniform pricing scheme.

The emerging 5G mobile networking promises ultrahigh network bandwidth and ultra-low communication latency (100ms), due to its store-and-forward design and the physical barrier from signal propagation speed, not to mention congestion that frequently happens. Caching is known to be effective to bridge the speed gap, which has become a critical component in the 5G deployment as well. Besides storage, 5G base stations (BSs) will also be powered with strong computing modules, offering mobile edge computing (MEC) capability. This paper explores the potentials of edge computing towards improving the cache performance, and we envision a learning-based framework that facilitates smart caching beyond simple frequency- and time-based replace strategies and cooperation among base stations. Within this framework, we develop DeepCache, a deep-learning-based solution to understand the request patterns in individual base stations and accordingly make intelligent cache decisions. Using mobile video, one of the most popular applications with high traffic demand, as a case, we further develop a cooperation strategy for nearby base stations to collectively serve user requests. Experimental results on real-world dataset show that using the collaborative DeepCache algorithm, the overall transmission delay is reduced by 14%~22%, with a backhaul data traffic saving of 15%~23%.

Real-time video streaming is now one of the main applications in all network environments. Due to the fluctuation of throughput under various network conditions, how to choose a proper bitrate adaptively has become an upcoming and interesting issue. To tackle this problem, most proposed rate control methods work for providing high video bitrates instead of video qualities. Nevertheless, we notice that there exists a trade-off between sending bitrate and video quality, which motivates us to focus on how to reach a balance between them.

Enabling users to interact with broadcasters and audience, the crowd-interactive livecast greatly improves viewer's quality of experience (QoE) and attracts millions of daily active users recently. In addition to striking the balance between resource utilization and viewers' QoE met in the traditional video streaming service, this novel service needs to take supererogatory efforts to improve the interaction QoE, which reflects the viewer interaction experience. To tackle this issue, we conduct measurement studies over a large-scale dataset crawled from a representative livecast service provider. We observe that the individual's interaction pattern is quite heterogeneous: only 10% viewers proactively participate in the interaction, and the rest viewers usually watch passively. Incorporating the insight into the emerging cloud-edge architecture, we propose a framework PIECE, which optimizes the Personalized Interaction Experience with Cloud-Edge architecture (PIECE) for intelligent user access control and livecast distribution. In particular, we first devise a novel deep neural network based algorithm to predict users' interaction intensity using the historical viewer pattern. We then design an algorithm to maximize the individual's QoE, by strategically matching viewer sessions and transcoding-delivery paths over cloud-edge infrastructure. Finally, we use trace-driven experiments to verify the effectiveness of PIECE. Our results show that our prediction algorithm outperforms the state-of-the-art algorithms with a much smaller mean absolute error (40% reduction). Furthermore, in comparison with the cloud-based video delivery strategy, the proposed framework can simultaneously improve the average viewers QoE (26% improvement) and interaction QoE (21% improvement), while maintaining a high streaming bitrate.

Rate control is widely adopted during video streaming to provide both high video qualities and low latency under various network conditions. However, despite that many work have been proposed, they fail to tackle one major problem: previous methods determine a future transmission rate as a single for value which will be used in an entire time-slot, while real-world network conditions, unlike lab setup, often suffer from rapid and stochastic changes, resulting in the failures of predictions.

Cellular data sponsoring (CDS) is a traditional data sponsor scheme widely used in cellular video delivery networks, where content providers (CPs) bear the cellular data downloading cost for mobile video users (MUs), so as to attract more MUs and achieve higher revenue (e.g., via more attached advertisements). Edge caching sponsoring (ECS) is a novel data sponsor scheme recently introduced in the emerging 5G network, where CPs cache popular video contents on the edge network in advance and deliver them to local MUs directly. Thus, it can not only achieve the benefits of CDS (i.e., attracting more MUs and achieving higher revenue), but also reduce the congestion of backhaul network. In this work, we will perform a competitive analysis of CDS and ECS for mobile video streaming. Specifically, we consider a mobile video delivery network with two CPs who adopt CDS and ECS, respectively. MUs can choose one or neither of these two sponsor schemes (from the corresponding CPs) for his video content requests. We formulate the interaction of CPs and MUs as a two-stage Stackelberg game, where CPs act as leaders determining the efforts of their adopted sponsor schemes in the first stage, and MUs act as followers choosing the best sponsor schemes for their content requests in the second stage. We analyze the sub-game perfect equilibrium systematically for both cooperative and competitive scenarios (depending on whether two CPs cooperate or compete with each other). Numerical results show that in the competitive scenario, the joint sponsor of ECS and CDS can increase the total MU payoff by 36% ~ 140%, comparing with that with only one sponsor scheme. Moreover, the CPs can benefit more from ECS than from CDS when the revenue is higher.

Federated learning algorithm solves the problem of training machine learning models over distributed networks that consist of a massive amount of modern smart devices. It overcomes the challenge of privacy preservation, unbalanced and Non-IID data distributions, and does its best to reduce the required communication rounds. However, communication costs are still the principle constraint compared to other factors, such as computation costs. In this paper, we adopt a two-stream model with MMD (Maximum Mean Discrepancy) constraint instead of the single model to be trained on devices in standard federated learning settings. Following experiments show that the proposed model outperforms baseline methods, especially in Non-IID data distributions, and achieves a reduction of more than 20% in required communication rounds.

The emergence of smart Wi-Fi access points (AP), which are equipped with huge storage space, opens a new research area on how to utilize these resources at the edge network to improve users' quality of experience (e.g., a short startup delay and smooth playback). One important research interest in this area is content prefetching which predicts and accurately fetches contents ahead of users' requests to shift the traffic away during peak periods. However, in practice, the different video watching patterns among users and the varying network connection status lead to the time-varying server load, which eventually makes the content prefetching problem challenging. To understand this challenge, this paper first performs a large-scale measurement study on users' AP connection and TV series watching patterns using real traces. Then, based on the obtained insights, we formulate the content prefetching problem as a Markov decision process. The objective is to strike a balance between the increased prefetching and storage cost incurred by incorrect prediction and the reduced content download delay because of successful prediction. A learning-based approach is proposed to solve this problem and another three algorithms are adopted as baselines. In particular, first we investigate the performance lower bound by using a random algorithm and the upper bound by using an ideal offline approach. Then, we present a heuristic algorithm as another baseline. Finally, we design a reinforcement learning algorithm that is more practical to work in the online manner. Through extensive trace-based experiments, we demonstrate the performance gain of our design. Remarkably, our learning-based algorithm achieves a better precision and hit ratio (e.g., 80%) with about 70% (resp. 50%) cost saving compared to the random (resp. heuristic) algorithm

The explosive growth of mobile video traffic introduces new challenges for the network infrastructure. Edge caching, as one of the key technologies in 5G wireless networks, has shown great potential to improve the quality of mobile video services by reducing the transmission overhead over backhaul links. With edge caching, content providers (CPs) need to decide not only the traditional data sponsoring strategy on cellular networks (where CPs cover part or all of the mobile users' cellular data cost), but also a novel cache sponsoring strategy on the edge caching networks (where CPs place part of contents on edge networks in advance). In this paper, we study the joint optimization of both sponsors on cellular and edge caching networks for a single CP, aiming at maximizing the CP's revenue. Specifically, we formulate the joint optimization problem as a two-stage sequential decision problem. In stage I, the CP determines the edge caching policy (for a relatively long time period). In stage II, the CP decides the real-time data sponsoring strategy for each content request within the period. We analyze this two-stage decision problem systematically. First, we propose an online sponsoring strategy in stage II based on Lyapunov optimization framework. Then, we propose an edge caching strategy in stage I via predicting the number of aggregate user requests. Simulations on real data traces show that such a joint optimization policy can increase the CP's revenue by 124%-154%, comparing with the traditional data sponsoring policy (i.e., without edge caching). Moreover, the proposed online strategy can achieve 90% of the maximum revenue in the offline benchmark.

Mobile personal livecast (MPL) services are emerging and have received great attention recently. In MPL, numerous and geo-distributed ordinary people broadcast their video contents to worldwide viewers. Different from conventional social networking services like Twitter and Facebook, which have a tolerance for interaction delay, the interactions (e.g., chat messages) in a personal livecast must be in real-time with low feedback latency. These unique characteristics inspire us to: (1) investigate how the relationships (e.g., social links and geo-locations) between viewers and broadcasters influence the user behaviors, which has yet to be explored in depth; and (2) explore insights to benefit the improvement of system performance. In this article, we carry out extensive measurements of a representative MPL system, with a large-scale dataset containing 11M users. In the current costly and limited cloud-based MPL system, which is faced with scalability problem, we find: (1) the long content uploading distances between broadcasters and cloud ingesting servers result in an impaired system QoS, including a high broadcast latency and a frequently buffering events; and (2) most of the broadcasters in MPL are geographically locally popular (the majority of the views come from the same region of the broadcaster), which consume vast computation and bandwidth resources of the clouds and Content Delivery Networks. Fortunately, the emergence of edge computing, which provides cloud-computing capabilities at the edge of the mobile network, naturally sheds new light on the MPL system; i.e., localized ingesting, transcoding, and delivering locally popular live content is possible. Based on these critical observations, we propose an edge-assisted MPL system that collaboratively utilizes the core-cloud and abundant edge computing resources to improve the system efficiency and scalability. In our framework, we consider a dynamic broadcaster assignment to minimize the broadcast latency while keeping the resource lease cost low. We formulate the broadcaster scheduling as a stable matching with migration problem to solve it effectively. Compared with the current pure cloud-based system, our edge-assisted delivery approach reduces the broadcast latency by about 35%.

Data sponsoring is a widely-used incentive method in today's cellular networks, where video content providers (CPs) cover part or all of the cellular data cost for mobile users so as to attract more video users and increase data traffic. In the forthcoming 5G cellular networks, edge caching is emerging as a promising technique to deliver videos with lower cost and higher quality. The key idea is to cache video contents on edge networks (e.g., femtocells and WiFi access points) in advance and deliver the cached contents to local video users directly (without involving cellular data cost for users). In this work, we aim to study how the edge caching will affect the CP's data sponsoring strategy as well as the users' behaviors and the data market. Specifically, we consider a single CP who offers both the edge caching service and the data sponsoring service to a set of heterogeneous mobile video users (with different mobility and video request patterns). We formulate the interactions of the CP and the users as a two-stage Stackelberg game, where the CP (leader) determines the budgets (efforts) for both services in Stage I, and the users (followers) decide whether and which service(s) they would like to subscribe to. We analyze the sub-game perfect equilibrium (SPE) of the proposed game systematically. Our analysis and experimental results show that by introducing the edge caching, the CP can increase his revenue by 105%.

Edge caching technique is becoming more and more important in 5G network architecture, which shows great performance in terms of mitigating the backbone network congestion and reducing the delivery latency. The key idea of this technique is to cache popular contents in the small base stations(SBSs) which locate at the edge of the wireless network, and deliver these contents to nearby mobile users(MUs) from the cache directly. The collaborative in-cluster prefetching is improving the network performance even more. In this paper, we study the case where one monopolistic state-owned mobile network operator (MNO) offers edge caching service to multiple Content Providers(CPs), and focus on the mechanism design for the cache space allocation in this market. As the caching policy on each edge node is coupled with each other due to in-cluster collaboration, one time auction or flat pricing will both harm the social welfare. We propose a 2-optimal sequential auction mechanism (SAM) to solve this problem, which improves the social welfare by 20%~70% compared to the traditional Vickery-Clarke-Groves(VCG) mechanism.

Recent years have witnessed a new content delivery paradigm named crowdsourced CDN, in which devices deployed at edge network can prefetch contents and provide content delivery service. Crowdsourced CDN offers high-quality experience to end-users by reducing their content access latency and alleviates the load of network backbone by making use of network and storage resources at millions of edge devices. In such paradigm, redirecting content requests to proper devices is critical for user experience. The uniqueness of request redirection in such crowdsourced CDN lies that: on one hand, the bandwidth capacity of the crowdsourced CDN devices is limit, hence devices located at a crowded place can be easily overwhelmed when serving nearby user requests; on the other hand, contents requested in one device can be significantly different from another one, making request redirection strategies used in conventional CDNs which only aim to balance request loads ineffective. In this paper, we explore request redirection strategies that take both workload balance of devices and content requested by users into consideration. Our contributions are as follows. First, we conduct measurement studies, coving 1.8M users watching 0.4M videos, to understand request patterns in crowdsourced CDN. We observe that the loads of nearby devices can be very different and the contents requested at nearby devices can also be significantly different. These observations lead to our design for request balancing at nearby devices. Second, we formulate the request redirection problem by taking both the content access latency and the content replication cost into consideration, and propose a request balancing and content aggregation solution. Finally, we evaluate the performance of our design using trace-driven simulations, and observe our scheme outperforms the traditional strategy in terms of many metrics, e.g., we observe a content access latency reduction by 50% over traditional mechanisms such as the Nearest/Random request routing scheme.

In recent years, Dynamic Adaptive Streaming over HTTP (DASH) has gained momentum as an effective solution for delivering videos on the Internet. This trend is further driven by the deployment of existing HTTP cache infrastructures in DASH systems to reduce the traffic load as well as to serve clients better. However, deploying conventional cache servers in DASH systems still suffers from low cache hit ratio and bitrate oscillations, which makes it challenging for content providers (CPs) to balance the user-perceived quality-of-experience (QoE) and the operating cost in cache-enabled DASH systems. To address this challenge, we propose a CP-operated DASH caching framework to provide good user QoE with low cost. In particular, we first formulate the caching decision problem as a stochastic optimization problem over a finite time horizon. The objective of this problem is to maximize a weighted sum of the user QoE and the operating cost, termed as the utility. Then we design a reinforcement learning based online algorithm which can obtain approximately optimal solution of this problem. Through extensive trace-driven experiments, we show that our approach not only achieves 40% average improvement of the overall utility compared to baseline approaches, but also adapts to the server load.

Crowdsourced mobile video streaming enables nearby mobile video users to aggregate their network resources to improve the video streaming performance. However, users are often selfish and may not be willing to cooperate without proper incentives. Designing an incentive mechanism for such a scenario is challenging due to the users' asynchronous downloading behaviors as well as their private valuations for multi-bitrate encoded videos. In this work, we propose a multi-object multi-dimensional auction-based incentive framework, through which users can download multiple video segments with different bitrates for multiple nearby users (and themselves). Based on this incentive framework, we propose a Vickrey-score auction, which is the first multi-object multi-dimensional auction that achieves both truthfulness and efficiency. Simulations with real traces show that crowdsourced mobile streaming outperforms noncooperative streaming by 48.6% (on average) in terms of social welfare. We further implement our proposed auction mechanism in a demostration system, and show that the crowdsourced framework together with the auction mechanism can substantially increase mobile user's welfare and video service stability.

The coarse-grained reconfigurable architecture (C-GRA) is a promising platform that provides both high performance and high power-efficiency. Dataflow graph (DFG) mapping is critical to tap the potentials of CGRAs. Inspired from the great progress made in tree search game using deep neural network, we proposed a frame work for learning convolutional neural network for mapping DFGs onto spatial programmable CGRAs. Considering the mapping process, we present a dual-input neural network capturing the features from both DFGs in applications and Process Element Array (PEA) in CGRA. In order to train the neural network, algorithms are designed to automatically generate a data set from PEA intermediate states of preprocessed DFG. Finally, experimental results demonstrate that our proposed mapping approach is competitive with state-of-the-art DFG mapping algorithms in performance while the compilation time is greatly reduced.

Driven by the exponentially increasing amount of mobile video traffic, caching videos closer to the end users has become an appealing solution to reduce the traffic through the backbone network while improving users' perceived quality-of-experience (e.g., better video quality and reduced service delay). This research interest has been gaining lots of momentums due to the emergence of smart Access Points (APs), which are equipped with large storage space (several GBs). To address the “small population” problem involved in the prefetching at the edge, we propose to prefetch videos to APs ahead of users' requests via tensor learning: We first adopt the weighted tensor model to mine the hidden semantic pattern to characterize both users' preference for different types of videos and the dynamic video popularity over time; Then, based on the resulting low-dimension matrixes generated by the tensor factorization, we adopt an exponential smoothing model to capture the temporal pattern to predict users' propensity to unwatched videos; Finally, based on the predicted video popularity, we proactively replicate videos from the original CDN server to the APs at the edge. Through trace-driven simulations, we show that the proposed prefetching solution can outperform the baseline algorithms: compared with the SVD-based prefetching strategy, our design achieves a better hit ratio (e.g., surpassing about 10%) and accuracy (e.g., surpassing about 15%); compared with the history based strategy, our design also have about 40% (resp. 20%) improvement in terms of hit ratio (resp. accuracy).

Mobile personal livecast (MPL) services are emerging and have received great attention recently. Unlike traditional livecast services with commercial content providers (e.g., live TV), the live contents in MPL are crowdsourced from and consumed among geo-distributed individuals. Although there exist typical social relationships in MPL (i.e., follower-followee), different from conventional social networking services like Twitter and Facebook, which have much of a tolerance for interaction delay, the interactions in MPL must be in real-time. These unique characteristics intrigue us to investigate how the relationships (e.g., social links and geo-locations) between viewers and broadcasters influence the user behaviors, which has yet to be explored in depth. In this paper, we carry out extensive measurements of Inke, one of the most popular MPL providers, with a large-scale dataset containing 11M users. Our key findings are as follows. First, compared with traditional livecast services, the user interests shift much more frequently and the average viewing duration is considerably shorter in MPL. Second, the existence of social relationships significantly strengthens viewer stickiness---followers dedicating longer viewing time (contributing 81% of the total viewing time) and being 2x more patient when suffering poor network connectivity than non-followers. Third, most of the broadcasts in MPL are geographically local-popular (the majority of the views come from the same region of the broadcaster). Based on these critical observations, we provide insights that can enhance the MPL system design from the perspectives of efficient resource allocation and envision a future MPL framework that collaboratively utilizes the cloud and edge computing resources to improve efficiency and scalability for Inke-like services.

Recently, Crowdsourced Interactive Live Streaming (CILS), such as Twitch.tv and Periscope, has emerged as one of the most popular streaming applications over the Internet. In such applications, a large number of geo-distributed users publish live sources to broadcast their game sessions, personal activities, and other events, while fellow viewers not only watch these live streams, but also contribute interactive messages to influence streaming content. Such explosively increasing popularity has posed significant challenges to predict viewing patterns using traditional time-series approaches, which lack the start/end knowledge of live streams and cannot capture the viewing burst very well.

This paper proposes a generic framework that enables a multiscale interaction in the cost aggregation step of stereo matching algorithms. Inspired by the formulation of image filters, we first reformulate cost aggregation from a weighted least-squares (WLS) optimization perspective and show that different cost aggregation methods essentially differ in the choices of similarity kernels. Our key motivation is that while the human stereo vision system processes information at both coarse and fine scales interactively for the correspondence search, state-of-the-art approaches aggregate costs at the finest scale of the input stereo images only, ignoring inter-consistency across multiple scales. This motivation leads us to introduce an inter-scale regularizer into the WLS optimization objective to enforce the consistency of the cost volume among the neighboring scales. The new optimization objective with the inter-scale regularization is convex, and thus, it is easily and analytically solved. Minimizing this new objective leads to the proposed framework. Since the regularization term is independent of the similarity kernel, various cost aggregation approaches, including discrete and continuous parameterization methods, can be easily integrated into the proposed framework. We show that the cross-scale framework is important as it effectively and efficiently expands state-of-the-art cost aggregation methods and leads to significant improvements, when evaluated on Middlebury, Middlebury Third, KITTI, and New Tsukuba data sets.

Mobile online social network services have seen rapid expansion; thus, the corresponding huge amounts of user-generated social media contents propagating between users via social connections have significantly challenged the traditional content delivery paradigm. First, replicating all the contents generated by users to edge servers that well “fit” the receivers becomes difficult due to limited bandwidth and storage capacities. Motivated by device-to-device (D2D) communication, which allows users with smart devices to transfer content directly, we propose replicating bandwidth-intensive social contents in a device-to-device manner. Based on large-scale measurement studies on social content propagation and user mobility patterns in edge-network regions, we observe the following: (1) Device-to-device replication can significantly help users download social contents from neighboring peers. (2) Both social propagation and mobility patterns affect how contents should be replicated. (3) The replication strategies depend on regional characteristics (e.g., how users move across regions). Using these measurement insights, we propose a propagationand mobility-aware content replication strategy for edge-network regions, in which social contents are assigned to users in edge-network regions according to a joint consideration of social graphs, content propagation, and user mobility. We formulate the replication scheduling as an optimization problem and design a distributed algorithm using only historical, local, and partial information to solve it. Trace-driven experiments further verify the superiority of our proposal: compared with conventional pure-movement-based and popularity-based approaches, our design can significantly improve (2 - 4-fold improvement) the amount of social content successfully delivered via device-to-device replication.

Group recommendation plays a significant role in today's social media systems, where users form social groups to receive multimedia content together and interact with each other, instead of consuming the online content individually. Limitations of traditional group recommendation approaches are as follows. First, they usually infer group members’ preferences by their historical behaviors, failing to captureinactiveusers’ preferences from the sparse historical data. Second, relationships between group members are not studied by these approaches, which fail to capture the inherent personality of members in a group. To address these issues, we propose a social-aware group recommendation framework that jointly utilizes both social relationships and social behaviors to not only infer a group's preference, but also model thetoleranceandaltruismcharacteristics of group members. Based on the observation that thefollowingrelationship in the online social network reflects common interests of users, we propose a group preference model based on external experts of group members. Furthermore, we model users’ tolerance (willingness to receive content not preferred) and altruism (willingness to receive content preferred by friends). Finally, based on the group preference model, we design recommendation algorithms for users under different social contexts. Experimental results demonstrate the effectiveness of our approach, which significantly improves the recommendation accuracy against traditional approaches, especially in the cases of inactive group members.

Popularly used to distribute a variety of multimedia content items in today's Internet, HTTP-based web content delivery still suffers from various content delivery failures, including server failures [1], network failures [2] and routing failures [3]. Hindered by the expensive deployment cost, the conventional CDN can not deploy as many edge servers as possible to successfully deliver content items to all users under these delivery failures. In this paper, we propose a joint CDN and peer-assisted web content delivery framework to address the delivery failure problem. Different from conventional peer- assisted approaches for web content delivery, which mainly focus on alleviating the CDN servers' bandwidth load, we study how to use a browser-based peer-assisted scheme, namely WebRTC, to resolve content delivery failures. To this end, we carry out large-scale measurement studies on how users access and view webpages. Our measurement results demonstrate the challenges (e.g., peers stay on a webpage extremely short) that can not be directly solved by conventional P2P strategies, and some important webpage viewing patterns (e.g., predictability of users' webpage viewing time). Due to these unique characteristics, WebRTC peers open up new possibilities for helping the web content delivery, coming with the problem of how to utilize the dynamic resources efficiently. We formulate the peer selection that is the critical strategy in our framework, as an optimization problem, and design a heuristic algorithm based on the mea- surement insights to solve it. Our simulation experiments driven by the traces from Tencent QZone, one of the most popular social service platforms in China, demonstrate the effectiveness of our design: compared with non-peer-assisted strategy and random peer selection strategy, our design significantly improves the successful relay ratio of web content items under network failures, e.g., our design improves the content download ratio up to 60% even when users located in a particular region (e.g., city) where none can connect to the regional CDN server.

In this work, we study the joint optimization of edge caching and data sponsoring for a video content provider (CP), aiming at reducing the content delivery cost and increasing the CP's revenue. Specifically, we formulate the joint optimization problem as a two-stage decision problem for the CP. In Stage I, the CP determines the edge caching policy (for a relatively long time period). In Stage II, the CP decides the real-time data sponsoring strategy for each content request within the period. We first propose a Lyapunov- based online sponsoring strategy in Stage II, which reaches 90% of the offline maximum performance (benchmark). We then solve the edge caching problem in Stage I based on the online sponsoring strategy proposed in Stage II, and show that the optimal caching policy depends on the aggregate user request for each content in each location. Simulations show that such a joint optimization can increase the CP's revenue by 30%↑100%, comparing with the purely data sponsoring (i.e., without edge caching).

Accurate mobility prediction is becoming increasingly important in human behavior research, mainly due to many location-based applications such as mobile social networks and mobile advertisements. In this work, we propose a new crowd-sourced human mobility prediction model for public regions. We first analyze human trajectories collected through a cluster of densely deployed Wi-Fi access points (AP) in a shopping mall, and then characterize the close relationship between the human mobility patterns and the spatio-temporal contexts. Based on the distinct features of human trajectories in different types of public regions, we further propose a Markov-based crowdsourced mobility prediction method utilizing spatio-temporal contexts. We evaluate the performance of the proposed method using real traces, and show that our method is 28% more accurate in predicting human location transitions and incurs 14% smaller error in stay time prediction than the baseline methods.

Recent years have witnessed a new video delivery paradigm: smartrouter-based video delivery network, which is enabled by smartrouters deployed at users' homes, together with the conventional video servers deployed in the datacenters. Recently, ChinaCache (a largest CDN provider) and Youku (a video service provider using smartrouters to assist video delivery) announced their cooperation to create a new paradigm of content delivery based on householders' network resources. This new paradigm is different from the conventional peer-to-peer (P2P) approach, because such dedicated smartrouters are inherently operated by the centralized video service provides in a coordinative manner. It is intriguing to study the strategies, performance and potential impact on the content delivery ecosystem of such peer CDN systems. In this paper, we study the Youku peer CDN, which has deployed over 300K smartrouter devices for its video streaming. In our measurement, 78K videos were investigated and 3TB traffic has been analyzed, over controlled peer nodes and players. Our contributions are the following measurement insights. First, a global replication and caching strategy is essential for the peer CDN systems, and proactively scheduling replication and caching on a daily basis can guarantee their performance. Second, such peer CDN deployment can itself form an effective QoS monitoring sub-system, which can be used for fine-grained user request redirection. We also provide our analysis on the performance issues and potential improvement to the peer CDN systems.

Stereoscopic image retargeting techniques aim to flexibly display 3D images with different aspect ratios and simultaneously preserve salient regions and comfortable depth perception. Various stereoscopic image retargeting techniques have been proposed recently. However, there is still no effective objective metric for visual quality assessment of retargeted stereoscopic images. In this paper, we build a stereoscopic image retargeting database and propose a learning-based objective method to evaluate the stereoscopic image retargeting quality. The perception quality of the database are evaluated by subjects. We extract new features of quality assessment and fuse them to assess stereoscopic image retargeting quality using neural network. Experiments conducted with above-mentioned database confirm the effectiveness of the proposed method. The results show the good consistency between the objective assessments and subjective rankings.

Recent years have witnessed a new video delivery paradigm: smartrouter-based peer video content delivery network, which is enabled by smartrouters deployed at users' homes. ChinaCache (one of the largest CDN providers in China) and Youku (a video provider using smartrouters to assist video delivery) announced their cooperation in 2015, to create a new paradigm of content delivery based on householders' network resources [2]. This new paradigm is different from the conventional peer-to-peer (P2P) approach, because millions of dedicated smartrouters are operated by the centralized video service providers in a coordinative manner. Thus it is intriguing to study the content placement strategies used in a smartrouter-based content delivery system, as well as its potential impact on the content delivery ecosystem. In this paper, we carry out measurement studies of Youku's peer video CDN, who has deployed over 300K smartrouter devices for its video delivery. In our measurement studies, 104K videos were investigated and 4TB traffic has been analyzed, over controlled smartrouter nodes and players. Our measurement insights are as follows. First, a global content replication strategy is essential for the peer CDN systems. Second, such peer CDN deployment itself can form an effective sub-system for end-to-end QoS monitoring, which can be used for fine-grained request redirection (e.g., user-level) and content replication. We also show our analysis on the performance limitations and propose potential improvements to the peer CDN systems.

Adaptive bitrate video streaming is a widely-used technology for mobile video streaming over HTTP. In this work, we study a crowdsourced video streaming framework, which enables nearby mobile users to crowdsource their radio resources for cooperatively adaptive bitrate video streaming. We propose a multi-dimensional auction based incentive mechanism to promote the user cooperation, supporting the asynchronous downloading and the bitrate adapting of video users. In this mechanism, each user initiates an auction whenever he is ready to download a new data segment in an asynchronous fashion, and all nearby users compete for the downloading opportunity by submitting a multidimensional bid consisting of the intended segment bitrate and the associated value. Design of such a multi-dimensional auction is very challenging, as we need to guarantee the user's truthful reporting on the information on multiple dependent dimensions. We first propose a truthful second-score (multi-dimensional) auction framework, within which we further derive the efficient mechanism that maximizes the social welfare (of each segment downloading) and the sub-optimal mechanism that approximately maximizes the auctioneer payoff. Experiment results show that our proposed crowdsourced streaming can achieve 60% ˜ 76% of the maximum social welfare even when 80 percentage of users lose their direct network connections.

Instant social video sharing which combines the online social network and user-generated short video streaming services, has become popular in today’s Internet. Cloud-based hosting of such instant social video contents has become a norm to serve the increasing users with user-generated contents. A fundamental problem of cloud-based social video sharing service is that users are located globally, who cannot be served with good service quality with a single cloud provider. In this paper, we investigate the feasibility of dispersing instant social video contents to multiple cloud providers. The challenge is that inter-cloud socialpropagationis indispensable with such multi-cloud social video hosting, yet such inter-cloud traffic incurs substantial operational cost. We analyze and formulate the multi-cloud hosting of an instant social video system as an optimization problem. We conduct large-scale measurement studies to show the characteristics of instant social video deployment, and demonstrate the trade-off between satisfying users with their ideal cloud providers, and reducing the inter-cloud data propagation. Our measurement insights of the social propagation allow us to propose a heuristic algorithm with acceptable complexity to solve the optimization problem, by partitioning a propagation-weighted social graph in two phases: a preference-aware initial cloud provider selection and a propagation-aware re-hosting. Our simulation experiments driven by real-world social network traces show the superiority of our design.

Knowing the Round-Trip Time (RTT) between a client and a server is important for an online interactive multi-media service to provide satisfactory quality of user experience. Due to the intrinsic dynamics of the topology and routing strategies in the Internet, it is however challenging to predict the RTT accurately from limited information, e.g., only the IP pair of the client and server. To address this challenge, we propose Guyot, a hybrid learning- and model-based approach to predict RTT, which requires significantly smaller amount of data to be collected than traditional approaches, while achieving a similar prediction accuracy. Our design is based on a large-scale measurement study from a content provider's perspective. Based on an information gain analysis, we design a hybrid RTT prediction approach involving two types of predictions: (1) Learning-based prediction: We train a decision tree to predict RTT between IP pairs with large geographic distance, requiring only a small set of features to be collected. (2) Model-based prediction: We use a model-based framework to predict RTT between IP pairs with small distance, providing an accurate RTT prediction over time. By strategically dividing RTT prediction tasks to these two types according to the distance of the inferred geo-locations of the IPs, our prediction approach can scale with satisfactory accuracy. Our experiments further confirm the superiority of our design.

Popularly used to distribute a variety of multimedia contents in today's Internet, HTTP-based web content delivery still suffers from failures occurring both in the network and at the servers. Limited by the deployment of edge servers, it is hard for a CDN to successfully deliver contents to all users under these network failures. Different from conventional peer-assisted approaches for web content delivery, which mainly focus on alleviating the CDN servers' bandwidth load, we study how to use a browser-based peer-assisted scheme to resolve content delivery failures. Based on large-scale measurement studies on how users access and view webpages, we observe the challenges (e.g., peers stay on a webpage extremely short) that cannot be directly solved by conventional P2P strategies, and some important webpage viewing patterns. Due to these unique characteristics, WebRTC peers give us a novel way to deliver web content, coming with the problem of how to utilize the dynamic resources efficiently. We formulate the peer selection as an optimization problem, and design a heuristic algorithm based on the measurement insights to strategically select the most appropriate peer. Our prototype implementation on Tencent QZone and simulation experiments further demonstrate the effectiveness of our design: compared with non-peer-assisted strategy and random peer selection strategy, our design significantly improves the successful relay ratio of web contents under network failures, e.g., our design achieves a successful content download ratio of 60% even when users located in a particular region are not able to connect to the CDN servers.

The stunning increase of video streaming has been a major part of the network flow in the past few years. It is essential for content providers to manage more flow servers to satisfy the demands of users. Therefore, to effectively manage large-scale servers and to identify problems, service node become crucial to guarantee video user experience. The rule-based approach that the traditional service providers use is effective but could hardly be applied to large-scale server clusters, let alone in a joint service resources and the maximization of user experience. Unlike traditional rule-based QoS monitoring platform, we design a framework on a basis of user experience metrics to detect the quality of service. In line with the characteristics of IP addresses, we also design a method to quickly pinpoint the fault location.

We propose a disparity-constrained retargeting method for stereoscopic 3D video, which simultaneously resizes a binocular video to a new aspect ratio and remaps the depth to the perceptual comfort zone. First, we model distortion energies to prevent important video contents from deforming. Then, to maintain depth mapping stability, we model disparity variation energies to constraint the disparity range both in spatial and temporal domains. The last component of our method is a non-uniform, pixel-wise warp to the target resolution based on these energy models. Using this method, we can process the original stereoscopic video to generate new, high-perceptual-quality versions at different display resolutions. For evaluation, we conduct a user study; we also discuss the performance of our method.

Dynamic adaptive streaming has emerged as a popular approach for video services in today's Internet. To date, the two important components in dynamic adaptive streaming, video transcoding that generates the adaptive bitrates of a video and video delivery that streams the videos to users, have been separately studied, resulting in a huge waste of computation and storage resource due to producing and caching different versions of videos regardless of their demands. We conduct extensive measurement studies of video sharing systems, including an IPTV service which streams regular, professionally made videos and an instant video clip sharing service which provides extremely short user-generated videos, as well as the availability of computation resource in conventional content delivery networks (CDNs). Based on the measurement insights, we propose an online joint transcoding and delivery approach for adaptive video streaming. We formulate optimization problems to enable high streaming quality for the users, and low computation and replication costs for the system. In particular, our strategy connects video transcoding and video delivery based on users' preferences of CDN regions and regional preferences of video versions. We analyze hardness of these problems and design distributed solutions. Extensive trace-driven experiments further demonstrate the superiority of our design.

There is an unprecedented trend that content providers (CPs) are building their own content delivery networks (CDNs) to provide a variety of content services to their users. By exploiting powerful CP-level information in content distribution, these CP-built CDNs open up a whole new design space and are changing the content delivery landscape. In this paper, we adopt a measurement-based approach to understanding why, how, and how much CP-level intelligences can help content delivery. We first present a measurement study of the CDN built by Tencent, a largest content provider based in China. We observe new characteristics and trends in content delivery which pose great challenges to the conventional content delivery paradigm and motivate the proposal of CPCDN, a CDN powered by CP-aware information. We then reveal the benefits obtained by exploiting two indispensable CP-level intelligences, namely context intelligence and user intelligence, in content delivery. Inspired by the insights learnt from the measurement studies, we systematically explore the design space of CPCDN and present the novel architecture and algorithms to address the new content delivery challenges that have arisen. Our results not only demonstrate the potential of CPCDN in pushing content delivery performance to the next level, but also identify new research problems calling for further investigation.

Online microblogging has been very popular in today's Internet, where users follow other people they are interested in and exchange information between themselves. Among these exchanges, video links are a representative type on a microblogging site. The impact is fundamental-not only are viewers in a video service directly coming from the microblog sharing and recommendation, but also are the users in the microblogging site representing a promising sample to all the viewers. It is intriguing to study a proactive service deployment for such videos, using the propagation patterns of microblogs. Based on extensive traces from Youku and Tencent Weibo, a popular video sharing site and a favored microblogging system, we explore how video propagation patterns in the microblogging system are correlated with video popularity on the video sharing site. Using influential factors summarized from the measurement studies, we further design a neural network-based learning framework to predict the number of potential viewers and their geographic distribution. We then design proactive video deployment algorithms based on the prediction framework, which not only determines the upload capacities of servers in different regions, but also strategically replicates videos to these regions to serve users. Our PlanetLab-based experiments verify the effectiveness of our design.

Human beings process stereoscopic correspondence across multiple scales. However, this bio-inspiration is ignored by state-of-the-art cost aggregation methods for dense stereo correspondence. In this paper, a generic cross-scale cost aggregation framework is proposed to allow multi-scale interaction in cost aggregation. We firstly reformulate cost aggregation from a unified optimization perspective and show that different cost aggregation methods essentially differ in the choices of similarity kernels. Then, an inter-scale regularizer is introduced into optimization and solving this new optimization problem leads to the proposed framework. Since the regularization term is independent of the similarity kernel, various cost aggregation methods can be integrated into the proposed general framework. We show that the cross-scale framework is important as it effectively and efficiently expands state-of-the-art cost aggregation methods and leads to significant improvements, when evaluated on Middlebury, KITTI and New Tsukuba datasets.

Dynamic adaptive video streaming has emerged as a popular approach for video streaming in today's Internet. To date the two important components in dynamic adaptive streaming, video transcoding which generates the adaptive bitrates of a video and video delivery which streams the videos to users, have been separately studied, resulting in a huge waste of computation and storage resource due to transcoding useless videos and suboptimal streaming quality due to homogeneous video replication. In this paper, we propose to jointly perform video transcoding and video delivery for adaptive streaming in an online manner. We conduct extensive measurement studies of a video sharing system and a CDN to motivate our design. We formulate and solve optimization problems to enable high streaming quality for the users, and low computation and replication costs for the system. In particular, our design connects video transcoding and video delivery based on users' preferences of CDN regions and regional preferences of video versions. Extensive trace-driven experiments further confirm the superiority of our design.

In this demonstration, we present a novel cloud-based system for casual users, which enables real-time 3D reconstruction and interaction based on stereo smartphones. The client is an Android application that captures scene in real-world, displays reconstructed 3D model and supports interaction. The server receives captured data from client and reconstructs 3D model in real-time. While users capturing data, both the reconstructed scenes and the marked uncertainty regions are visualized on the phones. So, it allows users to continuously update the reconstructed model in real-time by interaction. Meanwhile our system is implemented based on smartphones, which are accessible to casual users, while many existing systems always require extra or expensive specialized equipment. Also, we proposed multiple innovative methods to achieve continuously repairing model through interaction and address the challenge brought by the phone hardware limitations.

Receiving video streams on mobile devices has become a norm rather than an exception on today's Internet --- the year 2013 witnessed a global mobile traffic growth of 70%, in which video traffic accounts for over 50%. There is however a big gap between the download bandwidth of mobile devices and the rapidly increasing bitrate of online videos. To address this problem from a system perspective, we design a social-aware cooperative streaming framework called DECOMOD, in which local friends serve as peers such that an aggregated bandwidth can be achieved to ensure a high streaming quality. Our contributions in designing and implementing this system are summarized as follows: (1) We propose a social-aware cooperative DASH (Dynamic Adaptive Streaming over HTTP) streaming framework that aggregates bandwidths from local friends, over a hybrid mobile network (i.e., WIFI and Bluetooth); (2) We design a local bitrate selection strategy that jointly considers the download speed and battery level; (3) We implement the system and experimentally evaluate its performance --- high streaming quality and low freezing buffer time are provided in double to viewers.

We propose a method for local stereo matching that reduces the number of matching comparisons, which also reduces the running time. By using hierarchical structure with suitable disparity candidate set, the proposed method maintains accuracy of local matching methods. Theoretical analysis and experimental results show that the number of comparisons of the proposed method is independent of the disparity range, while the accuracy is nearly the same as the corresponding local method.

Social TV allows people to meet the demand of social experience while watching videos. Existing solutions cannot balance interest among group and provide good viewing quality for all members with different network bandwidth and devices. Realizing that people in group will be more tolerant, trusting and willing to help each other, we developed a cloud based social TV for online group video service. We emphasize group behavior in our application and introduce the concept of tolerance and trust between users to balance their interest. Group Recommendation results we show in our system can obtain overall high satisfaction while provides diverse content. We also design a collaborative video distribution technology to optimize network resource utility for fluency and high quality viewing experience. The entire operation of the system emphasizes on simplicity, collaboration and smooth, providing excellent social TV experience.

Online social network is emerging as a promising alternative for users to directly access video contents. By allowing users to import videos and re-share them through the social connections, a large number of videos are available to users in the online social network. The rapid growth of the user-generated videos provides enormous potential for users to find the ones that interest them; while the convergence of online social network service and online video sharing service makes it possible to perform recommendation using social factors and content factors jointly. In this paper, we design a joint social-content recommendation framework to suggest users which videos to import or re-share in the online social network. In this framework, we first propose a user-content matrix update approach which updates and fills in cold user-video entries to provide the foundations for the recommendation. Then, based on the updated user-content matrix, we construct a joint social-content space to measure the relevance between users and videos, which can provide a high accuracy for video importing and re-sharing recommendation. We conduct experiments using real traces from Tencent Weibo and Youku to verify our algorithm and evaluate its performance. The results demonstrate the effectiveness of our approach and show that our approach can substantially improve the recommendation accuracy.

Online video sharing and social networking are cross-pollinating rapidly in today's Internet: Online social network users are sharing more and more media contents among each other, while online video sharing sites are leveraging social connections among users to promote their videos. An intriguing development as it is, the operational challenge in previous video sharing systems persists, em i.e., the large server cost demanded for scaling of the systems. Peer-to-peer video sharing could be a rescue, only if the video viewers' mutual resource contribution has been fully incentivized and efficiently scheduled. Exploring the unique advantages of a social network based video sharing system, we advocate to utilize social reciprocities among peers with social relationships for efficient contribution incentivization and scheduling, so as to enable high-quality video streaming with low server cost. We exploit social reciprocity with two give-and-take ratios at each peer: (1) peer contribution ratio (em PCR), which evaluates the reciprocity level between a pair of social friends, and (2) system contribution ratio (em SCR), which records the give-and-take level of the user to and from the entire system. We design efficient peer-to-peer mechanisms for video streaming using the two ratios, where each user optimally decides which other users to seek relay help from and help in relaying video streams, respectively, based on combined evaluations of their social relationship and historical reciprocity levels. Our design achieves effective incentives for resource contribution, load balancing among relay peers, as well as efficient social-aware resource scheduling. We also discuss practical implementation and implement our design in a prototype social media sharing system. Our extensive evaluations based on PlanetLab experiments verify that high-quality large-scale social media sharing can be achieved with conservative server costs.

Online social networks have reshaped how multimedia contents are generated, distributed, and consumed on today's Internet. Given the massive number of user-generated contents shared in online social networks, users are moving to directly access these contents in their preferred social network services. It is intriguing to study the service provision of social contents for global users with satisfactory quality of experience. In this article, we conduct large-scale measurement of a real-world online social network system to study the social content propagation. We have observed important propagation patterns, including social locality, geographical locality, and temporal locality. Motivated by the measurement insights, we propose a propagation-based social-aware delivery framework using a hybrid edge-cloud and peer-assisted architecture. We also design replication strategies for the architecture based on three propagation predictors designed by jointly considering user, content, and context information. In particular, we design a propagation region predictor and a global audience predictor to guide how the edge-cloud servers backup the contents, and a local audience predictor to guide how peers cache the contents for their friends. Our trace-driven experiments further demonstrate the effectiveness and superiority of our design.

Social applications represent a paradigm shift on how the Internet is to be used, and have already changed the way we work, live, and play. When it comes to deploying social applications, cloud computing platforms are used to meet the Internet-scale, self-propagating, and fast-growing demands from these applications. Yet, to deploy social media applications in the most effective and economic fashion, we need to strategically design and follow a set of theoretical and practical principles. In this paper, we seek to design a set of new principles to guide social application deployment. Learning from large-scale measurement-based observations using a real-world social application, the gist of our principles is to detach the typically integrated "collection → processing → distribution" work ows in social applications into separate local processing and global distribution procedures, which can be effectively deployed using different cloud services. Moreover, based on a predictive model of regional propagation, we formulate the resource allocation problems in the processes of collecting/processing and distributing content as two optimization problems, which can be solved by efficient algorithms. Finally, based on our theoretical design, we have implemented an example social application on Amazon EC2 and Google AppEngine, where IaaS-based computation instances perform content collection and processing, and the PaaS-based platform is employed to distribute the contents that are widely propagating. Our PlanetLab-based trace-driven experiments have further confirmed the superiority of our design.

The fast development of time-of-flight (ToF) cameras in recent years enables capture of high frame-rate 3D depth maps of moving objects. However, the resolution of depth map captured by ToF is rather limited, and thus it cannot be directly used to build a high quality 3D model. In order to handle this problem, we propose a novel joint example-based depth map super-resolution method, which converts a low resolution depth map to a high resolution depth map, using a registered high resolution color image as a reference. Different from previous depth map SR methods without training stage, we learn a mapping function from a set of training samples and enhance the resolution of the depth map via sparse coding algorithm. We further use a reconstruction constraint to make object edges sharper. Experimental results show that our method outperforms state-of-the-art methods for depth map super-resolution.

Group recommendation plays a significant role in Social TV systems, where online friends form into temporary groups to enjoy watching video together and interact with each other. Online microblogging systems introduce the "following" relationship that reflects the common interests between users in a group and external representative followees outside the group. Traditional group recommendation only considers internal group members' preferences and their relationship. In our study, we measure the external followees' impact on group interest and establish group preference model based on external experts' guidance for group recommendation. In addition, we take advantage of the current watching video to improve context-aware recommendations. Experimental results show that our solution works much better in situations of high group dynamic and inactive group members than traditional approaches.

Online microblogging has been very popular in today's Internet, where users exchange short messages and follow various contents shared by people that they are interested in. Among the variety of exchanges, video links are a representative type on a microblogging site. More and more viewers of an Internet video service are coming from microblog recommendations. It is intriguing research to explore the connections between the patterns of microblog exchanges and the popularity of videos, in order to potentially use the propagation patterns of microblogs to guide proactive service deployment of a video sharing system. Based on extensive traces from Youku and Tencent Weibo, a popular video sharing site and a favored microblogging system in China, we explore how patterns of video link propagation in the microblogging system are correlated with video popularity on the video sharing site, at different times and in different geographic regions. Using influential factors summarized from the measurement studies, we further design neural network-based learning frameworks to predict the number of potential viewers of different videos and the geographic distribution of viewers. Experiments show that our neural network-based frameworks achieve better prediction accuracy, as compared to a classical approach that relies on historical numbers of views. We also briefly discuss how proactive video service deployment can be effectively enabled by our prediction frameworks.

Online social network has reshaped the way how video contents are generated, distributed and consumed on today's Internet. Given the massive number of videos generated and shared in online social networks, it has been popular for users to directly access video contents in their preferred social network services. It is intriguing to study the service provision of social video contents for global users with satisfactory quality-of-experience. In this paper, we conduct large-scale measurement of a real-world online social network system to study the propagation of the social video contents. We have summarized important characteristics from the video propagation patterns, including social locality, geographical locality and temporal locality. Motivated by the measurement insights, we propose a propagation-based social-aware replication framework using a hybrid edge-cloud and peer-assisted architecture, namely PSAR, to serve the social video contents. Our replication strategies in PSAR are based on the design of three propagation-based replication indices, including a geographic influence index and a content propagation index to guide how the edge-cloud servers backup the videos, and a social influence index to guide how peers cache the videos for their friends. By incorporating these replication indices into our system design, PSAR has significantly improved the replication performance and the video service quality. Our trace-driven experiments further demonstrate the effectiveness and superiority of PSAR, which improves the local download ratio in the edge-cloud replication by 30%, and the local cache hit ratio in the peer-assisted replication by 40%, against traditional approaches.

To recover a sharp version from a blurred image is a long-standing inverse problem. In this paper, we analyze the research on this topic both theoretically and experimentally through three paradigms: 1) the deterministic filter; 2) Bayesian estimation; and 3) the conjunctive deblurring algorithm (CODA), which performs the deterministic filter and Bayesian estimation in a conjunctive manner. We point out the weaknesses of the deterministic filter and unify the limitation latent in two kinds of Bayesian estimators. We further explain why the CODA is able to handle quite large blurs beyond Bayesian estimation. Finally, we propose a novel method to overcome several unreported limitations of the CODA. Although extensive experiments demonstrate that our method outperforms state-of-the-art methods with a large margin, some common problems of image deblurring still remain unsolved and should attract further research efforts.

Human action, as the basic unit of most human-relevant video content, bridges the gap between low-level visual features and high-level semantics. Human action recognition is of great significance in the applications of human-computer interaction, intelligent video surveillance, video retrieval and search. In this paper, we propose a novel unsupervised approach to mining categories from action video sequences, which consists of two modules: action representation for video data structurization and learning model for unsupervised categorization. In action representation, a novel view of video decomposition is presented. Videos are regarded as spatially distributed dynamic pixel time series, and these dynamic pixels are first quantized into pixel prototypes. After replacing the pixel time series with their corresponding prototype labels, the video sequences are compressed into two-dimensional action matrices. In the learning model, we put these matrices together to form an multi-action tensor, and propose the joint matrix factorization method to simultaneously cluster the pixel prototypes into pixel signatures, and matrices into action classes with the consideration of the duality between pixel clustering and action clustering. The approach is tested on public and popular Weizmann, and KTH datasets, and promising results are achieved.

Social influence has become the essential factor which drives the dynamic evolution process of social network structure and user behaviors. Previous research often focus on social influence analysis in network-level or topic-level. In this paper, we concentrate on predicting item-level social influence to reveal the users' influences in a more fine-grained level. We formulate the social influence prediction problem as the estimation of a user-post matrix, where each entry in the matrix represents the social influence strength the corresponding user has given the corresponding web post. To deal with the sparsity and complex factor challenges in the research, we model the problem by extending the probabilistic matrix factorization method to incorporate rich prior knowledge on both user dimension and web post dimension, and propose the Probabilistic Hybrid Factor Matrix Factorization (PHF-MF) approach. Intensive experiments are conducted on a real world online social network to demonstrate the advantages and characteristics of the proposed method.

Online games and social networks are cross-pollinating rapidly in today's Internet: Online social network sites are deploying more and more games in their systems, while online game providers are leveraging social networks to power their games. An intriguing development as it is, the operational challenge in the previous game persists, i.e., the large server operational cost remains a non-negligible obstacle for deploying high-quality multi-player games. Peer-to-peer based game network design could be a rescue, only if the game players' mutual resource contribution has been fully incentivized and efficiently scheduled. Exploring the unique advantage of social network based games (social games), we advocate to utilize social reciprocities among peers with social relationships for efficient contribution incentivization and scheduling, so as to power a high-quality online game with low server cost. In this paper, social reciprocity is exploited with two give-and-take ratios at each peer: (1) peer contribution ratio (PCR), which evaluates the reciprocity level between a pair of social friends, and (2) system contribution ratio (SCR), which records the give-and-take level of the player to and from the entire network. We design efficient peer-to-peer mechanisms for game state distribution using the two ratios, where each player optimally decides which other players to seek relay help from and help in relaying game states, respectively, based on combined evaluations of their social relationship and historical reciprocity levels. Our design achieves effective incentives for resource contribution, load balancing among relay peers, as well as efficient social-aware resource scheduling. We also discuss practical implementation concerns and implement our design in a prototype online social game. Our extensive evaluations based on experiments on PlanetLab verify that high-quality large-scale social games can be achieved with conservative server costs.

Recent development of depth sensors has facilitated the progress of 2D-plus-depth methods for 3D video representation, for which frame-rate up-conversion (FRUC) of depth video is a critical step. However, due to the computational cost of state-of-the-art FRUC methods, real time applications of 2D-plus-depth is still limited. In this paper, we present a method of speeding up the FRUC of the depth video by treating it as part of a video coding process, combined with a novel color-mapping algorithm is adopted to improve the quality of temporal upsampling. Experiments show that the proposed systems saves up to 99.5% of the frame interpolation time, while achieving virtually identical reconstructed depth video as state-of-the-art methods.

Online social network has emerged as the most popular approach for people to directly access multimedia contents. Among these contents, video sharing is a challenging task due to the demand on a large amount of uplink bandwidth at the dedicated server. We leverage a P2P paradigm to alleviate the server to distribute shared videos. By investigating traces obtained from a popular online social network in China, we observe that users' preferences can be predicted. We design a user preference guided prefetching strategy to reduce video startup delays, enabling smooth playback. Simulation experiments show that our design achieves high prefetch accuracy and short startup delay with conservative storage and bandwidth capacities at peers.

People and information are two core dimensions in a social network. People sharing information (such as blogs, news, albums, etc.) is the basic behavior. In this paper, we focus on predicting item-level social influence to answer the question Who should share What, which can be extended into two information retrieval scenarios: (1) Users ranking: given an item, who should share it so that its diffusion range can be maximized in a social network; (2) Web posts ranking: given a user, what should she share to maximize her influence among her friends. We formulate the social influence prediction problem as the estimation of a user-post matrix, in which each entry represents the strength of influence of a user given a web post. We propose a Hybrid Factor Non-Negative Matrix Factorization (HF-NMF) approach for item-level social influence modeling, and devise an efficient projected gradient method to solve the HF-NMF problem. Intensive experiments are conducted and demonstrate the advantages and characteristics of the proposed method.

It attracts many researchers' attention to find a stereo matching algorithm both accurate and fast. Yoon and Kweon's Adaptive Support-Weight(ASW) method is supposed to be a very successful algorithm in both accuracy and speed. However, it is very time consuming for ASW to take a large number of pixels into consideration for computing a disparity. In this paper, we present a stereo matching algorithm based on the ASW method to solve this problem. We introduce a time efficient technique, partial sum, to reduce the information of pixels in a large window into a small one so that disparities can be computed using the large window of pixels with low time complexity. Our method can obtain better representation of local properties of a pixel than the ASW method. Experimental results show that our method produces more accurate disparity maps than ASW method.

As compared to live peer-to-peer (P2P) streaming, modern P2P video-on-demand (VoD) systems have brought much larger volumes of videos and more interactive controls to the Internet users. Nevertheless, the larger number of available videos and the flexibility of allowing users to jump back and forth in a video, have led to much fewer numbers of concurrent peers watching at a similar pace, that reduces the chance for collaborative chunk supply among peers and thus significantly increases the server bandwidth cost. Towards the ultimate goal of maximizing peer resource utilization, in this paper, we design effective strategies for both cross-channel and intra-channel collaborations in multi- channel P2P VoD systems, such that individual peer's resources, including download/upload bandwidths and the cache capacity, are effectively utilized to maximize the streaming qualities in all the channels. In particular, each peer actively and strategically determines the supply-and-demand imbalance in different channels, as well as that among different chunks within each video, makes use of its surplus download capacity to fetch chunks with the most need, and then serves those chunks using its idle upload bandwidth, all without impairing its own streaming quality. Our extensive trace-driven simulations show the effectiveness of our strategies in reducing the server cost while guaranteeing high streaming qualities in the entire system, even during extreme scenarios such as unexpected flash crowds.

Content-based video retrieval systems are desired to fast and accurately find the nearest-neighbors of user input examples from very large video datasets. This poses a great challenge since exhaustive and redundant computation of similarities is required. Cluster based index approaches can be used to address this problem, but the similarity computation and clustering methods for videos are very time-consuming, thus preventing it from indexing very large video datasets. In this paper, we propose the Hierarchical Frame Affinity Group (HFAG), which is a hierarchy of frame clusters built using affinity propagation (AP) method, to represent video clusters. Our proposed video similarity metric and AP method guarantee the high performance of forming HFAG. We then build the cluster-based index structure to support retrieval of the nearest-neighbors of video sequences. The experiments on real large video datasets prove the effectiveness and efficiency of our approach.

As compared to live peer-to-peer (P2P) streaming, modern P2P video-on-demand (VoD) systems have brought much larger volumes of videos and more interactive controls to the Internet users. As the increase of bitrate of the videos and the full VCR controls of P2P VoD, the behavior “buffering” motivates us to design different schedule and service strategies for peers, to improve the playback performance, and the alleviation of the dedicated streaming server, by making best use of the bandwidth and cache capacities of these buffering peers. In our design, peers strategically decide which segments in the video to download first, and which requests to serve first. We conduct extended simulations to evaluate the performance of the strategies, and the results show our design outperforms the conventional sequential scheme, with respect to improving the playback quality and reducing the server load.

In 3D video transmission, the depth map is normally compressed by resolution reduction to save bandwidth. The lost information in resolution reduction is recovered by an appropriate upsampling algorithm in decoding step. Most previous work considers the depth upsampling problem as common 2D map upsampling problem and do not take the intrinsic property of depth map into consideration. In this paper, we propose a novel two-step depth map upsampling scheme to address the problem on 3D videos. The first step utilizes the full resolution 2D color map to direct the reconstruction of a more accurate full resolution depth map. And the second step further flats the reconstructed depth map to ensure its local uniformity. Test results show that the proposed novel upsam-pling scheme achieves up to 2dB coding gains for the rendering of free-viewpoint video, and improves its perceptual quality significantly.

Delivering high-quality multiview video (MV) through Internet is very challenging due to its excessive consumption on server bandwidth resources. Existing solutions encode video contents independently for each view and deliver them separately in isolated view channels, without leveraging the features of MV and taking advantage of multiview video coding(MVC). To minimize the server bandwidth costs, we introduce a novel overlay collaboration framework that unifies all view channels to cooperate in delivering MV: I pictures in MVC are shared among them instead of requesting from server respectively; Surplus resources of hotspot view channels are effectively utilized to help channels with insufficient resources, both of which contribute to remarkable reduction in server bandwidth costs. Simulation experiments show that our method achieves more than 40% reduced bandwidth costs on server while maintaining scalability and resilience to user dynamics.

In this paper, we propose a frame-level heuristic scheduling parallel emerging Multi-view Video Coding (MVC) using Directed Acyclic Graph (DAG) on Intel multi-core processor. We illustrate the reason to choose heuristic scheduling and formulate the problem. Through defining dependent degree and concurrent degree, we demonstrate why to choose frame as parallel granularity. Experimental results demonstrate the effectiveness and scalability of our parallel MVC.

Kernel estimate errors and image noise are major causes of visual artifacts in image motion deblurring. We propose an inter-scale non-blind image motion deblurring approach that significantly reduces those artifacts. We use Gaussian Scale Mixture Field of Experts (GSM FOE) model as image prior. The inter-scale smoothness constraint is adopted to suppress the ringing artifacts. In each scale, image details are recovered by the residual deconvolution and the cross bilateral filter (CBF). We further propose a std-controlled CBF to denoise the result. The experimental results are much better than those of previous methods.

Traditional non-blind motion deblurring methods are sensitive to kernel estimate errors and image noise, thus suffering from either ringing artifacts, enlarged image noise, or over-smoothed image details. We introduce a robust non-blind deblurring algorithm that produces high quality results even from many challenging images with noisy kernels. We adopt the Gaussian Scale Mixture Fields of Experts (GSM FOE) model and the smoothness constraint as image prior, and use the iterative re-weight least-square (IRLS) algorithm to produce the temporal result. The residual deconvolution suite is used to restore the lost image details. We denoise the result using our std-controlled cross bilateral filter. The experimental results are much better than those of previous approaches.

Ray tracing is a widely used rendering technique in computer graphics, and its intense computational requirement prohibits real-time ray tracing applications wide use in consumer markets. One main feature of ray tracing is parallelism and the mainstream computer market is switching to systems with multi-core. In this paper, in order to accelerate ray tracing to achieve real-time processing, we propose a data parallel kd-tree ray tracing algorithm on Cell Broadband Engine (Cell/B.E.) which is a state-of-the-art multi-core processor. This paper expounds the feasibility and key issues of kd-tree ray tracing on the Cell/B.E. processor and introduces the implementation and Cell-specific in ray tracing program design. The results highlight that our parallel algorithm for kd-tree ray tracing is scalable with a matrix of cores, resolutions or different object models. The execution time reduces from several minutes to several seconds.

Multiview video is well known to involve interactions with audience and offer better view experience than conventional single-view video. The interactive nature of multiview video has made the service very delay-sensitive and thus imposes great challenges for providing multiview streaming service in Internet. Currently, almost none of existing works have addressed the delay issue in multiview streaming. To ensure the quality of service and offer preferable view experience for users, we propose a novel streaming framework to provide delay-guaranteed service for interactive multiview video. The basic tradeoff between consumed server bandwidth and the required delay is carefully studied. We leverage the features of MVC to keep the server bandwidth costs low while guarantee the delay constraint. In addition, we introduce a neighbor-assisted view switching scheme: peer's neighbors are involved in its switching process and their upload bandwidth resources are effectively utilized to reduce the bandwidth costs on server. Simulation results show that our proposed framework meet the delay-guaranteed requirement with restrained server bandwidth costs while maintaining scalability and resilience to users dynamics.

This paper presents a head-shoulder detection approach using cascade SVM and histograms of oriented gradients (HOG). The HOG features which are extracted from variable-size blocks can capture salient features of head-shoulder automatically. A two stage cascade using SVM approach is designed to be the classifier. During detection, the majority of negative windows are rejected at the first stage, leaving a relatively small number of windows to be classified at the second stage, which improves the speed and precision of the detector. Due to the large number of possible target locations in an image, we applied camera self-calibration approach to facilitate the estimation for the size and location of the detection window. The experiments on surveillance videos from Trecvid 2008 proved that our approach can achieve fast and accurate head-shoulder detection.

In this paper, we study the properties of the bilateral pyramid, based on which a novel image representation-the bilateral wavelet pyramid (BWP) is proposed. The BWP provides a multiresolution technique to manipulate image variances simultaneously in radiometric, spatial and frequency domains. Application to texture retrieval is discussed to demonstrate the effectiveness of the BWP.

Recently, there has been a great deal of interest in statistics of natural images in linear domain. However, a detail study of image statistics in non-linear domain on a very large dataset is still missing. In this paper, we analyze natural images' statistics in ldquobilateral domainrdquo on different sub bands which are generated by the classical bilateral filter. Compared with those in linear domain, the statistics in bilateral domain are significantly superior in generalization, de-correlation, and local feature extraction. We also observe some entirely new but interesting features.

We argue that the conventional forward error correcting code (FEC) and retransmission solutions will become ineffective to transmit scalable video coding streaming in wireless mesh network (WMN) for the reason that video frame errors appear in bursts rather than random and the bad channel condition usually persists for a period in wireless channels. A novel SVC streaming cooperative transmission scheme is proposed in this paper. Our cooperative transmission scheme includes two stages among nodes in WMN. The first stage is selecting appropriate paths for the relevant SVC sub-layer streams and the second one is selecting nearby nodes to cache the network coding results of the adjacent sub-layer streams and retransmit the network coded packets to the user terminal for recovering the error packets. Simulation results are given to demonstrate the significant improvement of the SVC streaming reconstructive video quality.

In this paper, we propose a novel automatic algorithm for foreground/background labeling. We aim to generate ROI cutout automatically for further processing such as image editing, classification and information retrieval. Different from traditional semi-supervised segmentation method, we use a rather weak prior on boundary label. Accordingly, a global cost function is proposed to combine our prior knowledge with pixel-level feature. We compute fuzzy matting components as building blocks to construct semantically meaningful mattes. Finally, these mattes are hierarchically clustered and ranked by central preference. Experimental results on a large benchmark data set demonstrate the performance of our algorithm.

Background subtraction in highly dynamic scenes has been a critical challenge for traditional pixel-wise background models which perform poorly when the background has dynamic textures. In this paper, we consider background modelling in a spatial perspective and make an attempt to exploit more information from the outputs of pixel-wise model. We propose a background subtraction scheme using adaptive spatial fusing to refine the output of typical pixel-wise background model - mixture of Gaussians (MoG) and employ a MRF-MAP scheme to make foreground-background classification using the spatial correlation. Experiments on several challenge sequences show that our method is able to yield significantly better results than the traditional ones and is compelling with existing state of the art background subtraction algorithms. Additionally, we proved our algorithm has linear running time complexity and any pixel-wise background model could be easily integrated into our spatial fusing scheme which greatly enhanced its scalabilities and applications.

In this paper, using the Intel multicore architectures and the emerging multiview video coding standard, we introduce a framework for performing analysis, simulation, and evaluation of heuristics scheduling algorithms for implementing computationally intensive algorithms on multicore processors. The framework allows for accurate and quantitative characterization of the performance of dynamic scheduling algorithms for multimedia applications on different multicore processors without actual implementation of the scheduling algorithm and application on the actual platform. Experimental results demonstrate the effectiveness and scalability of our framework.

Multiple Object Tracking (MOT) poses three challenges to conventional well-studied Single Object Tracking (SOT) algorithms: 1) Multiple targets lead the configuration space to be exponential to the number of targets; 2) Multiple motion conditions due to multiple targets' entering, exiting and intersection make the prediction process degrade in precision; 3) Visual ambiguities among nearby targets make the trackers error prone. In this paper, we address the MOT problem by embedding contextual proposal distributions and contextual observation models into a mixture tracker which is implemented in a Particle Filter framework. The proposal distributions are adaptively selected by motion conditions of targets which are determined by context information, and the multiple features are combined according to their discriminative power between ambiguity prone objects. The induction of contextual proposal distribution and observation model can help to surmount the incapability of conventional mixture tracker in handling object occlusions, meanwhile retain its merits of flexibility and high efficiency. The final experiments show significant improvement in variable number objects tracking scenarios compared with other methods.

During recent years, the Internet has witnessed a rapid growth in deployment of data-driven (or swarming based) peer-to-peer (P2P) media streaming. In these applications, each node independently selects some other nodes as its neighbors (i.e. gossip-style overlay construction), and exchanges streaming data with the neighbors (i.e. data scheduling). To improve the performance of such protocol, many existing works focus on the gossip-style overlay construction issue. However, few of them concentrate on optimizing the streaming data scheduling to maximize the throughput of a constructed overlay. In this paper, we analytically study the scheduling problem in data-driven streaming system and model it as a classical min-cost network flow problem. We then propose both the global optimal scheduling scheme and distributed heuristic algorithm to optimize the system throughput. Furthermore, we introduce layered video coding into data-driven protocol and extend our algorithm to deal with the end-host heterogeneity. The results of simulation with the real world traces indicate that our distributed algorithm significantly outperforms conventional ad hoc scheduling strategies especially in stringent buffer and bandwidth constraints.

Robust transmission of live video over ad hoc wireless networks presents new challenges: high bandwidth requirements are coupled with delay constraints; nodes are constrained in processing power and storage capacity; ad hoc wireless networks suffer from bursty packet losses that drastically degrade the viewing experience. Accordingly, we propose a simple, but practical unbalanced multiple description (UMD) codec that uses single path transport only. At the UMD decoder, the lost Low- Resolution (LR) frames can be receded from the corresponding ones of the correctly received High- Resolution (HR) stream. This makes LR more robust to packet losses and ensures continuous video playback. The simulation results show that the proposed UMD codec has higher decoded quality, smaller quality fluctuation and lower probability of pause than other error resilience techniques, especially when channel burstiness becomes large. We also propose a novel sequence-based error concealment (EC) algorithm for our UMD decoder. It recursively uses multi-frame recovery principle to frame-by-frame reduce error drift in the HR description with respect to the LR description. In fact, the algorithm can be applied to most UMD schemes. It has the advantage of working on integer-, half- or quarter- pixel precision and looking backward without buffering future frames. Experimental results show that error decreases the fastest if the UMD decoder uses our EC algorithm. It can provide satisfactory performance on both objective and subjective evaluation.

Almost all existing peer-to-peer (P2P) streaming systems can only provide non-interactive service on Internet. In this paper, we focus on providing delay-guaranteed service to support delay-tolerant interactive applications (such as online auction, person interview, video sharing & commenting, etc.) using P2P streaming technology. In an interactive channel, all or part of the participants can interact with the presenter or publisher at the source. Unlike existing P2P streaming service, there is different level of delay and synchronization requirement in the interactive applications, meanwhile, the participant number in each channel is relatively small but the concurrent channel number is large. With such challenges, how much server bandwidth can be saved in P2P streaming is an interesting problem. We propose a very practical protocol iGridMedia and fully implement it. The basic tradeoff between consumed server bandwidth and the required delay is carefully studied. Simulation and real-world experiment show that our system consumes only about 4 times streaming rate with 5 sec guaranteed delay even if the peer churn rate is high. So far as we know, our work represents the state-of- the-art approach that using P2P technology to support emergent interactive live applications on Internet with delay-guaranteed requirement.

Internet videos have grown exponentially with the help from video sharing Websites. Automatic topic mining is therefore increasingly important for organizing and navigating such large video databases. Most of current solutions of topic detection and mining were done on news videos and cannot be directly applied on Web videos, because of their limited and noisy semantic information. In this paper, we will try to address this problem and propose an automatic topic mining framework on Web videos. We develop an iterative weight-updated co-clustering scheme to filter "noisy" tags and mine the "hot" topics. We then propose a visual-based clustering approach to further group the videos with similar content, and rank the visual-similar groups by their similarity to the topic center. Experiments on a large Web video database demonstrate the superior performance of our weight-updated co-clustering to both of the traditional co-clustering and k-means. The experiments also demonstrate significant improvement of users' experience by our visual-based clustering and ranking.

Isolated Distributed Stream Servers (IDSS) is the main form of video-on-demand (VOD) service architecture in industrial community nowadays. Contrast to the previous work on load sharing in distributed VOD system which all follows the idea of having high-speed inner network among service nodes, in this paper, we firstly focus on load sharing algorithm under IDSS architecture and introduce global stream distribution optimization, future user arrival rate estimation and more disk storage redundancy rate to make the effect of load sharing more satisfactory. Moreover the influence of video redundancy rate and some other parameters in our algorithm is detected through simulation. Preliminary experiment results suggest that our algorithm outperforms existing load sharing algorithms under IDSS architecture and sometimes even works better than existing ones with inner network.

In this paper, a novel unsupervised approach to mining categories from action video sequences is presented. This approach consists of two modules: action representation and learning model. Videos are regarded as spatially distributed dynamic pixel time series, which are quantized into pixel prototypes. After replacing the pixel time series with their corresponding prototype labels, the video sequences are compressed into 2D action matrices. We put these matrices together to form an multi-action tensor, and propose the joint matrix factorization method to simultaneously cluster the pixel prototypes into pixel signatures, and matrices into action classes. The approach is tested on public and popular Weizmann data set, and promising results are achieved.

Highlight detection is a challenge task in soccer video analysis. Using Web-casting text as external knowledge is proved to be a short cut to achieve both efficiency and effectiveness. Based on the previous framework using Web-casting text, we have improved the processes of video time detection and highlight boundary detection. Our method can detect the transparent time bar and can achieve acceptable precision in highlight boundary detection though the Web text time is not accurate at all. This progress can make the framework more robust in practice.

In this paper, we propose a novel scheme using computing complexity layered scalable video coding (CCLSVC) to optimize the user experience of broadcasting video in the computing capability limited handheld terminals. To address the heterogeneity of computing capability among different handheld devices, we employ hierarchal B reference structure of SVC to divide the frames into multiple computing complexity layers (CC Layers) in server side. The handheld clients simply choose to decode the frames in their corresponding layers in terms of their computation capability to maximize the video PSNR. We have proved that the optimal CC Layers division problem is a precedence constrained scheduling problem, which is an NP-complete problem. And we further propose our fast greedy method to approximately get optimized broadcasting video playback PSNR. The simulation shows that our method is superior to temporal SVC and random frame discarding method.

Automatic topic discovery and tracking on web-shared videos can greatly benefit both web service providers and end users. Most of current solutions of topic detection and tracking were done on news and cannot be directly applied on web videos, because the semantic information of web videos is much less than that of news videos. In this paper, we propose a bipartite graph model to address this issue. The bipartite graph represents the correlation between web videos and their keywords, and automatic topic discovery is achieved through two steps - coarse topic filtering and fine topic re-ranking. First, a weight-updating co-clustering algorithm is employed to filter out topic candidates at a coarse level. Then the videos on each topic are re-ranked by analyzing the link structures of the corresponding bipartite graph. After the topics are discovered, the interesting ones can also be tracked over a period of time using the same bipartite graph model. The key is to propagate the relevant scores and keywords from the videos of interests to other relevant ones through the bipartite graph links. Experimental results on real web videos from YouKu, a YouTube counterpart in China, demonstrate the effectiveness of the proposed methods. We report very promising results.

Automatic player detection, labeling and tracking in broadcast soccer video are significant while quite challenging tasks. In this paper, we present a solution to perform automatic multiple player detection, unsupervised labeling and efficient tracking. Players ’ position and scale are determined by a boosting based detector. Players ’ appearance models are unsupervised learned from hundreds of samples automatically collected by detection. Thereafter, these models can be utilized for player labeling (Team A, Team B and Referee). Player tracking is achieved by Markov Chain Monte Carlo (MCMC) data association. Some data driven dynamics are proposed to improve the Markov chain’s efficiency. The testing results on FIFA World Cup 2006 video demonstrate that our method can reach high detection and labeling precision, and reliably tracking in cases of scenes such as multiple player occlusion, moderate camera motion and pose variation. 1

In this paper we propose a technique to detect anomalies in individual and interactive event sequences. We categorize anomalies into two classes: abnormal event, and abnormal context, and model them in the Sequential Monte Carlo framework which is extended by Markov Random Field for tracking interactive events. Firstly, we propose a novel pixel-wise event representation method to construct feature images, in which each blob corresponds to a visual event. Then we transform the original blob-level features into subspaces to model probabilistic appearance manifolds for each event-class. With the probability of an observation associated with each event-class (or state) derived from probabilistic manifolds, and state transitional probability, the prior and posterior state distributions can be estimated. We demonstrate in experiments that the approach can reliably detect such anomalies with low false alarm rates.

The feature of rendering arbitrary view make layered depth image (LDI) be suitable to present multi-view video data and provide interactivity form such as free view video(FVV). However, visual artifacts occurred in the rendered result limit the practicality of LDI. In this paper, we proposed an approach to deal with this problem during generating of LDI, which refines colors of the depth pixels by choosing proper candidate depth pixels, removes matting effects by projecting LDI backward to reference views, and eliminate the gaps or holes due to disocclusion and undersample by local background interpolation. Experimental results show that our approach is practicable and efficient.

During recent years, the quick development of computer techniques has witnessed the ever-increasing surveillance video data, which essentially pose great challenge on the data storage, management, analysis and even retrieval. Considering that most of the high volume of data is with no interest, we mainly investigate the problem of effectively and efficiently discovering segments-of-interest (SoI) in this paper. To do so, we propose a novel event-oriented Sol discovery method in two steps: first, we represent an event by modeling pixels' change in temporal-spatial space, aiming to unify both the inter-frames and frames-background changes; second, with the benefit of unsupervised learning, the prototype-event models could be learned from these detected events and in turn exploited to measure the interest factor of each prototype-event. The experiment results demonstrate that the proposed method precisely discriminate different events and effectively discover Sols.

During recent years, the Internet has witnessed a rapid growth in deployment of peer to peer (P2P) based live media streaming systems. In this paper, we are motivated to investigate the problem of improving the streaming quality of service (QoS) of those systems by two universal recognitions: one is that the understanding to practical service experiences would benefit the performance enhancement, while the other is "the more, the better" design philosophy in P2P networks. As an interesting approach, we first retrieve the practical online duration information of end users from service traces in our GridMedia system, and then intentionally extend the online duration of each peer, which would result in a more stable peer overlay. The comparative simulations with original real-world traces and modified ones demonstrate that the quality of streaming service would be better if peers stay longer in the community. We claim that, this simple but positive result essentially validates both the influence of end users' behaviors and the need of an incentive to encourage users to spend more time in P2P live streaming systems.

Multi-view video introduces more forms of interactivity and can potentially be used for a variety of applications, such as free-view video (FVV) and three-dimensional video (3DTV). However, the huge volume video data and the synthesis of virtual view limit the practicality of multi-view video. In this paper, we propose an approach to multi-view video coding, in which the mutli-view video sequences are converted to layered depth images (LDIs) to represent scenes, and compressed layer by layer. To achieve better compression efficiency, we restructure layered images according depth data in LDIs. Experimental results show that our approach is practicable and efficient to meet the compression efficiency and flexible interactivity.

We propose a dynamic rate allocation scheme based on power-rate-distortion (PRD) optimization model among multiple video sources over ad hoc networks. This work is an extension of the PRD model for single source video streaming. With a total rate constraint and different power consumption constraints for each node, our optimization algorithm minimizes the average video distortion for all sources. The optimization is performed at the receiver of the video streams. Experimental results for a video surveillance scenario demonstrate that the proposed scheme outperforms a simple fixed-QP scheme. The proposed scheme promotes the average PSNR by 0.32-0.45 dB without shortening the system lifetime, or prolongs system lifetime by more than 20% without cutting down the overall PSNR.

Multi-view video coding technology is proposed to resolve the problem of huge data storage and transmission for free-view and 3D interactive video. How to support real time multi-view video encoding which has high computing complexity with sharply increased multi-view video data is essential In this paper, we proposed a solution of spatial and temporal data parallelization for multi-view video encoding algorithm based on IBM cell multiprocessor system using selections of optimal theories & methods. The performance of our tasks distributing scheme is eight times faster than the serial algorithm, speedup is notable.

Most of the real deployed peer-to-peer streaming systems adopt pull-based streaming protocol. In this paper, we demonstrate that, besides simplicity and robustness, with proper parameter settings, when the server bandwidth is above several times of the raw streaming rate, which is reasonable for practical live streaming system, simple pull-based P2P streaming protocol is nearly optimal in terms of peer upload capacity utilization and system throughput even without intelligent scheduling and bandwidth measurement. We also indicate that whether this near optimality can be achieved depends on the parameters in pull-based protocol, server bandwidth and group size. Then we present our mathematical analysis to gain deeper insight in this characteristic of pull-based streaming protocol. On the other hand, the optimality of pull-based protocol comes from a cost -tradeoff between control overhead and delay, that is, the protocol has either large control overhead or large delay. To break the tradeoff, we propose a pull-push hybrid protocol. The basic idea is to consider pull-based protocol as a highly efficient bandwidth-aware multicast routing protocol and push down packets along the trees formed by pull-based protocol. Both simulation and real-world experiment show that this protocol is not only even more effective in throughput than pull-based protocol but also has far lower delay and much smaller overhead. And to achieve near optimality in peer capacity utilization without churn, the server bandwidth needed can be further relaxed. Furthermore, the proposed protocol is fully implemented in our deployed GridMedia system and has the record to support over 220,000 users simultaneously online.

Video transmission over unreliable packet network is in general hampered by the packet losses and constraint by stringent playback deadline. With these two key factors in consideration, multiple description coding (MDC), comprising balanced and unbalanced MDC has been proposed as an error-robust source coding technique. Recently, transmitting multiple descriptions over a single path is interesting due to the unavailability of multiple independent paths. Therefore, in this paper, we investigate the problem of rate allocation for the high-resolution (HR) and low-resolution (LR) descriptions in UMDC transmission over single path. We first propose an approximate but efficient rate allocation model with the aid of two-state Markov link model and a simple distortion model at the sender side. Then we conduct extensive experiments to verity the proposed model and more excitedly the simulation results clearly demonstrate the effectiveness of proposed model

Video transmission over wireless ad hoc networks is hampered by packet losses. Even a single packet loss may cause error propagation until an intra-coded frame is received. Indeed, packet losses greatly degrade the video quality. In this paper, we propose an unbalanced multiple description coding (UMDC) scheme over a single path which requires only one single path as additional links are difficult to be guaranteed in reality and is capable of quickly recovering from packet losses and ensuring continuous playback. The proposed scheme uses two descriptions, the high-resolution (HR) description and the low-resolution (LR) one. It uses the 'peg frames' to limit error propagation in the HR description. The two descriptions can help each other recover from packet losses. The simulation results show that the proposed UMDC scheme over a single path has a comparable performance with our UMDC scheme with multiple path transmission (MPT) and has a better viewing experience than the state-of-the-art FEC-based scheme

The successful deployment of peer-to-peer (P2P) live video streaming systems has practically demonstrated that it can scale to reliably support a large population of peers. However, peers, representing rational end users, tend to be non-cooperative when it comes to the duty rather than the self-interests, running counter to the fundamental design philosophy of P2P concept. The objective of this paper is to investigate the problem of encouraging users to balance what they take from the system with what they contribute. We first make a statistical analysis to the service logs of a practical P2P live video streaming system and reveal intrinsic characteristic of users' online duration. Second, we thus propose a novel incentive mechanism based on the composite contributions which consist of two objective metrics, i.e. the on-line duration and effective upstream traffic. This mechanism offers service differentiation to users with different contributions and has some desirable properties: (1) distributed nature upon gossip-based overlay structure and (2) practical oriented evaluation criteria. The experiment results over PlanetLab further verify its effectiveness

Overlay network has emerged as a popular alternative to traditional client-server architecture for the distribution of media content. Intrinsic to the notion is a 'self-growing' community of end users. It is thus important to investigate practical issues in reality rather than following prescribed theoretical assumptions. This paper presents our research experiences on deployment of overlay network for live video streaming over Internet. In our implementation we adopt a gossip-based overlay construction protocol to accommodate topology changes and a composite scheduling mechanism to stream video contents. More importantly, with the service to totally more than 500,000 users and maximum 15,239 concurrent users provided by one common streaming server, we then exhibit insightful statistical results to reveal system performance and nodes properties in terms of capacity evolution, quality of service, request rate and online duration

The increasing use of mobile applications requires higher compression efficiency for video coding. However, in mobile applications, the capability of the processor is limited and the channel is unreliable. Since the emerging H.264/AVC video coding standard provides superior compression performance compared to all the existing video coding standards, many research efforts concentrate on the compression efficiency of intra-frame coding in H.264/AVC. In this paper, we propose a novel algorithm to reduce the bit rate compressed by H.264/AVC intra-frame coding. We focus on the method to code the intra-frame prediction modes. Our proposed method significantly reduces the bits used to code the modes and do not change the syntax of the bit stream. Experimental results demonstrate it is effective and efficient

In this paper, we emphasize salient experiences on peer to peer based live video streaming. First, we describe a two-layer overlay structure and a multi-sender based transmission algorithm to broadcast video programs. Although both numerical analysis and practical data reveal feasibility to support nearly 600 concurrent users in 500kbps streaming delivery, the tremendous workload imposed upon structure maintenance server limits its scalability. Therefore, in the second implementation, a gossip-based overlay protocol is deployed to provide distributed membership management, and an efficient scheduling mechanism is proposed to fetch video contents with lower delay. The experimental data from PlanetLab indicates that it ensures continuous playback, demanding bandwidth and low latency. More interestingly, the maximum concurrent users 15,239 with one common streaming server in 300kbps distribution sessions was tracked in Feb. 2005, which demonstrates that it offers a scalable and reliable streaming system for a large and highly dynamic population of end hosts

In this paper, we proposed a flexible video coding scheme for content-based video storage and retrieval. The proposed flexible video coding scheme applies shot clustering on the uncompressed video in the first pass and encodes the video in multi-clusters based on different video contents in the second pass. The compressed video can be flexibly decoding to meet the different requirements such as video indexing, video summary and structure extraction without fully decoding and re-analyzing. Also the flexible encoder can achieve global coding optimization by encoding similarity video shots continuously. It is very important to reduce the process complexity at the decoder and provide the flexible video content access on the compressed videos in many off-line video storage and retrieval applications. The experiments on sport and film videos illustrate the coding performance increase and the effect of the proposed coding scheme for content-based video storage.

The feature of rendering arbitrary view make layered depth image (LDI) be able to present multi-view video data and provide enhanced interactivity, such as free view video (FVV). However, visual artifacts occurred in the rendered result limit the practicality of LDI. In this paper, we proposed an approach to deal with this problem, which refines colors of the depth pixels by clustering to remove matting effects during generating of LDI, and applies median filter to rendered result to remove the gaps or holes caused by disocclusion and undersample. Experimental results show that our approach is practicable and efficient.

This paper presents an LSF-based framework for detecting arcs in broadcast soccer video. The successful identification of the arcs will evidently facilitate the soccer video analysis. The existing methods are not available for all playfield arcs including both the middle field circle and penalty box arcs. A new algorithm is proposed in the paper improved from LSF, called ALSF (advanced least square fitting), which can be used to detect arcs even though they are only 1/4 of eclipses (the same as penalty box arcs). With the improvement, we proposed a new framework to detect the arcs in broadcast soccer video. The proposed method first removes the points of straight lines. Then, all points of each connective area are transformed into a new reference frame to use LSF to get the equations of arcs. Experiments on more than 3 hours broadcast soccer video show the proposed method is effective with above 98% precision and 70% recall.

In this paper, we propose a method for two typical mid-level descriptors extraction of soccer video: view type and playfield position. The output reveals high-level structure of the soccer video. After the robust playfield segmentation, we use the grass-area-ratio and the boundary approximate lines between playfield and non-playfield to classify each frame into three kinds of view types according to a series of rules. We then classify frames captured by the main camera into fifteen kinds of positions by two steps: first, we classify the frames into five kinds according to the orientation of boundary lines; secondly, we use lines detection to further determine the specific playfield position. Experiments on three hours soccer videos show promising results.

In this paper, a new motion-compensated interpolation method (MCI) for frame rate up-conversion is proposed. In conventional MCI algorithms using transmitted motion vectors (MVs), the block artifacts are caused by the unreliable MVs from the encoder. In our proposed method, motion vectors correction (MVC) is applied on the transmitted MVs before interpolation considering the spatial correlation. In consideration of the temporal correlation, generative model is used to generate the temporal-predicted MVs. Then position prediction method is applied on the transmitted MVs and the temporal-predicted MVs separately to predict the positions the interpolated blocks really move to, which makes the MVs used for interpolation nearer to the true motion. The final MVs are selected from the two candidates of position prediction by the minimum sum of absolute difference (SAD). Applied to the H.264 decoder, our proposed method achieves significant increase on PSNR and obvious decrease of the block artifacts.

This paper proposed a novel video shot clustering algorithm using spectral method by joint modeling of inter and intra shot. Gauss Mixture Model (GMM) is used for probabilistic space-time modeling of intra-shot pixels. The spectral clustering method is applied on the GMM parameters. The problem of automatic model selection is currently an open issue for spectral method. Here we propose a novel automatic model selection based on joint inter-intra model optimization to achieve the global optimization in both model parameters and cluster numbers. We compare the proposed method with the conventional spectral method for sports video clustering. The simulation results show more accuracy on how many clusters and clustering results.

Currently, a large number of ALM algorithms have been proposed to build the overlay networks and transmit data through spanning trees over the overlay networks. However, few works focus on the congestion of spanning tree, the key to the performance of application. This paper studies the impact of spanning tree congestion on ALM performance and proposes a model to quantify and evaluate the congestion. A statistical link model is introduced to describe congestion occurring in ALM networks and a dependency-degree factor is presented to express the dependence between adjacent links. Furthermore, two schemes, the inter-level redundant links (INTER-RL) scheme and the intra-level redundant links (INTRA-RL) scheme, are proposed to enhance the resilience of ALM spanning trees against congestion. Simulation results indicate that the proposed statistical model well describes the congestion in ALM networks and two redundant link schemes both significantly boost the robustness of ALM even in tough network environments.

The virtual teleconferencing system provides a collaborative virtual environment for applications such as collaborative seminar. The challenge of the natural interaction is for the participants to be aware of the virtual environment. A video avatar is a computer-synthesized three-dimension image created using live video. We study the factors contributing to the participant's natural interaction and introduce a video avatar based participant model to implement the natural, accurate and convenient interaction among participants. Our virtual teleconferencing prototype provides a seamless virtual collaboration work environment including eye contacts and gaze awareness. Our experiments on a three parties collaborative seminar show that natural interaction synthesizing can significantly improve the communication and that natural multi-model human computer interaction is necessary in collaboration.

Article Free Access Share on VCS: a virtual environment support for awareness and collaboration Authors: Maojun Zhang Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. China Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. ChinaView Profile , Linda Wu Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. China Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. ChinaView Profile , Lifeng Sun Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. China Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. ChinaView Profile , Yunhao Li Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. China Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. ChinaView Profile , Bing Yang Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. China Department of System Engineering and Mathematics, National University of Defense Technology, Changsha, Hunan, P. R. ChinaView Profile Authors Info & Claims MULTIMEDIA '99: Proceedings of the seventh ACM international conference on Multimedia (Part 2)October 1999 Pages 163–165https://doi.org/10.1145/319878.319922Published:01 October 1999Publication History 2citation221DownloadsMetricsTotal Citations2Total Downloads221Last 12 Months2Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF