In mission-critical Internet-of-Things environments, where data from diverse data sources (stationary and mobile) must be analyzed rapidly, ensuring reliable service provisioning for the execution of analytics is critical. Under dynamic conditions and changing infrastructure capabilities, adaptive methods are required to handle surges in data volumes and execute complex services for IoT applications. In this paper, we propose AMPHI–an adaptive microservice provisioning framework that handles various mission-aware workflows in combined stationary-mobile IoT environments to enable flexible deployment of containerized microservices. AMPHI exploits the variability in cost and performance among operators that implement similar functionalities but with distinct cost/quality/time tradeoffs, and provides solutions for intelligent selection, placement, and sharing of operators across a hybrid set of devices (stationary sensors and unmmaned aerial/ground vehicle) to maximize Quality of Service (QoS) and resource efficiency under dynamic situations. We formulate the microservice provisioning and instantiation as an (NP-hard) optimization problem and design efficient heuristic approaches to deploy microservices by utilizing application and system context. The system’s robustness and resilience are guaranteed by two variant paradigms in AMPHI’s design, one provides with optimal microservice provisoining solutions, and the other makes adaptation operations on existing solutions when abnormal events occur. We study a basic version of AMPHI that focuses on optimal microservice provisioning; the basic version is then enhanced to create a robust and extensible variant of AMPHI that ensures dependable operation through operators that can deal with unexpected failures and urgent service requests. AMPHI is evaluated in the context of smart firefighting with high-rise fires utilizing building sensors and autonomous aerial mobile units. Through real-world testbeds and extensive simulations, we show how AMPHI allows flexible and cost-effective execution of dynamically changing IoT workflows.

Modern smart glasses, when combined with Large Language Models (LLMs), offer a promising new paradigm for assisting visually impaired individuals in daily navigation and scene understanding. Yet, the effectiveness of such assistants depends critically on factors such as camera Angles of View (AoV), semantic extraction, network conditions, and model selection, which have not been systematically studied. To fill this gap, we construct a dataset of egocentric video sequences with multiple AoVs and systematically generated Q&A, and we implement an LLM-based assistant with edge offloading to evaluate different design choices. In particular, we considered four representative multimodal LLMs: MiniCPM-o 2.6 8B, LLaVA-OneVision 7B, Qwen2.5-VL 7B, and Qwen2.5-VL 3B, to cover a diverse range of model architectures and sizes. Through extensive experiments, we find that: (i) current LLMs are still limited in recognizing 360° videos, but semantic extractors improve accuracy by up to 33.77% with minimal impact on delay; (ii) capturing with 360° cameras raises accuracy by an average of 20.6% and achieves up to 85.64% on position-sensitive queries without additional inference time; (iii) higher-bandwidth networks such as WiFi reduce transmission delay, resulting in acceptable response time (sub 1-second); and (iv) different LLMs perform better on different question categories, highlighting the need for careful modeling and offloading strategies in future work.

3D Gaussian Splatting (3DGS) is an emerging media representation that reconstructs real-world 3D scenes in high fidelity, enabling 6-degrees-of-freedom (6-DoF) navigation in virtual reality (VR). However, developing and evaluating 3DGS-enabled applications and optimizing their rendering performance require realistic user navigation data. Such data is currently unavailable for photorealistic 3DGS reconstructions of real-world scenes. This paper introduces EyeNavGS, the first publicly available 6-DoF navigation dataset featuring traces from 46 participants exploring twelve diverse, real-world 3DGS scenes. The dataset was collected at two sites, using the Meta Quest Pro headsets, recording the head pose and eye gaze data for each rendered frame during free world standing 6-DoF navigation. For each of the twelve scenes, we performed careful scene initialization to correct for scene tilt and scale, ensuring a perceptually-comfortable VR experience. We also release our open-source SIBR viewer software fork with record-and-replay functionalities and a suite of utility tools for data processing, conversion, and visualization. The EyeNavGS dataset and its accompanying software tools provide valuable resources for advancing research in 6-DoF viewport prediction, adaptive streaming, 3D saliency, and foveated rendering for 3DGS scenes. The EyeNavGS dataset is available at: https://symmru.github.io/EyeNavGS/

Remote rendering enables high-fidelity virtual reality (VR) experiences on standalone headsets by offloading intensive graphics workloads to remote servers. However, streaming high-quality VR graphics imposes substantial bandwidth and latency challenges. Spatial compression is a form of foveation which addresses this challenge by leveraging the human visual system's varying acuity, allocating higher visual quality around the user's gaze while reducing resolution in the periphery. In this work, we implement three gaze-adaptive foveation methods: Dynamic Axis-Aligned Distortion Transmission (D-AADT2 and D-AADT3) and Dynamic Foveated Radial Warp (D-FRW)) of which only D-AADT2 has been previously presented. These methods dynamically adapt spatial compression based on gaze-tracking input, ensuring optimal perceptual quality. We integrate these methods together with their static counterparts into the open-source Air Light VR (ALVR) remote-rendering framework, enabling native (72 FPS) framerates. We conclude a comprehensive objective evaluation across diverse VR games and demonstrate that the dynamic methods significantly outperform traditional static approaches in both encoding efficiency and perceptual quality metrics. A complementary subjective user study further validates these findings, confirming that dynamic gaze-adaptive foveation substantially enhances visual quality, immersion, and user interaction experience.

We present a novel DASH-based streaming system for dynamic 3D Gaussian Splatting (3DGS) scenes, addressing the challenges of streaming large amounts of 3DGS data over diverse and dynamic networks. Our Layer, Tile, and Segment Adaptive streaming (LTS) system combines three key features: (i) multi-layer streaming, which adapts to diverse client capabilities while balancing visual quality and bandwidth usage, (ii) tiled streaming, which reduces unnecessary data transmission by focusing on the user's viewport, and (iii) segment streaming, which divides dynamic 3DGS scenes into segments, letting clients request them dynamically to handle network fluctuations. Our experimental results demonstrate that our LTS system achieves superior performance in both live and on-demand streaming of dynamic 3DGS scenes compared to the baselines. For example, in live streaming, LTS could achieve up to 99.70% reduction in missing frames on average and deliver a maximum PSNR (Peak Signal-to-Noise Ratio) improvement of 10.08 dB. In on-demand streaming, LTS could reduce the freeze time by up to 92.01%, and increase the synthesized view quality by up to 5.14 dB in PSNR and 0.11 in SSIM (Structural Similarity Index). Our source codes are available at: https://github.com/AIINS-NTHU/LTS-DASH-Streaming-System-for-3DGS.

3D Gaussian Splatting (3DGS) is an emerging approach for training and representing real-world 3D scenes. Due to its photorealistic novel view synthesis and fast rendering speed (e.g., over 100 FPS), it has the potential to transform how scenes that can be explored in 6 degrees-of-freedom (6-DoF) are represented. However, a limiting factor of 3DGS is its large size, which requires high network bandwidth for streaming reconstructed real-world 3D scenes.

The growth of smart communities has expanded the use of IoT as a critical element in the urban planning toolkit by city officials whose goal is to improve the quality of life for citizens. Smart applications such as air-pollution monitoring, intelligent transportation, and smart buildings pose diverse information needs from the underlying sensing, communication, and computation infrastructure. In this article, we propose SmartParcels, a framework that exploits the service needs of communities to generate a comprehensive and cost-effective plan for instrumenting designated regions (often called parcels). SmartParcels embeds a cross-layer approach incorporating information/data, infrastructure, and geospatial layout as interdependent layers. We explore a suite of algorithms (optimal, partial optimal, heuristic) that can be composed in a plug-and-play manner to achieve performance-cost tradeoffs. SmartParcels can be utilized for clean-slate planning (from scratch) or retrofitting communities with existing smart infrastructure. SmartParcels allows planners to explore the possible impact of unexpected events to improve infrastructure resilience with what-if analyzers. Two real-world settings at Hsinchu, Taiwan, and Irvine, California, are leveraged in the evaluation, which reveals that SmartParcels enables a 2×–7× improvement in cost/performance metrics compared to baseline algorithms and achieves a 57% higher communication throughput while dealing with unexpected events.

Dynamic 3D point clouds enable the immersive user experience and thus have become increasingly more popular in volumetric video streaming applications. When being streamed over best-effort networks, point cloud frames may suffer from lost or late packets, leading to non-trivial quality degradation. To solve this problem, we proposed the very first error concealment pipeline framework, which comprises five stages: pre-processing, matching, motion estimation, prediction, and post-processing. Alternative algorithms can be developed for each stage, while algorithms of different stages could be mixed and matched into pipelines for end-to-end performance evaluations. We discussed the design goal and proposed multiple algorithms for each stage. These algorithms were then quantitatively compared using dynamic 3D point cloud sequences with diverse characteristics. Based on the comparison results, we proposed four representative pipelines for: (i) diverse degrees of motion variance, i.e., minor versus significant, and (ii) different application requirements, i.e., high quality versus low overhead. Extensive end-to-end evaluations of our proposed pipelines demonstrated their superior concealed quality over the 3D frame-copy method in both: (i) 3D metrics, by up to 5.32 dB in GPSNR and 1.7 dB in CPSNR,and (ii) 2D metrics, by up to 2.22 dB in PSNR, 0.06 in SSIM, and 11.67 in VMAF. Adding to that, a user study with 15 subjects indicated that our best-performing pipeline achieved 100% preference winning rate over the state-of-the-art learning-based interpolation algorithms while consuming merely up to 8.55% of running time.

Cloud Virtual Reality (VR) gaming offloads computationally intensive VR games to resourceful data centers. However, ensuring good Quality of Experience (QoE) in cloud VR gaming is inherently challenging as VR gamers demand high visual quality, short response time, and negligible cybersickness. In this article, we study the QoE of cloud VR gaming and build a QoE-optimized system in a few steps. First, we establish a cloud VR gaming testbed capable of emulating various network conditions. Using the testbed, we conduct comprehensive QoE evaluations using a user study to evaluate the influence of diverse factors, such as encoding settings, network conditions, and game genres, on gamer QoE scores. Second, we construct the very first QoE models for cloud VR gaming using our QoE evaluation results. Our QoE models achieve up to 0.93 ( \(\sigma=0.02\) ) in Pearson Linear Correlation Coefficient (PLCC) and 0.92 ( \(\sigma=0.02\) ) in Spearman Rank-Order Correlation Coefficient (SROCC), where \(\sigma\) stands for the standard deviation. Last, we leverage our QoE models for dynamically adapting encoding settings in our testbed. Extensive experiments revealed that, compared to the current practice, our adaptive cloud VR gaming system improves: (i) overall quality by 0.87 ( \(\sigma=0.44\) ), (ii) visual quality by 0.61 ( \(\sigma=0.45\) ), and (iii) interaction quality by 1.20 ( \(\sigma=0.48\) ) on average in 5-point Mean Opinion Score (MOS).

The rapid deployment of Internet-of-Things (IoT) devices in smart environments such as smart campuses and cities necessitates robust Quality-of-Service (QoS) management across heterogeneous networks. In this paper, we extend the concept of Network Digital Twin (NDT) to networked IoT devices, presenting the Network Digital Twin Controller (NDTC) that enhances the functionality and performance of smart environ-ments. Our NDTC addresses key challenges by creating Digital Twins (DTs) of Physical Twins (PTs), synchronizing their states, and performing QoS-related what-if analyses. Specifically, we built a DT-enabled IoT-instrumented smart environment using an open-source Software-Defined Network (SDN) controller. We formulated and solved the state synchronization problem using our proposed Optimal Update (OU) and Gradient-driven Update (GU) algorithms, carefully adjusting the update frequency and data granularity to minimize DT/PT state deviation within given network bandwidth budgets. We also formulated and addressed the what-if analysis problem by selecting optimal what-if analyzers using our Optimal Selection (OS) algorithm for the most accurate QoS predictions under a given computing time budget. Our extensive experiments on a real testbed demonstrated the merits of our proposed solution: (i) our developed NDTC and algorithms meet the functional requirements, (ii) our OU and GU algorithms significantly reduce the state deviation between PTs and DTs, and (iii) our OS algorithm largely reduces the prediction errors of what-if analyses.

Volumetric video streaming has become increasingly popular in recent years due to its support of 6 degrees-of-freedom (6-DoF) exploration. There is, however, a shortage of dynamic 6-DoF content suitable for comparing the performance among heterogeneous volumetric video representations. This paper introduces a software toolkit for creating both a dataset of dynamic 6-DoF content in point clouds and a dataset for training and testing neural-based representations such as neural radiance fields (NeRF). Starting with freely available 3D assets online, our software toolkit uses the Blender Python API to generate training and testing datasets for neural-based dynamic volumetric model training. The created datasets are compliant with existing neural-based model training and rendering frameworks. The software can also construct point cloud sequences derived from synthetic dynamic 3D meshes. This further facilitates comparing point clouds and neural-based methods for volumetric video representation. We release the software toolkit along with a rich set of sequence datasets generated in compliance with the permissions granted by the original 3D asset creators. With our toolkit and dataset, we aim to facilitate research from the multimedia systems community to support practical volumetric streaming. Our software toolkit and dataset are available at: https://6-dof-dynamic-content-software.github.io/.

Driver activity recognition has become crucial for intelligent transportation and automotive safety systems. However, existing studies mainly focus on fatigue-related behaviors while neglecting other activities for analyzing driver behavior and intent. In this work, we introduce a novel dataset for fine-grained driver activities, utilizing diverse sensors such as mmWave radars, RGB, and depth cameras, each of which includes three camera angles: body, face, and hands. This multi-modal and multi-angle approach allows for comprehensive driver behavior analysis, including hand gestures, head movement, and object interactions. Moreover, including mmWave radars provides significant privacy advantages, as the sparse dynamic point clouds prevent the identification of the driver's face and other personal information. This dataset is valuable for researchers and developers on driver activity recognition and behavior analysis. It enables the development and evaluation of robust, privacy-conscious solutions for improving road safety, driver assistance, and in-vehicle interaction. Furthermore, the multi-modal nature of the data enables the exploration of sensor fusion techniques, unlocking the full potential of diverse sensing modalities to understand complex driver behaviors.

In mission-critical IoT environments, where data from diverse data sources (stationary and mobile) must be analyzed rapidly, ensuring reliable service provisioning for the execution of analytics is critical. Under dynamic conditions and changing infrastructure capabilities, adaptive methods are required to handle surges in data volumes and execute complex services for IoT applications. In this paper, we propose AMPHI - an adaptive microservice provisioning framework that handles various mission-aware workflows in combined stationary-mobile IoT environments to enable flexible deployment of containerized microservices. AMPHI exploits the variability in cost and performance among operators that implement similar functionalities but with distinct cost/quality/time tradeoffs, and employs apriori and on-the-fly techniques for intelligent selection, placement, and sharing of operators across a hybrid set of devices (stationary sensors, drones, and rovers) to maximize Quality of Service (QoS) and resource efficiency under dynamic situations. We formulate the microservice provisioning and instantiation as an (NP-hard) optimization problem and design efficient heuristic approaches to deploy microservices by utilizing application and system context. AMPHI is evaluated in the context of smart firefighting with high-rise fires utilizing building sensors and autonomous aerial mobile units. Through real-world testbeds and extensive simulations, we show how AMPHI allows flexible and cost-effective execution of dynamically changing IoT workflows.

Smart city transportation infrastructure will soon demand the development of reliable underground IoT (IoUT) communication. In this paper, we develop a novel analytical model, MAME (Material Aware Measurement Enhanced), to capture signal propagation properties in wireless IoUT networks to achieve reliable data transport. A driving motivation is monitoring underground infrastructure systems (e.g., pipelines and storm drains) for early detection of anomalies and failures to guide human investigation and intervention. We analyze the feasibility of successfully receiving wireless data packets from underground (UG) sensor nodes through multiple material layers and under diverse environmental conditions. Our proposed approach integrates physics-based modeling and empirical studies with small-scale testbeds (in our lab and outdoors) with multiple channel setups and physical layer attributes. We derive a novel MAME approach to model signal propagation in both 802.11-based WiFi and LoRaWAN networks. The resulting MAME model is shown to capture communication behavior in WiFi and LoRaWAN networks accurately. The MAME model is used to augment the popular NS3 simulator to explore scaled-up underground networks and varying channel conditions (e.g., soil moisture level). Such a combined analytical-empirical approach will enable the communication control plane and application layer to better predict channel conditions for improved IoUT network design.

Data from multi-modal sensors, such as Red-Green-Blue (RGB) cameras, thermal cameras, microphones, and mmWave radars, have gradually been adopted in various classification problems for better accuracy. Some sensors, like RGB cameras and microphones, however, capture privacy-invasive data, which are less likely to be used in centralized learning. Although the Federated Learning (FL) paradigm frees clients from sharing their sensor data, doing so results in reduced classification accuracy and increased training time. In this article, we introduce a novel Heterogeneous Privacy Federated Learning (HPFL) paradigm to better capitalize on the less privacy-invasive sensor data, such as thermal images and mmWave point clouds, by uploading them to the server for closing the performance gap between FL and centralized learning. HPFL not only allows clients to keep the more privacy-invasive sensor data private, such as RGB images and human voices, but also gives each client total freedom to define the levels of their privacy concern on individual sensor modalities. For example, more sensitive users may prefer to keep their thermal images private, while others do not mind sharing these images. We carry out extensive experiments to evaluate the HPFL paradigm using two representative classification problems: semantic segmentation and emotion recognition. Several key findings demonstrate the merits of HPFL: (i) compared to FedAvg, it improves foreground accuracy by 18.20% in semantic segmentation and boosts the F1-score by 4.20% in emotion recognition, (ii) with heterogeneous privacy concern levels, it achieves an even larger F1-score improvement of 6.17–16.05% in emotion recognition, and (iii) it also outperforms the state-of-the-art FL approaches by 12.04–17.70% in foreground accuracy and 2.54–4.10% in F1-score.

Connected and Autonomous Vehicles (CAVs) often come with sensors, such as LiDARs, to improve road safety. Although dynamic LiDAR point clouds could be streamed over wireless networks from CAVs to edge servers for computationally-intensive classification tasks, the problem of incomplete point cloud frames caused by unreliable wireless networks has yet to be investigated in the literature. In this paper, we propose the very first LiDAR Error Concealment (LEC) algorithm, to conceal incomplete point cloud frames due to packet loss or late packets to minimize the distortion in Chamfer distance. Driven by machine learning techniques, our LEC algorithm adaptively performs Temporal Prediction (TP), Spatial Interpolation (SI), or Temporal Interpolation (TI) to conceal incomplete point cloud frames. We evaluate the performance of our LEC algorithms with a comprehensive co-simulator of the popular CARLA and NS-3 implemented by us and the KITTI Odometry dataset. Our simulation results reveal that the proposed LEC algorithm outperforms the TP, SI, and TI algorithms by up to 82.68% and 30.17% in Chamfer and Hausdorff distances, and terminates in 360–570 ms in a C-V2X network. Moreover, our LEC algorithm also outperforms other algorithms by up to 87.43% and 66.58% in Chamfer and Hausdorff distances in a DSRC network.

Online 6-DoF view touring has become increasingly popular due to hardware advances and the recent pandemic. One way for content creators to support many 6-DoF clients is by transmitting 3D content to them, which leads to content leakage. Another way for content creators is to render and stream novel views for 6-DoF clients, which incurs staggering computational and networking workloads. In this paper, we develop a blind streaming system that leverages cloud service providers between content creators and 6-DoF clients. Our system has two core design objectives: (i) to generate high-quality novel views for 6-DoF clients without retrieving 3D content from content creators, (ii) to support many 6-DoF clients without overloading the content creators. We achieve these two goals in the following steps. First, we design a source view request/response interface between cloud service providers and content creators for efficient communications. Second, we design novel view optimization algorithms for cloud service providers to intelligently select the minimal set of source views while considering the workload of content creators. Third, we employ scalable client side view synthesis for 6-DoF clients with heterogeneous device capabilities and personalized 6-DoF client poses and preferences. Our evaluation results demonstrate the merits of our solution, compared to the state-of-the-arts, our system: (i) improves synthesized novel views by 2.27 dB in PSNR and 12 in VMAF on average and (ii) reduces the bandwidth consumption by 94% on average. In fact, our solution approaches the performance of an unrealistic optimal solution with unlimited source views, achieving performance gaps as small as 0.75 dB in PSNR and 3.8 in VMAF.

Motion estimation in a 3D point cloud sequence is a fundamental operation with many applications, including compression, error concealment, and temporal upscaling. While there have been multiple research contributions toward estimating the motion vector of points between frames, there is a lack of a dynamic 3D point cloud dataset with motion ground truth to benchmark against. In this paper, we present an open dynamic 3D point cloud dataset to fill this gap. Our dataset consists of synthetically generated objects with pre-determined motion patterns, allowing us to generate the motion vectors for the points. Our dataset contains nine objects in three categories (shape, avatar, and textile) with different animation patterns. We also provide semantic segmentation of each avatar object in the dataset. Our dataset can be used by researchers who need temporal information across frames. As an example, we present an evaluation of two motion estimation methods using our dataset.

Virtual Reality (VR) applications offer an immersive user experience at the expense of privacy leakage caused by inevitably streaming various new types of user data. While some privacy-preserving approaches have been proposed for protecting one type of data, how to design and evaluate approaches for multiple types of user data are still open. On the other hand, preserving privacy will degrade the quality of experience of VR applications or say the utility of user data. How to achieve efficient utility-privacy tradeoff with multiple types of data is also open. Both call for a dataset that contains multiple types of user data and personal attributes of users as ground-truth values. In this paper, we collect a 6 degree-of-freedom VR dataset of 3D virtual worlds for the investigation of privacy-preserving approaches and utility-privacy tradeoff.

Human activity recognition, which involves recognizing human activities from sensor data, has drawn a lot of interest from researchers and practitioners as a result of the advent of smart homes, smart cities, and smart systems. Existing studies on activity recognition mostly concentrate on coarse-grained activities like walking and jumping, while fine-grained activities like eating and drinking are understudied because it is more difficult to recognize fine-grained activities than coarse-grained ones. As such, food intake activity recognition in particular is under investigation in the literature despite its importance for human health and well-being, including telehealth and diet management. In order to determine sensors' practical recognition accuracy, preferably with the least amount of privacy intrusion, a dataset of food intake activities utilizing sensors with varying degrees of privacy sensitivity is required. In this study, we collected such a dataset by collecting fine-grained food intake activities using sensors of heterogeneous privacy sensitivity levels, namely a mmWave radar, an RGB camera, and a depth camera. Solutions to recognize food intake activities can be developed using this dataset, which may provide a more comprehensive picture of the accuracy and privacy trade-offs involved with heterogeneous sensors.

Cloud Virtual Reality (VR) gaming offloads the computationally-intensive rendering tasks from resource-limited Head-Mounted Displays (HMDs) to cloud servers, which consume a staggering amount of bandwidth for high-quality gaming experiences. One way to cope with such high bandwidth demands is to capitalize on human vision systems by allocating a higher bitrate to the foveal region of HMD viewport, which is known as foveation in the literature. Although foveation was employed by remote VR gaming, existing open-source projects all adopt static foveation, in which the HMD gamer gaze position is assumed to be fixed at the viewport center. In this paper, we construct the very first cloud VR gaming system that supports dynamic foveation. That is, the real-time gaze positions of gamers are streamed from eye-trackers on HMDs to cloud servers, which in turn adjust the foveation parameters, such as foveal region size/location and peripheral region quality degradation, accordingly. Using our developed cloud VR gaming system, we design and carry out a user study using a game called Fruit Ninja VR 2 to find the foveation parameters in static and dynamic foveation for maximizing the gaming Quality of Experience (QoE) in Mean Opinion Score (MOS). With the chosen foveation parameters, we found that, compared to cloud VR gaming without foveation, static foveation leads to a MOS increase of 0.60 and a bitrate reduction of 8.71%. Furthermore, adopting dynamic foveation results in an additional 0.60 increase on MOS while saving 9.81% bitrate, compared to static foveation. Our findings demonstrate the potential of dynamic foveation in cloud VR gaming, which dictates both high visual quality and short response time. The optimization techniques developed in this and follow-up work could benefit other cloud-rendered applications that typically have less strict requirements than cloud VR gaming.

With the growth of Extended Reality (XR) and capturing devices, point cloud representation has become attractive to academics and industry. Point Cloud Compression (PCC) algorithms further promote numerous XR applications that may change our daily life. However, in the literature, PCC algorithms are often evaluated with heterogeneous datasets, metrics, and parameters, making the results hard to interpret. In this article, we propose an open-source benchmark platform called PCC Arena. Our platform is modularized in three aspects: PCC algorithms, point cloud datasets, and performance metrics. Users can easily extend PCC Arena in each aspect to fulfill the requirements of their experiments. To show the effectiveness of PCC Arena, we integrate seven PCC algorithms into PCC Arena along with six point cloud datasets. We then compare the algorithms on ten carefully selected metrics to evaluate the quality of the output point clouds. We further conduct a user study to quantify the user-perceived quality of rendered images that are produced by different PCC algorithms. Several novel insights are revealed in our comparison: (i) Signal Processing (SP)-based PCC algorithms are stable for different usage scenarios, but the trade-offs between coding efficiency and quality should be carefully addressed, (ii) Neural Network (NN)-based PCC algorithms have the potential to consume lower bitrates yet provide similar results to SP-based algorithms, (iii) NN-based PCC algorithms may generate artifacts and suffer from long running time, and (iv) NN-based PCC algorithms are worth more in-depth studies as the recently proposed NN-based PCC algorithms improve the quality and running time. We believe that PCC Arena can play an essential role in allowing engineers and researchers to better interpret and compare the performance of future PCC algorithms.

Recently standardized MPEG Video-based Point Cloud Compression (V-PCC) codec has shown promise in achieving a good rate-distortion ratio of dynamic 3D point cloud compression. Current error concealment methods of V-PCC, however, lead to significantly distorted 3D point cloud frames under imperfect network conditions. To address this problem, we propose a general framework for concealing distorted and lost 3D point cloud frames due to packet loss. We also design, implement, and evaluate a suite of tools for each stage of our framework, which can be combined into multiple variants of error concealment algorithms. We conduct extensive experiments using seven dynamic 3D point cloud sequences with diverse characteristics to understand the strengths and limitations of our proposed error concealment algorithms. Our experiment results show that our algorithms outperform: (i) the method employed by V-PCC by at least 3.58 dB in Geometry Peak Signal-to-Noise Ratio (GPSNR) and 10.68 in Video Multi-Method Assessment Fusion (VMAF) and (ii) point cloud frame copy method by at most 5.8 dB in (3D) GPSNR and 12.0 in (2D) VMAF. Further, the proposed error concealment framework and algorithms work in the 3D domain, and thus are agnostic to the codecs and are applicable to future point cloud compression standards

High-rise fires are among the largest threats to safety in modern cities, and autonomous drones with multi-modal sensors can be employed to enhance situational awareness in such unfortunate disasters. In this paper, we study the fine-grained measurement selection problem for drones being dispatched to perform situation monitoring tasks in high-rise fires. Our problem considers multiple sensor/media types, classifier designs, and measurement locations, which were overlooked in prior waypoint scheduling studies. For concrete discussion, we adopt window openness as the target situation, while other situations can be readily supported by our solution as well. More specifically, we: (i) develop diverse window openness classifiers, (ii) mathematically formulate the fine-grained measurement selection problem and solve it using two algorithms, and (iii) create a photo-realistic simulator and an event-driven simulator to evaluate our algorithms. The evaluation results demonstrate that our proposed algorithms achieve higher classification accuracy (up to 50% improvement), deliver more feasible solutions (up to 100% improvement), and reduce energy consumption (up to 6.78 times reduction), compared to the current practices.

The importance of IoT analytics in smart deploy-ments has resulted in an increased use of powerful Deep Neural Network (DNN) models to extract insights from the growing amount of IoT sensor data. Traditional approaches that entirely offload computation and model deployment to cloud servers have been shown to be inefficient due to network congestion and latency concerns. However, with the improved capabilities of IoT devices, it has now become possible to distribute and host DNNs across IoT devices, edge servers and the cloud. In this paper, we propose a multi-user system, called T2C, to dynamically choose, deploy, monitor and control DNN-driven IoT analytics in a thing-to-cloud continuum. T2C leverages strategies such as multi-task learning, hitchhiking, early exit, and dynamic reconfiguration, to maximize the number of served user requests while simultaneously satisfying accuracy and latency requirements. We propose a suite of deployment planning and reconfiguration algorithms to dynamically deploy and migrate DNN layers between IoT devices, edge servers, and the cloud. We implement T2C in a prototype testbed and show that our system: (i) achieves 6.8X throughput boost compared to baseline algorithms in the planning phase, and (ii) improves the satisfied ratio by up to 35% in the operation and reconfiguration phase.

Conducting user studies to quantify the Quality of Experience (QoE) of watching the increasingly more popular 360° videos in Head-Mounted Displays (HMDs) is time-consuming, tedious, and expensive. Deriving QoE models, however, is very challenging because of the diverse viewing behaviors and complex QoE features and factors. In this article, we compile a wide spectrum of QoE features and factors that may contribute to the overall QoE. We design and conduct a user study to build a dataset of the overall QoE, QoE features, and QoE factors. Using the dataset, we derive the QoE models for both the Mean Opinion Score (MOS) and Individual Score (IS), where MOS captures the aggregated QoE across all subjects, while IS captures the QoE of individual subjects. Our derived overall QoE models achieve 0.98 and 0.91 in Pearson’s Linear Correlation Coefficient (PLCC) for MOS and IS, respectively. Besides, we make several new observations on our user study results, such as (1) content factors dominate the overall QoE across all factor categories, (2) Video Multi-Method Assessment Fusion (VMAF) is the dominating factor among content factors, and (3) the perceived cybersickness is affected by human factors more among others. Our proposed user study design is useful for QoE modeling (specifically) and subjective evaluations (in general) of emerging 360° tiled video streaming to HMDs.

Immersive video streaming technologies improve Virtual Reality (VR) user experience by providing users more intuitive ways to move in simulated worlds, e.g., with 6 Degree-of-Freedom (6DoF) interaction mode. A naive method to achieve 6DoF is deploying cameras at numerous different positions and orientations that may be required based on users’ movement, which unfortunately is expensive, tedious, and inefficient. A better solution for realizing 6DoF interactions is to synthesize target views on-the-fly from a limited number of source views. While such view synthesis is enabled by the recent Test Model for Immersive Video (TMIV) codec, TMIV dictates manually-composed configurations, which cannot exercise the tradeoff among video quality, decoding time, and bandwidth consumption. In this article, we study the limitation of TMIV and solve its configuration optimization problem by searching for the optimal configuration in a huge configuration space. We first identify the critical parameters in the TMIV configurations. Then, we introduce two Neural Network (NN) -based algorithms from two heterogeneous aspects: (i) a Convolutional Neural Network (CNN) algorithm solving a regression problem and (ii) a Deep Reinforcement Learning (DRL) algorithm solving a decision making problem, respectively. We conduct both objective and subjective experiments to evaluate the CNN and DRL algorithms on two diverse datasets: an equirectangular and a perspective projection dataset. The objective evaluations reveal that both algorithms significantly outperform the default configurations. In particular, with the equirectangular (perspective) projection dataset, the proposed algorithms only require 95% (23%) decoding time, stream 79% (23%) views, and improve the utility by 6% (73%) on average. The subjective evaluations confirm the proposed algorithms consume fewer resources while achieving comparable Quality of Experience (QoE) than the default and the optimal TMIV configurations.

We develop, implement, and demonstrate an open-source simulator, called AirSimN, for evaluating drone-based wireless networks in this extended abstract. AirSimN is different from all prior attempts in the literature because it concurrently supports aerodynamic, computer vision, and network simulations. We carefully design it to minimize the effort of realizing virtually arbitrary drone applications, thanks to the active and popular AirSim and NS-3 projects. Many mobile computing and wireless networking projects on, e.g., drone feedback controllers, drone vision algorithms, and 5G/6G cellular network planning, can leverage AirSimN for large-scale evaluations.

We present a study on the impact of packet loss on dynamic 3D point cloud streaming, encoded with MPEG Video-based Point Cloud Compression (V-PCC) standard. We show the distortion when different channels of V-PCC bitstream are lost, with the loss of occupancy and geometry data impacting the quality most significantly. Our results point to the need for better error concealment techniques. We end the paper by presenting preliminary thoughts and experimental results of two naive error concealment techniques in the point cloud domain, for attributes and geometry data, respectively, and highlight the limitations of each.

In this paper, we propose DragonFly, a drone-based data collection framework to enhance real-time situational awareness in high-rise buildings, focusing specifically on mission-critical high-rise fire scenarios. The goal of our proposed solution is to use multiple drones with visual sensors to collect reliable and timely data for monitoring the exterior of a high-rise building. Drones are especially useful in obtaining data from hard-to-access regions in high-rise fires that are used to monitor fire/smoke that might have propagated to higher floors, detect the presence of humans requiring assistance near windows, and determine window open/close states which can have a significant impact on the speed and direction of fire spread. Given a dynamically evolving set of events and multiple drones, the core challenge addressed is to develop a plan for multiple drones to gather a set of observations that can improve both the coverage (identify more events) and accuracy (obtain fine-grained for improved event detection). We develop a solution for the Multi-drone Waypoint scheduling problem (NP-hard) in two steps: 1) allocation of monitoring tasks (AMT) to individual drones and 2) dynamic waypoint scheduling (DWS) that determines the waypoint sequence for each drone to visit. We evaluate our proposed approach using a simulated high-rise fire scenario with a realistic fire spread model and study the applicability and efficiency of the proposed algorithms compared to baseline techniques. The simulation results demonstrate the superior performance of the proposed AMT-DWS algorithms. DragonFly achieve 33% fewer missing events and up to 39 times gain in accuracy, captured as the minimum weighted AUC (Area Under Curve) as compared to baseline algorithms. DragonFly delivers over 85% missing events and about 1.3 times the minimum weighted AUC in comparison to current approaches.

Dynamic Adaptive Streaming over HTTP (DASH) has been widely used by several popular streaming services, such as YouTube, Netflix, and Facebook. Adopting DASH requires to pre-determine a set of encoding configurations, called encoding ladder, to generate a set of representations stored on the streaming server. These representations are adaptively requested by clients according to their network conditions during streaming sessions. In this article, we aim to solve the optimal laddering problem that determines the optimal encoding ladder to maximize the client viewing quality. In particular, we consider video models, viewing probability, and client distribution to formulate the mathematical problem. We use a divide-and-conquer approach to decompose the problem into two subproblems: (i) per-class optimization for clients with different bandwidths and (ii) global optimization to maximize the overall viewing quality under the storage limit of the streaming server. We propose two algorithms for each of the per-class optimization and global optimization problems. Analytical analysis and real experiments are conducted to evaluate the performance of our proposed algorithms, compared to other state-of-the-art algorithms. Based on the results, we recommend a combination of the proposed algorithms to solve the optimal laddering problem. The evaluation results show the merits of our recommended algorithms, which: (i) outperform the state-of-the-art algorithms by up to 52.17 and 26.35 in Viewport Video Multi-Method Assessment Fusion (V-VMAF) in per-class optimization, (ii) outperform the state-of-the-art algorithms by up to 43.14 in V-VMAF for optimal laddering in global optimization, (iii) achieve good scalability under different storage limits and number of bandwidth classes, and (iv) run faster than the state-of-the-art algorithms.

Real-time screen-sharing provides users with ubiquitous access to remote applications, such as computer games, movie players, and desktop applications (apps), anywhere and anytime. In this article, we study the performance of different screen-sharing technologies, which can be classified into native and clientless ones. The native ones dictate that users install special-purpose software, while the clientless ones directly run in web browsers. In particular, we conduct extensive experiments in three steps. First, we identify a suite of the most representative native and clientless screen-sharing technologies. Second, we propose a systematic measurement methodology for comparing screen-sharing technologies under diverse and dynamic network conditions using different performance metrics. Last, we conduct extensive experiments and perform in-depth analysis to quantify the performance gap between clientless and native screen-sharing technologies. We found that our WebRTC-based implementation achieves the best overall performance. More precisely, it consumes a maximum of 3 Mbps bandwidth while reaching a high decoding ratio and delivering good video quality. Moreover, it leads to a steadily high decoding ratio and video quality under dynamic network conditions. By presenting the very first rigorous comparisons of the native and clientless screen-sharing technologies, this article will stimulate more exciting studies on the emerging clientless screen-sharing technologies.

Internet-of-Things (IoT) devices are connected to the Internet through a gateway, which can host IoT analytics encapsulated in containers to convert raw sensor data into more condensed processed data. In this paper, we study two research problems to maximize the overall Quality-of-Service (QoS) level of all IoT analytics that run on both data center servers and gateways. The first problem is to select additional IoT analytics to deploy on a gateway to save upload bandwidth due to transmitting raw sensor data. The second problem is to allocate the residue upload bandwidth among all IoT analytics to maximize the overall QoS level. We propose several algorithms to solve these two research problems. We have implemented real testbeds to evaluate our proposed system and algorithms. Our experiment results reveal that the proposed algorithms: (i) capitalize the download bandwidth and storage space of the gateway for saving the upload bandwidth consumption and (ii) achieve high QoS levels without overloading the network and gateway.

Increasingly more surveillance cameras in smart environments stream videos to storage servers for on-demand video analytics queries in the future. Unlike on-demand video services, in which maximizing the user-perceived video quality is the design objective, the considered storage servers aim to retain as much information as possible while offering enough space for incoming video clips. In this paper, we design, optimize, and implement an analytics-aware storage server on a smart campus testbed at NTHU, Taiwan, which consists of eight smart street lamps equipped with various sensors, network devices, analytics servers, and a storage server. We focus on the design and implementation of the storage server, and consider two key research problems: (i) how to efficiently determine the information amount of individual video clips and (ii) how to intelligently downsample individual video clips. More specifically, the first problem is to sample video frames from the stored video clips to analyze for approximations of the information amount without overloading the storage server. The resulting information amount is fed into the second problem to decide the video downsampling approaches for retaining as much information amount as possible without consuming excessive storage space. We propose two efficient algorithms to solve these two problems and compare their performance with the current practices via real experiments on our smart campus testbed. Our experiment results reveal the practicality and efficiency of our proposed design and algorithms, e.g., compared to the current practices, our storage server: (i) improves the per-request information amount by up to ~ 4 times, (ii) increases the total information amount by at most ~ 20%, (iii) boosts the number of saved video clips by up to ~ 35%, (iv) runs in real-time, and (v) scales well with larger storage space.

An increasing number of community spaces are being instrumented with heterogeneous IoT sensors and actuators that enable continuous monitoring of the surrounding environments. Data streams generated from the devices are analyzed using a range of analytics operators and transformed into meaningful information for community monitoring applications. To ensure high quality results, timely monitoring, and application reliability, we argue that these operators must be hosted at edge servers located in close proximity to the community space. In this paper, we present a Resource Efficient Adaptive Monitoring (REAM) framework at the edge that adaptively selects workflows of devices and operators to maintain adequate quality of information for the application at hand while judiciously consuming the limited resources available on edge servers. IoT deployments in community spaces are in a state of continuous flux that are dictated by the nature of activities and events within the space. Since these spaces are complex and change dynamically, and events can take place under different environmental contexts, developing a one-size-fits-all model that works for all types of spaces is infeasible. The REAM framework utilizes deep reinforcement learning agents that learn by interacting with each individual community spaces and take decisions based on the state of the environment in each space and other contextual information. We evaluate our framework on two real-world testbeds in Orange County, USA and NTHU, Taiwan. The evaluation results show that community spaces using REAM can achieve > 90% monitoring accuracy while incurring ~ 50% less resource consumption costs compared to existing static monitoring and Machine Learning driven approaches.

In a question & answer (Q& A) system, credible users provide answers of higher correctness. However, in a distributed social network based Q& A (SNQ& A) system, an asker does not know a k-hop answerer's credibility, thus making it difficult for the asker to assess the answer correctness. Therefore, a credibility-enabled distributed SNQ& A system is crucial for determining the correctness of the answers. To this end, we propose CRED, a credibility-enabled distributed SNQ& A system, which facilitates each user to assess the correctness of the provided answers. CRED utilizes subjective logic to build interestwise friend-to-friend credibility opinions under uncertainties. The developed opinions are then accumulated by CRED to get each user's aggregated credibility opinion, which may reflect the user's real credibility. CRED forwards a question to users with highest credibility beliefs in the question interest category. Our evaluation results show that, on average, CRED accomplishes higher success ratio, higher answer correctness, and lower answer uncertainty by 12.1%, 16.4%, and 22.2%, respectively, as compared to the best-performing baseline systems.

We study the problem of predicting the viewing probability of different parts of 3600 videos when streaming them to head-mounted displays. We propose a fixation prediction network based on recurrent neural network, which leverages sensor and content features. The content features are derived by computer vision (CV) algorithms, which may suffer from inferior performance due to various types of distortion caused by diverse 3600 video projection models. We propose a unified approach with overlapping virtual viewports to eliminate such negative effects, and we evaluate our proposed solution using several CV algorithms, such as saliency detection, face detection, and object detection. We find that overlapping virtual viewports increase the performance of these existing CV algorithms that were not trained for 3600 videos. We next fine-tune our fixation prediction network with diverse design options, including: 1) with or without overlapping virtual viewports, 2) with or without future content features, and 3) different feature sampling rates. We empirically choose the best fixation prediction network and use it in a 3600 video streaming system. We conduct extensive trace-driven simulations with a large-scale dataset to quantify the performance of the 3600 video streaming system with different fixation prediction algorithms. The results show that our proposed fixation prediction network outperforms other algorithms in several aspects, such as: 1) achieving comparable video quality (average gaps between -0.05 and 0.92 dB), 2) consuming much less bandwidth (average bandwidth reduction by up to 8 Mb/s), 3) reducing the rebuffering time (on average 40 s in bandwidth-limited 4G cellular networks), and 4) running in real-time (at most 124 ms).

Although watching 360° videos with Head-Mounted Displays (HMDs) is getting increasingly more popular, our understanding of its Quality-of-Experience (QoE) is rather limited. This paper develops models to predict the QoE levels of watching 360° videos with HMDs. We implement a 360° video player for diverse projection schemes and conduct user studies to explore the implications of different factors affecting the QoE levels of 360° videos. Multiple QoE models are constructed based on these factors. Evaluation results on the performance of our QoE models are quite positive. Furthermore, we offer our recommendation on model selections under different circumstances. To the best of our knowledge, this paper is among the few attempts to build QoE models for watching 360° videos with HMDs.

Modern IoT devices are equipped with media-rich sensors that generate a heavy burden to local access networks. To improve the efficiency of data collection, we introduce the concept of "IoT zones" as geographically-correlated clusters of local IoT devices with well connected wireless networks that may have limited access to the Internet. We develop techniques to create a cost-effective data collection network using existing transportation fleets with predefined schedules to collect sensor data from IoT zones and upload them at locations with better network connectivity. Specifically, we provide solutions to the upload point placement and upload path planning problems given tradeoffs between collection quality, timing needs (QoS), and installation cost. We evaluate our approaches using a real-world bus network in Orange County, CA and study the applicability and efficiency of the proposed method as compared to several other approaches. The trace-driven simulations reveal that our best-performing algorithm: upload point selection (UPS) algorithm significantly outperforms others, e.g., in one of the scenarios with 160 total cost, it achieves sub-21 sec data transfer time (15+ times improvement), sub 3.2% late delivery ratio (about 12 times improvement), and above 96% data delivery ratio (about 50% improvement). In addition, it achieves the above performance without excessive installation cost: even when a cost limit of 640 is given, UPS algorithm opts for a solution with about 160 total cost (versus 640 from others).

In urban environments, mobile crowdsensing can be used to augment in-situ sensing deployments (e.g. for environmental and community monitoring) in a flexible and cost-efficient manner. The additional participation provided by crowdsensing enables improved data collection coverage and enhances timeliness of data delivery. However, as the number of participating devices/users increases, efficient management is required to handle the increased operational cost of the infrastructure and associated cloud services - exploiting spatiotemporal redundancy in sensing can help cost-efficient utilization of resources. In this paper, we develop solutions to exploit the mobility of the crowd and manage the sensing capability of participating devices to effectively meet application/user demands for hybrid urban sensing applications. Specifically, we address the spatiotemporal scheduling problem to create high-resolution maps (e.g. for pollution sensing) by developing a common framework to capture spatiotemporal impact of multiple sensor types that generate heterogeneous data at different levels of granularity. We develop an online scheduling approach that leverages the knowledge of device location and sensing capability to selectively activate nodes and sensors. We build a multi-sensor platform that enables data collection, data exchange, and node management. Prototype deployments in three different campus/community testbeds were instrumented for measurements. Traces collected from the testbeds are used to drive extensive large scale simulations. Results show that our proposed solution achieves improved data coverage and utility under data constraints with lower costs (30% fewer active nodes) than naive approaches.

Over the past years, 360° video streaming is getting popular. Watching these videos with Head-Mounted Displays (HMDs), also known as Virtual Reality (VR) headsets, gives more immersive experience than using traditional planar monitors. To fulfill a real immersive experience, there are several challenges, such as high bandwidth consumption, latency-sensitive, and heterogeneous HMD devices. In this paper, we propose an edge-assisted 360° video streaming system, which leverages edge servers to render viewports for viewers of 360° videos. We formulate an optimization problem to determine which HMD clients should be served by the edge server. We design an algorithm to solve this problem, and implement a real testbed as a proof-of-concept. The resulting edge-assisted 360° video streaming system is extensively evaluated with a public 360° viewing dataset. Leveraging edge servers, we reduce the bandwidth usage and computational workload on HMD clients. Moreover, lower network latency is achieved. The evaluation results show that compared to current 360° video streaming platforms, our edge-assisted rendering platform: (i) saves up to 62% in bandwidth consumption, (ii) achieves higher viewing quality, (iii) reduces the computation workload for those lightweight HMDs, and (iv) saves the battery life of HMD clients.

We demonstrate a managed edge computing platform for Internet-of-Things (IoT) devices, which supports dynamic deployment of virtualized containers running distributed analytics. We build a model city, and install multiple Raspberry Pis as minions, and a mini PC as the master. Through the web dashboard on the master, we show how users can remotely monitor, manage, and upgrade the IoT analytics and devices. Multiple concrete IoT analytics, namely: (i) air quality monitor, (ii) sound classifier, and (iii) image recognizer are demonstrated. Several sample measurements on deployment speed, Quality-of- Service (QoS) achievements, and event-driven mechanisms are also carried out on the testbed.

We present the first Media-Aware Network Element (MANE) for intelligently streaming scalable video sequences in P4 programming language. Our MANE selectively drops queued scalable video packets when the queue occupancy exceeds a threshold. Three packet discarding logics are implemented: (i) tail, (ii) enhancement-layer, and (iii) rate-distortion optimized. Our P4-based MANE implementation is demonstrated in: (i) larger emulated networks in mininet with P4 software switches and (ii) a small real network with a physical P4 switch and multiple Raspberry Pis running P4 software switches.

Mobile devices are getting increasingly popular all over the world. Mobile users in developing countries however rarely have Internet access which puts them at economic and social disadvantages compared to their counterparts in developed countries. We propose mBridge: A distributed system to disseminate multimedia content to mobile users with intermittent Internet access and opportunistic ad hoc connectivity. By disseminating various multimedia content such as news reports notification messages targeted advertisements movie trailers and TV shows mBridge aims to eliminate the digital divide. We formulate an optimization problem to compute personalized distribution plans for individual mobile users to maximize the overall user experience under various resource constraints. Our formulation jointly considers the characteristics of multimedia content mobile users and intermittent networks. We present an efficient distribution planning algorithm to solve our problem and we develop several online heuristics to adapt to the system and network dynamics. We implement a prototype system and demonstrate that our algorithm outperforms the existing algorithms by up to 206% 472% and 188% in terms of user experience disk efficiency and energy efficiency respectively. In addition we conduct trace-driven simulations to rigorously evaluate the proposed system in different environments and for large-scale deployments. Our simulation results demonstrate that the proposed algorithm substantially outperforms the closest ones in the literature in all performance measures. We believe that mBridge can allow multimedia content providers to reach out to more mobile users and mobile users to access multimedia content without always-on Internet access.

International audience

While cloud servers provide a tremendous amount of resources for networked video applications, most successful stories of cloud-assisted video applications are presentational video services, such as YouTube and NetFlix. This article surveys the recent advances on delay-sensitive video computations in the cloud, which are crucial to cloud-assisted conversational video services, such as cloud gaming, Virtual Reality (VR), Augmented Reality (AR), and telepresence. Supporting conversational video services with cloud resources is challenging because most cloud servers are far away from the end users while these services incur the following stringent requirements: high bandwidth, short delay, and high heterogeneity. In this article, we cover the literature with a top-down approach: from applications and experience, to architecture and management, and to optimization in and outside of the cloud. We also point out major open challenges, hoping to stimulate more research activities in this emerging and exciting direction.

Best Papers of the ACM Multimedia Systems (MMSys) Conference 2017 and the ACM Workshop on Network and Operating System Support for Digital Audio and

Watching 360° videos using Head-Mounted Display (HMD) allows users to only see a part of the whole 360° videos. With this feature, tiled videos become a potential solution for aggressively reducing the required bandwidth for 360° video streaming, turning it into a reality in cellular networks. In this paper, we design several experiments for quantifying the performance of tile-based 360° video streaming over a real cellular network on our campus. In particular, we empirically investigate the impacts of tile streaming over 4G networks, such as coding efficiency, bandwidth saving, and scalability. Our experiments lead to interesting findings, for example, (i) only streaming the tiles viewed by the viewer achieves bitrate reduction by up to 80% and (ii) the coding efficiency of 3×3 tiled videos may be higher than non-tiled videos at higher bitrates. We believe this work will stimulate more studies in the emerging area of mobile AR/VR (Augmented Reality and Virtual Reality) over 4G networks.

Modern Internet-of-Things (IoT) applications produce large amount of data and require powerful analytics approaches, such as Deep Learning to extract useful information. Existing IoT applications transmit the data to resource-rich data centers for analytics. However, it may congest networks, overload data centers, and increase security vulnerability. In this paper, we implement a platform, which integrates resources from data centers (servers) to end devices (IoT devices). We launch distributed analytics applications among the devices without sending everything to the data centers. We analyze challenges to implement such a platform and carefully adopt popular open-source projects to overcome the challenges. We then conduct comprehensive experiments on the implemented platform. The results show: (i) the benefits/limitations of distributed analytics, (ii) the importance of decisions on distributing an application across multiple devices, and (iii) the overhead caused by different components in our platform.

Modern IoT analytics are computational and data intensive. Existing analytics are mostly hosted in cloud data centers, and may suffer from high latency, network congestion, and privacy issues. In this paper, we design, implement, and evaluate a fog computing platform that runs analytics in a distributed way on multiple devices, including IoT devices, edge servers, and data-center servers. We focus on the core optimization problem: making deployment decisions to maximize the number of satisfied IoT analytics. We carefully formulate the deployment problem and design an efficient algorithm, named SSE, to solve it. Moreover, we conduct a detailed measurement study to derive system models of the IoT analytics based on diverse QoS levels and heterogeneous devices to facilitate the optimal deployment decisions. We implement a testbed to conduct experiments, which show that the system models achieve reasonably good accuracy. More importantly, 100% of the deployed IoT analytics satisfy the QoS targets. We also conduct extensive simulations for larger-scale scenarios. The simulation results reveal that our SSE algorithm outperforms a state-of-the-art algorithm by up to 89.4% and 168.3% in terms of the number of satisfied IoT analytics and active devices. In addition, our SSE algorithm reduces CPU, RAM, and network resource consumptions by 18.4%, 12.7%, and 898.3%, respectively, and terminates in polynomial time.

Foveated rendering leverages human visual system to increase video quality under limited computing resources for Virtual Reality (VR). More specifically, it increases the frame rate and the video quality of the foveal vision via lowering the resolution of the peripheral vision. Optimizing foveated rendering systems is, however, not an easy task, because there are numerous parameters that need to be carefully chosen, such as the number of layers, the eccentricity degrees, and the resolution of the peripheral region. Furthermore, there is no standard and efficient way to evaluate the Quality of Experiment (QoE) of foveated rendering systems. In this paper, we propose a framework to compare the performance of different subjective assessment methods on foveated rendering systems. We consider two performance metrics: efficiency and consistency, using the perceptual ratio, which is the probability of the foveated rendering is perceivable by users. A regression model is proposed to model the relationship between the human perceived quality and foveated rendering parameters. Our comprehensive study and analysis reveal several insights: 1) there is no absolute superior subjective assessment method, 2) subjects need to make more observations to confirm the foveated rendering is imperceptible than perceptible, 3) subjects barely notice the foveated rendering with an eccentricity degree of 7.5 degrees+ and peripheral region of a resolution of 540p+, and 4) QoE levels are highly dependent on the individuals and scenes. Our findings are crucial for optimizing the foveated rendering systems for future VR applications.

360° videos and Head-Mounted Displays (HMDs) are getting increasingly popular. However, streaming 360° videos to HMDs is challenging. This is because only video content in viewers' Field-of-Views (FoVs) is rendered, and thus sending complete 360° videos wastes resources, including network bandwidth, storage space, and processing power. Optimizing the 360° video streaming to HMDs is, however, highly data and viewer dependent, and thus dictates real datasets. However, to our best knowledge, such datasets are not available in the literature. In this paper, we present our datasets of both content data (such as image saliency maps and motion maps derived from 360° videos) and sensor data (such as viewer head positions and orientations derived from HMD sensors). We put extra efforts to align the content and sensor data using the timestamps in the raw log files. The resulting datasets can be used by researchers, engineers, and hobbyists to either optimize existing 360° video streaming applications (like rate-distortion optimization) and novel applications (like crowd-driven camera movements). We believe that our dataset will stimulate more research activities along this exciting new research direction.

We study the problem of predicting the Field-of-Views (FoVs) of viewers watching 360° videos using commodity Head-Mounted Displays (HMDs). Existing solutions either use the viewer's current orientation to approximate the FoVs in the future, or extrapolate future FoVs using the historical orientations and dead-reckoning algorithms. In this paper, we develop fixation prediction networks that concurrently leverage sensor- and content-related features to predict the viewer fixation in the future, which is quite different from the solutions in the literature. The sensor-related features include HMD orientations, while the content-related features include image saliency maps and motion maps. We build a 360° video streaming testbed to HMDs, and recruit twenty-five viewers to watch ten 360° videos. We then train and validate two design alternatives of our proposed networks, which allows us to identify the better-performing design with the optimal parameter settings. Trace-driven simulation results show the merits of our proposed fixation prediction networks compared to the existing solutions, including: (i) lower consumed bandwidth, (ii) shorter initial buffering time, and (iii) short running time.

Social-network-based Question Answering (Q&A) systems are recently emanated due to their capabilities of outperforming classical search engines in answering non-factual questions. Social network users have different expertises and activity times, and identifying answerers with proper expertises, short response times, and high response rates is challenging for Q&A systems. To address this problem, we propose an optimal Q&A system that identifies answerers with required expertises and routes the questions with minimum possible response time in dynamic social networks. Our proposed system uses a hybrid model for estimating the expertise of each user, in order to identify the suitable answerers; besides, it avoids bottleneck answerers in the network, so as to increase the response rate. We conduct trace- driven simulations, which show that our Q&A system: (i) achieves up to 27% higher average response rate than the state-of-the-art systems, and (ii) reduces the average maximal response time by up to 60%. Moreover, the results show that, by varying the number of answerers, the number of keywords per question, the arrival rate of questions, and the predictability against the maximal response time, our Q&A system consistently outperforms the state-of-the-art systems.

Video conferences over the Internet with multiple participants have to be wirelessly connected to improve the communication efficiency. However, video conferences are often held in places suffering from crowded wireless medium, such as offices and schools. This leads to many challenges on providing high video conferencing experience. In this paper, we study the problem of video streaming over crowded wireless networks considering the interference. We conducted extensive experiments to model the unlicensed spectrum activity and design, implement, and evaluate an interference-aware bandwidth estimation and rate adaptation algorithm. The experiment results show that our proposed solution (i) reduces the retransmission ratio to lower than 6.5%, (ii) reduces the packet loss rate to 0.5% on average, (iii) achieves higher throughput, and (iv) leads to higher video quality than others by at least 8.3 dB in video quality.

Several applications, such as smart cities, smart homes and smart hospitals adopt Internet of Things (IoT) networks to collect data from IoT devices. The incredible growing speed of the number of IoT devices congests the networks and the large amount of data, which are streamed to data centers for further analysis, overload the data centers. In this paper, we implement a fog computing platform that leverages end devices, edge networks, and data centers to serve the IoT applications. In this paper, we focus on implementing a fog computing platform, which dynamically pushes programs to the devices. The programs pushed to the devices pre-process the data before transmitting them over the Internet, which reduces the network traffic and the load of data centers. We survey the existing platforms and virtualization technologies, and leverage them to implement the fog computing platform. Moreover, we formulate a deployment problem of the programs. We propose an efficient heuristic deployment algorithm to solve the problem. We also implement an optimal algorithm for comparisons. We conduct experiments with a real testbed to evaluate our algorithms and fog computing platform. The proposed algorithm shows near-optimal performance, which only deviates from optimal algorithm by at most 2% in terms of satisfied requests. Moreover, the proposed algorithm runs in real-time, and is scalable. More precisely, it computes 1000 requests with 500 devices in <; 2 seconds. Last, the implemented fog computing platform results in real-time deployment speed: it deploys 20 requests <; 10 seconds.

Modern distributed multimedia applications are resource-hungry, and they often leverage on-demand cloud services to reduce their expenses. Existing cloud services deploy many servers in a few data centers, which consume a lot of electricity to power up and cold down, and thus are expensive and environmentally unfriendly. In this paper, we present a multimedia fog computing platform that utilizes resources from public crowds, edge networks, and data centers to serve distributed multimedia applications at lower costs. We use animation rendering as a case study, and identify several challenges for optimizing it on our multimedia fog computing platform. Among these challenges, we focus on the problem of predicting the completion time of each rendering job. We propose an efficient algorithm based on state-of-the-art machine learning algorithms. We also fine-tune the algorithm using multi-fold cross-validation for higher prediction accuracy. With real datasets, we conduct trace-driven simulations to quantify the performance of our prediction algorithm and that of the whole platform. The simulation results show that our proposed algorithm outperforms a state-of the-art statistical model in several aspects: completed job ratio by 20%, makespan by 2 times, and normalized deviation by 30 times, on average. Moreover, the overall performance of the platform with our proposed algorithm is fairly close to that with an Oracle of the actual job completion time: a small factor of 1.48 in terms of makespan is observed.

We propose the very first non-intrusive measurement methodology for quantifying the performance of commodity Virtual Reality (VR) systems. Our methodology considers the VR system under test as a black-box and works with any VR applications. Multiple performance metrics on timing and positioning accuracy are considered, and detailed testbed setup and measurement steps are presented. We also apply our methodology to several VR systems in the market, and carefully analyze the experiment results. We make several observations: (i) 3D scene complexity affects the timing accuracy the most, (ii) most VR systems implement the dead reckoning algorithm, which incurs a non-trivial correction latency after incorrect predictions, and (iii) there exists an inherent trade-off between two positioning accuracy metrics: precision and sensitivity.

Smart Lenses refer to detachable, orientable and zoomable lenses that stream live videos over wireless networks to heterogeneous computing devices, including tablets and smartphones. Various novel applications are made possible by smart lenses, including mobile photography, smart surveillance cameras, and Unmanned Aerial Vehicle (UAV) cameras. However, to our best knowledge, existing smart lenses are closed and proprietary, and thus we initiate an open-source project called Smart Beholder for end-to-end solutions of smart lenses. The code and documents of Smart Beholder can be found at our website http://www.smartbeholder.org. Our Smart Beholder platform are useful to researchers for fast prototyping, developers for rapid development, and amateurs for hobbies. We have implemented Smart Beholder server (camera) using a popular embedded Linux platform, called Raspberry Pi. We have also realized Smart Beholder client (controller) on various OS's, including Android. Our experimental results show the practicality and efficiency of our proposed Smart Beholder: we outperform commercial products in the market in terms of both objective and subjective metrics. We believe the release of Smart Beholder will stimulate future studies on novel multimedia applications enabled by smart lenses.

Providing high-quality video conferencing experience over the best-effort Internet and wireless networks is challenging, because 2D videos are bulky. In this paper, we exploit the common structure of conferencing videos for an ultra-low-bitrate video conferencing system. In particular, we design, implement, optimize, and evaluate a video conferencing system, which: (i) extracts facial landmarks, (ii) transmits the selected facial landmarks and 2D images, and (iii) warps the untransmitted 2D images at the receiver. Several optimization techniques are adopted for minimizing the running time and maximizing the video quality, e.g., the image and warping frames are optimally determined based on network conditions and video content. The experiment results from real conferencing videos reveal that our proposed system: (i) outperforms the state-of-the-art x265 by up to 11.05 dB in PSNR (Peak Signal-to-Noise Ratio), (ii) adapts to different video content and network conditions, and (iii) runs in real-time at about 12 frame-per-second.

Wearable and mobile devices are widely used for crowdsensing, as they come with many sensors and are carried everywhere. Among the sensing data, videos annotated with temporal-spatial metadata contain huge amount of information, but consume too much precious storage space. In this paper, we solve the problem of optimizing cloud-based video crowdsensing in three steps. First, we study the optimal transcoding problem on wearable and mobile cameras. We propose an algorithm to optimally select the coding parameters to fit more videos at higher quality on wearable and mobile cameras. Second, we empirically investigate the throughput of different file transfer protocols from wearable and mobile devices to cloud servers. We propose a real-time algorithm to select the best protocol under diverse network conditions, so as to leverage the intermittent WiFi access. Last, we look into the performance of cloud databases for sensor-annotated videos, and implement a practical algorithm to search videos overlapping with a target geographical region. Our measurement study on three popular opensource cloud databases reveals their pros and cons. The three proposed algorithms are evaluated via extensive simulations and experiments. The evaluation results show the practicality and efficiency of our algorithms and system. For example, our proposed transcoding algorithm outperforms existing approaches by 12 dB in video quality, 87% in energy saving, and one-quarter in delivery delay. Another example is, by intelligently choosing a proper cloud database, our system may reduce the insertion time by up to one-third, or the lookup time by up to one-fourth.

In this article, we have classified cloud gaming platforms into three types based on how games are integrated with platforms. We have also reviewed the history of cloud gaming services, and noted that it is a key moment for cloud gaming services to increase their penetration rates. Last, built upon our extensive research experience in cloud gaming, we share several of our visions into future cloud gaming technologies, business models, and social impacts, in the format of forecasts. While our forecasts may not be an exhausted list, we firmly believe this article will stimulate more discussions among cloud gaming researchers and practitioners, resulting in a sustainable cloud gaming ecosystem.

Existing channel-aware scheduling work has mainly focused on scheduling in small timescales, that is, tens to hundreds of seconds. We propose to use long-term user profiles to provide useful statistical information on future network conditions in large timescales. We design scheduling algorithms based on Markov decision theory. We collect and use a large set of real-life traces from the general public. Extensive trace-driven evaluations show that many real mobile users can benefit from our framework. In addition, we compare our framework against state-of-the-art algorithms and observe significant performance differences because the existing algorithms were not designed for the large timescale scenario.

Current cellular networks support video streaming over unicast or multicast. However, there exists a tradeoff between utilizing the two: i) unicast leads to higher network load, but lower energy consumption of mobile devices, and ii) multicast results in lower network load, but higher energy consumption. To make the best out of both, we propose to concurrently utilize unicast and multicast for minimizing the energy consumption of mobile devices and minimizing the load on cellular networks. Cellular networks support two multicast schemes: i) independent cell networks and ii) multi-cell single frequency networks, where multiple adjacent base stations operate on the same frequency. We first consider the less-complicated independent cell networks, and then extend our solution to single frequency networks for better performance. We formulate the resource allocation in hybrid multicast -unicast streaming systems as a binary integer programming problem. We describe optimal algorithms for the two multicast schemes. We then propose two efficient, heuristic, algorithms that run faster and provide close to optimal results. While our solution is general, for concreteness, we conduct detailed LTE packet-level simulations using OPNET. Our simulation results show the proposed algorithms i) scale to many more mobile devices than the state-of-the-art unicast-only approaches and ii) result in lower energy consumption than the latest multicast-only approaches. In addition, the algorithms designed for multi-cell single frequency networks outperform the algorithms designed for independent cell networks in all aspects, such as service ratio, spectral efficiency, energy saving, video quality, frame loss rate, initial buffering time, and number of re-buffering events.

In this article, we design a crowdsourcing system, CrowdMAC, where mobile devices form a local community or marketplace to share network access and transfer data for each other. CrowdMAC enables (i) mobile clients to select and exploit multiple mobile hotspots in its vicinity for data transfer and (ii) mobile hotspots to open their cellular connectivity to admit/serve delay-bounded requests from mobile users for a fee. The evaluations of CrowdMAC indicate that (i) mobile clients can tune preferred trade-offs between cost and delay through a control knob, (ii) mobile hotspots comply with all delay bounds, and (iii) the system ensures stable and efficient transfer.

The binding between computing devices and displays is becoming dynamic and adaptive, and screencast technologies enable such binding over wireless networks. In this article, we design and conduct the first detailed measurement study on the performance of the state-of-the-art screencast technologies. Several commercial and one open-source screencast technologies are considered in our detailed analysis, which leads to several insights: (1) there is no single winning screencast technology, indicating room to further enhance the screencast technologies; (2) hardware video encoders significantly reduce the CPU usage at the expense of slightly higher GPU usage and end-to-end delay, and should be adopted in future screencast technologies; (3) comprehensive error resilience tools are needed as wireless communication is vulnerable to packet loss; (4) emerging video codecs designed for screen contents lead to a better Quality of Experience (QoE) of screencast; and (5) rate adaptation mechanisms are critical to avoiding degraded QoE due to network dynamics. As a case study, we propose a nonintrusive yet accurate available bandwidth estimation mechanism. Real experiments demonstrate the practicality and efficiency of our proposed solution. Our measurement methodology, open-source screencast platform, and case study allow researchers and developers to quantitatively evaluate other design considerations, which will lead to optimized screencast technologies.

Software-Defined Networking (SDN) is an emerging network architecture that enables network programmability. Recent research activities on SDN make it important to develop an emulator that accurately emulates OpenFlow-enabled SDN networks. However, existing emulators and simulators focus on either data plane performance or software switches. This motivates us to develop a network emulator that provides accurate emulation on both control plane and data plane performance of an OpenFlow network and supports all types of OpenFlow switches. In this paper, we design a configurable emulator that allows users to input configurations and performance characteristics of different OpenFlow switches. Based on the performance measurement results from OpenFlow switches, we derive the configurable parameters and propose an accurate control plane performance model. Our implementation is based on a popular opensource OpenFlow emulator, Mininet with OpenvSwitch (OVS). The evaluation results show that the emulator we developed provides better performance fidelity than original Mininet/OVS.

Software Defined Networking (SDN) has been a prevalent networking paradigm in recent years. Due to its simplicity to quickly deploy and fully manage networks, both academia and industry are interested in this technology. Nevertheless, there are still many issues to be solved. In this paper, we propose to divide flow table into two separate tables, namely path table and traffic table, to minimize flow initialization latency in SDN. We designed a SDN controller system following this concept and proposed two heuristic algorithms to calculate the two tables. The algorithms take both initialization latency and link utilization into consideration. We implement the proposed system on Ryu controller and perform experiments in a Mininet-based testbed. The experimental results show that our proposed solution outperforms the default spanning tree protocol (STP) application on Ryu controller by reducing: (i) the initialization latency up to 60%, (ii) the maximal link utilization up to 4.78%, and (iii) the average link utilization up to 1.45%.

As the huge growth of mobile traffic amount, conventional Radio Access Networks (RANs) suffer from high capital and operating expenditures, especially when new cellular standards are deployed. Software, and cloud RANs have been proposed, but the stringent latency requirements e.g., 1 ms transmission time interval, dictated by cellular networks is difficult to satisfy. We first present a real software RAN testbed based on an opensource LTE implementation. We also investigate the issue of quality assurance when deploying such software RANs in cloud. In particular, running software RANs in cloud leads to high latency, which may violate the latency requirements. We empirically study the problem of minimizing computational and networking latencies in lightweight container cloud. Our experiment results show the feasibility of running software RANs in real-time container cloud. More specifically, a feasible solution to host software RANs in cloud is to adopt lightweight containers with real-time kernels and fast packet processing networking.

In this paper, we introduce Smart Source, a crowd sourcing based mobile Question & Answer (Q&A) system that aims to provide mobile information seekers with timely, trustworthy and accurate answers while ensuring that information providers are not inappropriately burdened. We tackle this challenge by taking advantage of both static and dynamic context and semantics from mobile users (e.g., Geolocation, social network, expertise/interest, device sensor profiles, battery level) to identify sources of information (i.e., Workers) that are trusted by the user and accurate enough for the questions at hand. Given a question, the Smart Source broker middleware executes a scalable and efficient worker selection algorithm that uses a Lyapunov optimization framework to maximize the utility of worker selection while guaranteeing the stability of the overall system. An associated assignor selection is used to scale the selection process to a large number of users. We implement the Smart Source prototype system on an Android test bed and thoroughly evaluate the system using real world applications and data, in particular those that involve geospatial questions and answers. Evaluation results indicate that Smart Source is efficient and provides superior worker selection compared to baseline approaches. Smart Source is also highly customizable: it employs a general utility function and provides a control knob to tradeoff the optimality and responding time. We believe that Smart Source will pave a way for new mechanisms of interaction among mobile users.

We present a complete system, called Challenged Content Delivery Network (CCDN), to efficiently deliver multimedia content to mobile users who live in developing countries, rural areas, or over-populated cities with no or weak network infrastructure. These mobile users do not have always-on Internet access. We demo our CCDN, implemented on a Linux server, Raspberry Pi proxies, and Android phones from three aspects: multimedia, networking, and machine learning tools. We propose multiple optimization algorithm modules that compute personalized distribution plans, and maximize the overall user experience. CCDN allows people living in area with challenged networks access to multimedia content, like news reports, using mobile devices, such as smartphones. This in turn will help in eliminating the digital divide, which refers to information inequality to persons with different Internet accessing abilities.

Smart lenses are detachable lenses connected to mobile devices via wireless networks, which are not constrained by the small form factor of mobile devices, and have potential to deliver better photo (video) quality. However, the viewfinder previews of smart lenses on mobile devices are difficult to optimize, due to the strict resource constraints on smart lenses and fluctuating wireless network conditions. In this paper, we design, implement, and evaluate an open-source smart lens, called Smart Beholder. It achieves three design goals: (i) cost effectiveness, (ii) low interaction latency, and (iii) high preview quality by: (i) selecting an embedded system board that is just powerful enough, (ii) minimizing per-component latency, and (iii) dynamically adapting the video coding parameters to maximizing Quality of Experience (QoE), respectively. Several optimization techniques, such as anti-drifting mechanism for video frames and QoE-driven resolution/frame rate adaptation algorithm, are proposed in this paper. Our measurement study shows that Smart Beholder outperforms Altek Cubic and Sony QX100 in terms of lower bitrate, lower latency, slightly higher frame rate, and better preview quality. We also demonstrate that sys adapts to network dynamics. Smart Beholder has been made public at http://www.smartbeholder.org as an experimental platform for researchers and developers to optimize smart lenses and other embedded real-time video streaming systems.

Dynamic and adaptive binding between computing devices and displays is increasingly more popular, and screencast technologies enable such binding over wireless networks. In this paper, we design and conduct the first detailed measurement study on the performance of the state-of-the-art screencast technologies. Several commercial and one open-source screencast technologies are considered in our detailed analysis, which leads to several insights: (i) there is no single winning screencast technology, indicating rooms to further enhance the screencast technologies, (ii) hardware video encoders significantly reduce the CPU usage at the expense of slightly higher GPU usage and end-to-end delay, and should be adopted in future screencast technologies, (iii) comprehensive error resilience tools are needed as wireless communication is vulnerable to packet loss, (iv) emerging video codecs designed for screen contents lead to better Quality of Experience (QoE) of screencast, and (v) rate adaptation mechanisms are critical to avoiding degraded QoE due to network dynamics. Furthermore, our measurement methodology and open-source screencast platform allow researchers and developers to quantitatively evaluate other design considerations, which will lead to optimized screencast technologies.

Optimizing cloud gaming experience is no easy task due to the complex tradeoff between gamer quality of experience (QoE) and provider net profit. We tackle the challenge and study an optimization problem to maximize the cloud gaming provider's total profit while achieving just-good-enough QoE. We conduct measurement studies to derive the QoE and performance models. We formulate and optimally solve the problem. The optimization problem has exponential running time, and we develop an efficient heuristic algorithm. We also present an alternative formulation and algorithms for closed cloud gaming services with dedicated infrastructures, where the profit is not a concern and overall gaming QoE needs to be maximized. We present a prototype system and testbed using off-the-shelf virtualization software, to demonstrate the practicality and efficiency of our algorithms. Our experience on realizing the testbed sheds some lights on how cloud gaming providers may build up their own profitable services. Last, we conduct extensive trace-driven simulations to evaluate our proposed algorithms. The simulation results show that the proposed heuristic algorithms: (i) produce close-to-optimal solutions, (ii) scale to large cloud gaming services with 20,000 servers and 40,000 gamers, and (iii) outperform the state-of-the-art placement heuristic, e.g., by up to 3.5 times in terms of net profits.

We study the problem of optimally adapting ongoing cloud gaming sessions to maximize the gamer experience in dynamic environments. The considered problem is quite challenging because: 1) gamer experience is subjective and hard to quantify; 2) the existing open-source cloud gaming platform does not support dynamic reconfigurations of video codecs; and 3) the resource allocation among concurrent gamers leaves a huge room to optimize. We rigorously address these three challenges by: 1) conducting a crowdsourced user study over the live Internet for an empirical gaming experience model; 2) enhancing the cloud gaming platform to support frame rate and bitrate adaptation on-the-fly; and 3) proposing optimal yet efficient algorithms to maximize the overall gaming experience or ensure the fairness among gamers. We conduct extensive trace-driven simulations to demonstrate the merits of our algorithms and implementation. Our simulation results show that the proposed efficient algorithms: 1) outperform the baseline algorithms by up to 46% and 30%; 2) run fast and scale to large (≤8000 gamers) problems; and 3) achieve the user-specified optimization criteria, such as maximizing average gamer experience or maximizing the minimum gamer experience. The resulting cloud gaming platform can be leveraged by many researchers, developers, and gamers.

We focus on the problem of efficient communications over access networks with asymmetric bandwidth and capability. We propose a resource-constrained asymmetric redundancy elimination algorithm (RCARE) to leverage downlink bandwidth and receiver capability to accelerate the uplink data transfer. RCARE can be deployed on a client or a proxy. Different from existing asymmetric algorithms, RCARE uses a flexible matching mechanism to identify redundant data and allocates a small sender cache to absorb the high downlink traffic overhead. Compared to existing redundancy elimination algorithms, RCARE provides a scalable sender cache that is adaptive based on resource and performance. We evaluate RCARE with real traffic traces collected from multiple servers and a campus gateway. The trace-driven simulation results indicate that RCARE achieves higher goodput gains and reduces downlink traffic compared to existing asymmetric communication algorithms. We design an adaptation algorithm for resource-constrained senders sending multiple data streams. Our algorithm takes samples from data streams and predicts how to invest cache size on individual data streams to achieve maximal uplink goodput gain. The adaptation algorithm improves the goodput gain by up to 87% compared to the baseline. In first 10% of data streams (sorted by the optimal goodput gains), RCARE achieves up to 42% goodput gain on average.

Marathon is very popular in recent years. However, finishing the game is no easy task, especially for beginners. Regular practices and training are needed. With the availability of wearable devices, it is possible to develop virtual coaches that monitor the progresses of individual runners closeup and guide them through tailor-made training schedules. Unfortunately, most existing wearable devices only record physiological signals of the runners and rely on off-line processing to provide feedbacks. In this paper, we present the design and development of an on-line virtual coach, which performs real-time tracking and analysis of the physiological status of the runner and suggests appropriate adjustments on the exercise intensity. The proposed virtual coach is a pure software solution and can work with any wearable device that monitors the heart rate and running speed of the runner. The main challenge of our system is to predict when the runner will reach the various running states and instruct the runner to adjust the speed just ahead of time so that her/his body can react in time to maintain the required training intensity. Experiments on real users show that our proposed algorithms can correctly predict the running states of the runners and help them to better maintain the required intensity to maximize the training effects.

In this paper, we explore the problem of supporting efficient access to social media contents on social network sites for mobile devices without requiring mobile users to be online all the time. We propose and implement a broker/proxy based architecture that stages data at a broker/proxy, and selectively downloads to the mobile device only those contents that have a high likelihood of being viewed. The system determines the relevance of social media updates that continuously arrive (e.g., Facebook friend updates) for each user. Using knowledge of this relevance and current network/system conditions, we develop scheduling algorithms that determine which social contents are sent to the devices. We develop an Android app providing offline access to Facebook. Our experimental results indicate that our system is energy efficient, which saves energy by 6.9 times for WiFi and 9.1 times for cellular connections. We also use data traces gathered from our app to further drive extensive simulation based evaluations which show that our proposed algorithms provide efficient facilities for tuning the system's performance.

The popularity of GPS sensors in mobile devices has enabled a new generation of location-based services, in which users record their traveling routes and upload the GPS track points (GTPs) on-the-fly for real-time social sharing, jog journaling, life logging, health care, and map generation. However, directly uploading all GTPs wastes battery energy and network bandwidth on the mobile device, because GTPs are highly redundant. To reduce the redundancies, we propose in this paper the Corner Extraction Technique (CET) to extract the corner points from the GTPs of individual users that can be used to characterize and reconstruct the routes that the user has traveled. Storing corner GTPs requires much less storage space, and transmitting them allows saving in both network bandwidth and energy consumption, leading to optimized GPS-enabled mobile applications. We have conducted both trace-driven simulations and real experiments to demonstrate the merits of the proposed CET approach. The experimental results indicate that: (1) CET results in up to 33 times of compression ratio, (2) CET closely follows the original road segments, and (3) CET saves energy consumption by up to 72%.

IP multicast dictates high-end routers and incurs high administrative overhead, which prevent them from being deployed in many video streaming scenarios. In this paper, we study the problem of computing the multipath multicast routes for streaming videos in Software-Defined Networks (SDNs), which adopt less expensive switches and reduce administrative overhead for lower CAPEX/OPEX. The objectives of the considered problem are robustness, load balance, SDN compatibility, and adaptiveness. We formulate this routing problem into a mathematical optimization problem, and propose two algorithms to address this problem. We implement the proposed algorithms on a popular OpenFlow controller to demonstrate its practicality, and we conduct extensive experiments to evaluate the proposed algorithms. The experiment results clearly show the merits of our algorithms over the IP multicast, e.g., we observe: (i) frame loss rate reduction between 19% and 95%, (ii) video quality improvement between 4 dB and 15 dB, (iii) sink throughput increase between 25% and 66%, and (iv) maximal link utilization reduction between 15% and 50%. We also show the tradeoff between optimality and run time of the two proposed algorithms: one of them is more suitable for smaller and more static networks, and the other one is more suitable for larger and more dynamic networks.

Displays without associated computing devices are increasingly more popular, and the binding between computing devices and displays is no longer one-to-one but more dynamic and adaptive. Screencast technologies enable such dynamic binding over ad hoc one-hop networks or Wi-Fi access points. In this paper, we design and conduct the first detailed measurement study on the performance of state-of-the-art screencast technologies. By varying the user demands and network conditions, we find that Splashtop and Miracast outperform other screencast technologies under typical setups. Our experiments also show that the screencast technologies either: (i) do not dynamically adjust bitrate or (ii) employ a suboptimal adaptation strategy. The developers of future screencast technologies are suggested to pay more attentions on the bitrate adaptation strategy, e.g., by leveraging cross-layer optimization paradigm.

Mobile on-demand videos are getting tremendously popular and incurring staggering overhead on cellular net-works. Fortunately, next generation cellular networks support video streaming over either unicast or multicast, but how to capitalize both unicast and multicast for optimal on-demand video streaming remains an open question. In this paper, we consider a resource allocation problem that concurrently utilizes unicast/multicast in order to support many more mobile streaming users and minimize the energy consumption of the battery-powered mobile devices. We formulate this problem as a Binary Integer Programming (BIP) problem. We present an optimal algorithm, SCOPT, for this problem. We also develop an efficient heuristic algorithm, SCG, for lower overhead. We conduct detailed packet-level simulations to evaluate the algorithms in LTE networks using OPNET. Our simulation study shows that the proposed algorithms: (i) result in lower energy consumption than multicast-only approach, (ii) scale to many more mobile users than unicast-only approach, and (iii) are more energy efficient with more network bandwidth or fewer videos. In addition, we discuss how our solution can be extended to support Single Frequency Networks in which multiple adjacent base stations operate on the same frequency.

MultiNets is a system supporting seamless switch-over between wireless interfaces on mobile devices in real-time. MultiNets is configurable to run in three different modes: (i) Energy Saving mode --for choosing the interface that saves the most energy based on the condition of the device, (ii) Offload mode --for offloading data traffic from the cellular to WiFi network, and (iii) Performance mode --for selecting the network for the fastest data connectivity. MultiNets also provides a powerful API that gives the application developers: (i) the choice to select a network interface to communicate with a specific server, and (ii) the ability to simultaneously transfer data over multiple network interfaces. MultiNets is modular, easily integrable, lightweight, and applicable to various mobile operating systems. We implement MultiNets on Android devices as a show case. MultiNets does not require any extra support from the network infrastructure and runs existing applications transparently. To evaluate MultiNets, we first collect data traces from 13 actual Android smartphone users over three months. We then use the collected traces to show that, by automatically switching to WiFi whenever it is available, MultiNets can offload on average 79.82% of the data traffic. We also illustrate that, by optimally switching between the interfaces, MultiNets can save on average 21.14 KJ of energy per day, which is equivalent to 27.4% of the daily energy usage. Using our API, we demonstrate that a video streaming application achieves 43--271% higher streaming rate when concurrently using WiFi and 3G interfaces. We deploy MultiNets in a real-world scenario and our experimental results show that depending on the user requirements, it outperforms the state-of-the-art Android system either by saving up to 33.75% energy, achieving near-optimal offloading, or achieving near-optimal throughput while substantially reducing TCP interruptions due to switching.

We study the problem of disseminating videos to mobile users by using a hybrid cellular and ad hoc network. In particular, we formulate the problem of optimally choosing the mobile devices that will serve as gateways from the cellular to the ad hoc network, the ad hoc routes from the gateways to individual devices, and the layers to deliver on these ad hoc routes. We develop a Mixed Integer Linear Program (MILP)-based algorithm, called POPT, to solve this optimization problem. We then develop a Linear Program (LP)-based algorithm, called MTS, for lower time complexity. While the MTS algorithm achieves close-to-optimum video quality and is more efficient than POPT in terms of time complexity, the MTS algorithm does not run in real time for hybrid networks with large numbers of nodes. We, therefore, propose a greedy algorithm, called THS, which runs in real time even for large hybrid networks. We conduct extensive packet-level simulations to compare the performance of the three proposed algorithms. We found that the THS algorithm always terminates in real time, yet achieves a similar video quality to MTS. Therefore, we recommend the THS algorithm for video dissemination over hybrid cellular and ad hoc networks.

Cloud gaming, i.e., real-time game playing via thin clients, relieves users from being forced to upgrade their computers and resolve the incompatibility issues between games and computers. As a result, cloud gaming is generating a great deal of interests among entrepreneurs, venture capitalists, general publics, and researchers. However, given the large design space, it is not yet known which cloud gaming system delivers the best user-perceived Quality of Service (QoS) and what design elements constitute a good cloud gaming system. This study is motivated by the question: How good is the QoS of current cloud gaming systems? Answering the question is challenging because most cloud gaming systems are proprietary and closed, and thus their internal mechanisms are not accessible for the research community. In this paper, we propose a suite of measurement techniques to evaluate the QoS of cloud gaming systems and prove the effectiveness of our schemes using a case study comprising two well-known cloud gaming systems: OnLive and StreamMyGame. Our results show that OnLive performs better, because it provides adaptable frame rates, better graphic quality, and shorter server processing delays, while consuming less network bandwidth. Our measurement techniques are general and can be applied to any cloud gaming systems, so that researchers, users, and service providers may systematically quantify the QoS of these systems. To the best of our knowledge, the proposed suite of measurement techniques have never been presented in the literature.

We present the first open source cloud gaming system, called GamingAnywhere. In addition to its openness, we have designed, GamingAnywhere for high extensibility, portability, and reconfigurability. We implemented it on Windows, Linux, OS X, and Android. We conducted extensive experiments to evaluate its performance. Our experimental results indicate that GamingAnywhere is efficient, scalable, adaptable to network conditions, and achieves high responsiveness and streaming quality. GamingAnywhere can be employed by researchers, game developers, service providers, and end users for setting up cloud gaming testbeds, which we believe, will stimulate more research into innovations for cloud gaming systems and applications.

We design, implement, and evaluate an H.264/SVC decoder and an HTTP video streaming client on multi-core mobile devices. The decoder employs multiple decoder threads to leverage the multi-core CPUs, and the streaming server/client support adaptive HTTP video streaming. To evaluate the decoder performance, we conduct experiments using real H.264/SVC videos on a tablet and a smart phone running Android 4.0. Our experimental results demonstrate that real-time H.264/SVC decoding is feasible on multi-core mobile devices. For example, for 960×544 videos, our decoder achieves up to 20.72 FPS (Frame-Per-Second), and for 480×272 videos, it achieves up to 42.03 FPS. We also conduct extensive HTTP video streaming experiments over live WiFi and 3G cellular networks, which show that high frame rate (up to ~42 FPS), and short initial delay (as small as ~2.5 sec) are possible. We make our testbed publicly available to the research communities.

While cloud gaming opens new business opportunity, it also poses tremendous challenges as the Internet only provides best-effort service and gamers are hard to please. Although researchers have various ideas to improve cloud gaming systems, existing cloud gaming systems are closed and proprietary, and cannot be used to evaluate these ideas. We present GamingAnywhere, the first open-source cloud gaming system, which is extensible, portable, and configurable. GamingAnywhere may be used by: (i) researchers and engineers to implement and test their new ideas, (ii) service providers to develop cloud gaming services, and (iii) gamers to set up private cloud gaming systems. Details on GamingAnywhere are given in this paper. We firmly believe GamingAnywhere will stimulate future studies on cloud gaming and real-time interactive distributed systems.

Cloud gaming is a promising application of the rapidly expanding cloud computing infrastructure. Existing cloud gaming systems, however, are closed-source with proprietary protocols, which raises the bars to setting up testbeds for experiencing cloud games. In this paper, we present a complete cloud gaming system, called GamingAnywhere, which is to the best of our knowledge the first open cloud gaming system. In addition to its openness, we design GamingAnywhere for high extensibility, portability, and reconfigurability. We implement GamingAnywhere on Windows, Linux, and OS X, while its client can be readily ported to other OS's, including iOS and Android. We conduct extensive experiments to evaluate the performance of GamingAnywhere, and compare it against two well-known cloud gaming systems: OnLive and StreamMyGame. Our experimental results indicate that GamingAnywhere is efficient and provides high responsiveness and video quality. For example, GamingAnywhere yields a per-frame processing delay of 34 ms, which is 3+ and 10+ times shorter than OnLive and StreamMyGame, respectively. Our experiments also reveal that all these performance gains are achieved without the expense of higher network loads. The proposed GamingAnywhere can be employed for setting up cloud gaming testbeds, which, we believe, will stimulate more research innovations on cloud gaming systems.

Over the past few years, personal sensing applications, such as travel path sharing and location recording, have been more and more popular. These applications use GPS sensors to record track points on smartphones and upload the track points to clouds in real time for information sharing. However, uploading a lot of GPS points may lead to heavy network traffic and much higher power consumption. To address the problem, we present corner extraction by GPS filtering (CEGF) that extracts corner feature GPS points (CFGPs) from GPS track points (GTPs). Applications only need to upload the CFGPs to save the uploading energy on smartphones. CFGPs can be regarded as characteristic points of corners to represent the corresponding roads. To reduce the number of uploaded points, we use CEGF to filter out the CFGPs from a large amount of GTPs.

No abstract available.

No abstract available.

No abstract available.

We propose region- and action-aware virtual world clients. To develop such clients, we present a parameterized network traffic model, based on a large collection of Second Life traces gathered by us. Our methodology is also applicable to virtual worlds other than Second Life. With the traffic model, various optimization criteria can be adopted, including visual quality, response time, and energy consumption. We use energy consumption as the show case, and demonstrate via trace-driven simulations that, compared to two existing schemes, a mobile client can save up to 36% and 41% communication energy by selectively turning on its WiFi network interface.

We consider the problem of scalable video streaming from a server to multinetwork clients over heterogeneous access networks, with the goal of minimizing the distortion of the received videos. This problem has numerous applications including: 1) mobile devices connecting to multiple licensed and ISM bands, and 2) cognitive multiradio devices employing spectrum bonding. In this paper, we ascertain how to optimally determine which video packets to transmit over each access network. We present models to capture the network conditions and video characteristics and develop an integer program for deterministic packet scheduling. Solving the integer program exactly is typically not computationally tractable, so we develop heuristic algorithms for deterministic packet scheduling, as well as convex optimization problems for randomized packet scheduling. We carry out a thorough study of the tradeoff between performance and computational complexity and propose a convex programming-based algorithm that yields good performance while being suitable for real-time applications. We conduct extensive trace-driven simulations to evaluate the proposed algorithms using real network conditions and scalable video streams. The simulation results show that the proposed convex programming-based algorithm: 1) outperforms the rate control algorithms defined in the Datagram Congestion Control Protocol (DCCP) by about 10–15 dB higher video quality; 2) reduces average delivery delay by over 90% compared to DCCP; 3) results in higher average video quality of 4.47 and 1.92 dB than the two developed heuristics; 4) runs efficiently, up to six times faster than the best-performing heuristic; and 5) does indeed provide service differentiation among users.

We consider the problem of asymmetric communications, which are common in many access networks. We propose a new asymmetric communication algorithm, called CacheQuery, to leverage on the already deployed downlink bandwidth and receiver capability to accelerate the uplink data transfer from one or multiple senders to a receiver. The design of CacheQuery differs from all previous asymmetric communication algorithms in two ways: (i) CacheQuery supports more flexible matching mechanism to identify redundant packet payload and (ii) CacheQuery allocates a small sender cache to absorb the potentially high downlink traffic overhead incurred by asymmetric communications. The trace-driven simulations indicate that, compared to existing asymmetric communication algorithms, CacheQuery achieves higher uplink transfer speed, yet reduces downlink traffic overhead.

Asymmetric access networks with downlink bandwidth 10-1000 times higher than uplink bandwidth are common, and upgrading these links may not be economically feasible in many usage scenarios. In this paper, we design, implement, and evaluate a parameterized Asymmetric Communication Layer (ACL) protocol that capitalizes the otherwise idling downlink bandwidth to boost the uplink goodput. The ACL protocol is different from existing techniques such as caching, protocol-independent redundancy elimination, WAN optimization, and online compression algorithms, because the ACL protocol is the first concrete network protocol to increase uplink goodput using downlink bandwidth. We implement the ACL protocol in ns-2 simulator, and conduct extensive simulations using synthetic traffic traces. The simulation results show the potential of the ACL protocol: compared to plain TCP/IP, 40-90% uplink goodput gain is possible. Users can also trade the goodput gain for lower delivery delay, as low as 1 RTT, or lower downlink payload amount, as low as 2.13 times of the actual data size. We also conduct a trace-driven simulation using a one-week real traffic trace of a Web server. We observe 24% uplink goodput gain, which shows the practicality of the ACL protocol.

Today's mobile Internet is heavily overloaded by the increasing demand and capability of mobile devices, in particular, multimedia traffic. However, not all traffic is created equal, and a large portion of multimedia contents on the mobile Internet is delay tolerant. We study the problem of capitalizing the content transfer opportunities under better network conditions via postponing the transfers without violating the user-specified deadlines. We propose a new framework called SmartTransfer, which offers a unified content transfer interface to mobile applications. We also develop two scheduling algorithms to opportunistically schedule the content transfers. Via extensive trace-driven simulations, we show that our algorithms outperform a baseline scheduling algorithm by far: up to 17 times improvement in upload throughput and/or at most 20 dBm boost in signal strength. The simulation results also reveal various tradeoff between the two proposed scheduling algorithms. We have implemented our framework and one of the scheduling algorithms on Android, to demonstrate their practicality and efficiency.

In this paper we present Multi Nets, a system which is capable of switching between wireless network interfaces on mobile devices in real-time. Multi Nets is motivated by the need of smart phone platforms to save energy, offload data traffic, and achieve higher throughput. We describe the architecture of Multi Nets and demonstrate the methodology to perform switching in Linux based mobile OSes such as Android. Our analysis on mobile data traces collected from real users shows that with real-time switching we can save 27.4% of the energy, offload 79.82% of the data traffic, or achieve 7 times more throughput on average. We deploy Multi Nets in a real world scenario and our experimental results show that depending on the user requirements, it outperforms the state-of-the-art Android system either by saving up to 33.75% energy, or achieving near-optimal offloading, or achieving near-optimal throughput while substantially reducing TCP interruptions due to switching.

We design, implement, and evaluate a middleware system, HybCAST, that leverages a hybrid cellular and ad hoc network to disseminate rich contents from a source to all mobile devices in a predetermined region. HybCAST targets information dissemination over a range of scenarios (e.g., military operations, crisis alerting, and popular sporting events) in which high reliability and low latency are critical and existing fixed infrastructures such as wired networks, 802.11 access points are heavily loaded or partially destroyed. HybCAST implements a suite of protocols that: (i) structures the hybrid network into a hierarchy of two-level ad hoc clusters for better scalability, (ii) employ both data push and pull mechanisms for high reliability and low latency dissemination of rich content, and (iii) implement a near-optimal gateway selection algorithm to minimize the transmission redundancy. To demonstrate its practicality and efficiency, we have implemented and deployed the HybCAST middleware on several Android smart phones and an in-network Linux machine that acts as a dissemination server. The system is evaluated via real experiments using a UMTS network and extensive packet-level simulations. Our experimental results from a live network show that HybCAST achieves 100% reliability with shorter latencies and lower overall energy consumption. Simulation results confirm that HybCAST outperforms other state-of-the-art systems in the literature. For example, HybCAST exhibits a 5 times reduction in the dissemination latencies as compared to other hybrid dissemination protocols, while its energy consumption is a third of a cellular-only dissemination system. Furthermore, the simulation results demonstrate that HybCAST scales well and maintains good performance under varying numbers of mobile devices, diverse content sizes, and device mobility.

This article presents the design of a complete, open-source, testbed for broadcast networks that offer mobile TV services. Although basic architectures and protocols have been developed for such networks, detailed performance tuning and analysis are still needed, especially when these networks scale to serve many diverse TV channels to numerous subscribers. The detailed performance analysis could also motivate designing new protocols and algorithms for enhancing future mobile TV networks. Currently, many researchers evaluate the performance of mobile TV networks using simulation and/or theoretical modeling methods. These methods, while useful for early assessment, typically abstract away many necessary details of actual, fairly complex, networks. Therefore, an open-source platform for evaluating new ideas in a real mobile TV network is needed. This platform is currently not possible with commercial products, because they are sold as black boxes without the source code. In this article, we summarize our experiences in designing and implementing a testbed for mobile TV networks. We integrate off-the-shelf hardware components with carefully designed software modules to realize a scalable testbed that covers almost all aspects of real networks. We use our testbed to empirically analyze various performance aspects of mobile TV networks and validate/refute several claims made in the literature as well as discover/quantify multiple important performance tradeoffs.

Retrieving information about books on a bookshelf by snapping a photo of book spines with a mobile device is very useful for bookstores, libraries, offices, and homes. In this paper, we develop a new mobile augmented reality system for book spine recognition. Our system achieves very low recognition delays, around 1 second, to support real-time augmentation on a mobile device's viewfinder. We infer user interest by analyzing the motion of objects seen in the viewfinder. Our system initiates a query during each low-motion interval. This selection mechanism eliminates the need to press a but ton and avoids using degraded motion-blurred query frames during high-motion intervals. The viewfinder is augmented with a book's identity, prices from different vendors, average user rating, location within the enclosing bookshelf, and a digital compass marker. We present a new tiled search strategy for finding the location in the bookshelf with improved accuracy in half the time as in a previous state-of-the-art system. Our AR system has been implemented on an Android smartphone.

Virtual worlds, such as Second Life, are computer-simulated spaces divided into multiple regions, in which each user controls an avatar to perform actions (such as run and fly) in order to interact with other users. Second Life incurs diverse traffic patterns in different regions and with different actions. Hence, we propose region- and action-aware Second Life clients, which adapt to and take advantages of the diverse traffic patterns for user-specified optimization criterion, such as high visual quality, low energy consumption, and short response time. To achieve this, we develop a parameterized traffic model to predict Second Life traffic patterns. We systematically derive the traffic model parameters using public Second Life traces [1], and we validate the model accuracy using another set of real network traces. To the best of our knowledge, region- and action-aware virtual world clients have never been considered in the literature. In addition, the proposed parameterized traffic model is of interest in its own right to various parties, including: (i) virtual world developers, (ii) researchers, and (iii) Internet Service Providers (ISPs).

The emerging cloud gaming service has been growing rapidly, but not yet able to reach mobile customers due to many limitations, such as bandwidth and latency. We introduce a 3D image warping assisted real-time video coding method that can potentially meet all the requirements of mobile cloud gaming. The proposed video encoder selects a set of key frames in the video sequence, uses the 3D image warping algorithm to interpolate other non-key frames, and encodes the key frames and the residues frames with an H.264/AVC encoder. Our approach is novel in taking advantage of the run-time graphics rendering contexts (rendering viewpoint, pixel depth, camera motion, etc.) from the 3D game engine to enhance the performance of video encoding for the cloud gaming service. The experiments indicate that our proposed video encoder has the potential to beat the state-of-art x264 encoder in the scenario of real-time cloud gaming. For example, by implementing the proposed method in a 3D tank battle game, we experimentally show that more than 2 dB quality improvement is possible.

Despite the successful use of local image features for large-scale object recognition, they are not effective in recognizing book spines on bookshelves. This is because some book spines contain only text components that do not yield distinguishing image features. To overcome this issue, we develop a new approach that combines a text-based spine recognition pipeline with an image feature-based spine recognition pipeline. The text within the book spine image is recognized and used as keywords to search a book spine text database. The image features of the book spine image are searched through a book spine image database. The search results of the two approaches are then carefully combined to form the final result. We implement the proposed hybrid book recognition pipeline used in a book inventory management system, and conduct extensive experiments to evaluate its performance. The experimental results show that while text-based or image feature-based systems only achieve a recall of 72%, the proposed hybrid system achieves a recall of ~91%.

Although network traces of virtual worlds are valuable to ISPs (Internet service providers), virtual world software developers, and research communities, they do not exist in the public domain. In this work, we implement a complete testbed to efficiently collect and analyze network traces from a popular virtual world: Second Life. We use the testbed to gather traces from 100 regions with diverse characteristics. The network traces represent more than 60 hours of virtual world traffic and the trace files are created in a well-structured and concise format. Our preliminary analysis on the collected traces is consistent with previous work in the literature. It also reveals some new insights: for example, local avatar/object density imposes clear implications on traffic patterns. The developed testbed and released trace files can be leveraged by research communities for various studies on virtual worlds. For example, accurate traffic models can be derived from our trace files, which in turn can guide developers for better virtual world designs

This paper addresses the problem of disseminating multiple live videos to mobile users by using a hybrid cellular and ad hoc network. Specifically, we develop techniques to optimize the overall quality of video delivery by: (a) exploiting the flexibility of layered videos for in-network adaptation to reduce the gap between video coding rate and network capacity, and (b) alleviating the load of individually handling a large number of flows at the cell tower by using device-to-device ad hoc connectivity to deliver videos. We study the problem of optimally choosing the mobile devices that will serve as gateways from the cellular to the ad hoc network, the ad hoc routes from the gateway to individual devices, and the layers to deliver on these ad hoc routes. We develop a Mixed Integer Linear Program (MILP) based solution to the considered problem. We also develop a heuristic algorithm to select the devices, routes, and layers more efficiently than the ideal, but potentially time-consuming MILP-based algorithm. We evaluate the proposed techniques via through simulations. The simulation results show that the proposed algorithms significantly outperform the current solution in terms of overall video quality, transmission latency, delivery ratio, and missed frame ratio. For example, compared to the current cellular network, the MILP-based and the heuristic algorithms result in up to 20 dB higher video quality. Furthermore, the heuristic algorithm runs efficiently yet achieves near-optimal quality: at most 2.3 dB gap across all experiments.

Peer-to-peer (P2P) systems generate a major fraction of the current Internet traffic, and they significantly increase the load on ISP networks and the cost of running and connecting customer networks (e.g., universities and companies) to the Internet. To mitigate these negative impacts, many previous works in the literature have proposed caching of P2P traffic, but very few (if any) have considered designing a caching system to actually do it. This paper demonstrates that caching P2P traffic is more complex than caching other Internet traffic, and it needs several new algorithms and storage systems. Then, the paper presents the design and evaluation of a complete, running, proxy cache for P2P traffic, called pCache. pCache transparently intercepts and serves traffic from different P2P systems. A new storage system is proposed and implemented in pCache. This storage system is optimized for storing P2P traffic, and it is shown to outperform other storage systems. In addition, a new algorithm to infer the information required to store and serve P2P traffic by the cache is proposed. Furthermore, extensive experiments to evaluate all aspects of pCache using actual implementation and real P2P traffic are presented.

We study the scalable video broadcasting problem in mobile TV broadcast networks, where each TV channel is encoded into a scalable video stream with multiple layers, and several TV channels are concurrently broadcast over a shared air medium to many mobile devices with heterogeneous resources. Our goal is to encapsulate and broadcast video streams encoded in scalable manner to enable heterogeneous mobile devices to render the most appropriate video substreams while achieving high energy saving and low channel switching delay. The appropriate streams depend on the device capability and the target energy consumption level. We propose two new broadcast schemes, which are flexible in the sense that they allow diverse bit rates among layers of the same stream. Such flexibility enables videos to be optimally encoded in terms of coding efficiency, and allows the coded video streams to be better matched with the capability of mobile devices. We analyze the performance of the proposed broadcast schemes. In addition, we have implemented the proposed schemes in a real mobile TV testbed to show their practicality and efficiency. Our extensive experiments confirm that the proposed schemes enable energy saving differentiation: between 75 and 95 percent were observed. Moreover, one of the schemes achieves low channel switching delays: 200 msec is possible with typical system parameters.

Long network latency negatively impacts the performance of online games, and thus mechanisms are needed to mitigate its effects in order to provide a high-quality gaming experience. In this paper, we propose an indirect relay system (IRS) to forward game-state updates over detour paths in order to reduce the round-trip time (RTT) among players. We first collect extensive traces for RTTs among actual players in online games. We then analyze these traces to quantify the potential performance gain of the detour routing. Our analysis reveals that substantial reduction in the RTTs is possible. For example, our results indicate that more than 40% of players can observe at least 100 ms of RTT reduction by routing game-state updates through 1-hop detour paths. Because of the reduction in RTTs, players can join more gaming sessions that were not available to them due to long RTTs of the direct paths. Most importantly, we design and implement a complete IRS system for online games. To the best of our knowledge, this is the first system that directly reduces RTTs among players in online games, while previous works in the literature mitigate the long RTT issue by either hiding it from players or preventing players with high RTTs from being in the same game session. We implement the proposed IRS system and deploy it on 500 PlanetLab nodes. The results from real experiments show that the IRS system improves the online gaming quality from several aspects, while incurring negligible network and processing overheads. In particular, we observe that, with the proposed IRS system, more than 80% of game sessions achieve 100 ms or higher RTT reduction.

In mesh-based peer-to-peer (P2P) streaming systems, each video sequence is divided into segments, which are then streamed from multiple senders to a receiver. The receiver needs to coordinate the senders by specifying a transmission schedule for each of them. We consider the problem of scheduling segment transmission in P2P streaming systems, where different segments have different weights in terms of quality improvements to the received video. Our goal is to compute the transmission schedule for each receiver in order to maximize the perceived video quality. We first show that this scheduling problem is NP-Complete. We then present an integer linear programming (ILP) formulation for it, so that it can be solved with any ILP solver. This optimal solution, however, is computationally expensive and is not suitable for real-time P2P streaming systems. Thus, we propose two approximation algorithms to solve this segment scheduling problem. These algorithms provide theoretical guarantees on the worst-case performance. The first algorithm considers the weight of each video segment. The second algorithm is simpler and it assumes that segments carry equal weights. We analyze the performance and complexity of the two algorithms. In addition, we rigorously evaluate the proposed algorithms with simulations and experiments using a prototype implementation. Our simulation and experimental results show that the proposed algorithms outperform other algorithms that are commonly used in deployed P2P streaming systems and that have been recently proposed in the literature.

We present a general framework for optimizing the quality of video streaming in wireless networks that are composed of multiple wireless stations. The framework is general because: (i) it can be applied to different wireless networks, such as IEEE 802.11e WLAN and IEEE 802.16 WiMAX, (ii) it can employ different objective functions for the optimization, and (iii) it can adopt various models for the wireless channel, the link layer, and the distortion of the video streams in the application layer. The optimization framework controls parameters in different layers to optimally allocate the wireless network resources among all stations. More specifically, we address this video optimization problem in two steps. First, we formulate an abstract optimization problem for video streaming in wireless networks in general. This formulation exposes the important interaction between parameters belonging to different layers in the network stack. Then, we instantiate and solve the general problem for the recent IEEE 802.11e WLANs, which support prioritized traffic classes. We show how the calculated optimal solutions can efficiently be implemented in the distributed mode of the IEEE 802.11e standard. We evaluate our proposed solution using extensive simulations in the OPNET simulator, which captures most features of realistic wireless networks. In addition, to show the practicality of our solution, we have implemented it in the driver of an off-the-shelf wireless adapter that complies with the IEEE 802.11e standard. Our experimental and simulation results show that significant quality improvement in video streams can be achieved using our solution, without incurring any significant communication or computational overhead. We also explain how the general video optimization problem can be applied to other wireless networks, in particular, to the IEEE 802.16 WiMAX networks, which are becoming very popular.

Many mobile TV standards dictate using energy saving schemes to increase the viewing time on mobile devices, since mobile receivers are battery powered. The most common scheme for saving energy is to make the base station broadcast the video data of a TV channel in bursts with a bit-rate much higher than the encoding rate of the video stream, which enables mobile devices to turn off their radio frequency circuits when not receiving bursts. Broadcasting TV channels in bursts, however, increases channel switching delay. The switching delay is important, because long and variable switching delays are annoying to users and may turn them away from the mobile TV service. In this article, we first analyze the burst broadcasting scheme currently used in many deployed mobile TV networks, and we show that it is not efficient in terms of controlling the channel switching delay. We then propose new schemes to guarantee that a given maximum switching delay is not exceeded and that the energy consumption of mobile devices is minimized. We prove the correctness of the proposed schemes and analytically analyze the achieved energy saving. We also use scalable video coding to generalize the proposed schemes in order to support mobile devices with heterogeneous resources. We implement the proposed schemes in a mobile TV testbed to show their practicality and to validate our theoretical analysis. The experimental results show that the proposed schemes: (i) significantly increase the energy saving achieved on mobile devices: up to 95% saving is observed, and (ii) support both homogeneous and heterogeneous mobile devices.

We address the problem of broadcasting multiple video streams over a broadcast network to many mobile devices, so that: (i) streaming quality of mobile devices is maximized, (ii) energy consumption of mobile devices is minimized, and (iii) goodput in the network is maximized. We consider two types of broadcast networks: closed-loop networks, in which all video streams are jointly encoded to ensure their total bit rate does not exceed the broadcast network bandwidth, and open-loop networks, in which videos are encoded using standalone coders, and thus must be carefully broadcast to avoid playout glitches. We first show that the problem of optimally broadcasting multiple videos is NP-complete. We then propose an approximation algorithm to construct burst schedules for multiple VBR (Variable-Bit-Rate) streams. The proposed algorithm frees network operators from the manual and error-prone bandwidth reservation process which is currently used in practice. We prove that the proposed algorithm achieves optimal goodput and near-optimal energy saving. We show that it produces glitch-free schedules in closed-loop networks, and it minimizes number of glitches in open-loop networks. We implement the proposed algorithm in a trace-driven simulator, and conduct extensive simulations for both open- and closed-loop networks. The simulation results show that the proposed algorithm outperforms the existing algorithms in many aspects, including number of late frames, number of concurrently broadcast video streams, and energy saving of mobile devices. To show the practicality and efficiency of the proposed algorithm, we also implement it in a real mobile TV testbed as a proof of concept. The results from the testbed confirm that the proposed algorithm: (i) does not result in playout glitches, (ii) achieves high energy saving, and (iii) runs in real time.

We consider multihomed scalable video streaming, where videos are transmitted by a single server to multiple clients over heterogeneous access networks. The specific problem that we address is to determine which video packets to transmit over each network, in order to minimize a cost function of the expected video distortion at the clients. We present a network model and a video model that capture the network conditions and video characteristics, respectively. We develop an integer program for deterministic packet scheduling. We propose different cost functions in order to provide service differentiation and address fairness among users. We propose several suboptimal convex problems for randomized packet scheduling, and study their performance and complexity. We propose an algorithm that yields a good performance and is suitable for real-time applications. We conduct extensive trace-driven simulations to evaluate the proposed algorithms using real network conditions and scalable video streams. The simulation results show that the proposed algorithm: (i) outperforms the rate control algorithms defined in the Datagram Congestion Control Protocol (DCCP) by about 10 dB, (ii) results in video quality, of 4.33 dB and 1.84 dB higher than the two heuristics developed in [1], (iii) runs efficiently, up to six times faster than one of the heuristics, and (iv) indeed can provide service differentiation among users.

Manual generation of a book inventory is time-consuming and tedious, while deployment of barcode and radio-frequency identification (RFID) management systems is costly and affordable only to large institutions. In this paper, we design and implement a mobile book recognition system for conveniently generating an inventory of books by snapping photos of a bookshelf with a smartphone. Since smartphones are becoming ubiquitous and affordable, our inventory management solution is cost-effective and very easy to deploy. Automatic and robust book recognition is achieved in our system using a combination of spine segmentation and bag-of-features image matching. At the same time, the location of each book is inferred from the smartphone's sensor readings, including accelerometer traces, digital compass measurements, and WiFi signatures. This location information is combined with the image recognition results to construct a location-aware book inventory. We demonstrate the effectiveness of our book spine recognition and location estimation techniques in recognition experiments and in an actual mobile book recognition system.

Increasingly more users use mobile devices to watch videos streamed over wireless networks, and they demand more content at better quality. For example, market forecasts reveal that mobile video streaming, such as mobile TV, will catch up with gaming and music, and become the most popular application on mobile devices. In this tutorial, we will present different approaches to deliver multimedia content over various wireless networks to a large number of mobile users. We will study and analyze the main research problems in modern wireless networks that need to be addressed in order to enable efficient mobile video services. The tutorial will cover common research problems in wireless networks such as HSDPA, MBMS, WiMAX, LTE, DVB-H, MediaFLO, and ATSC M/H. After giving the preliminaries of the considered wireless network standards, we will focus on important research problems and present their solutions in details. Finally, we will discuss open problems and future research directions in mobile video. The tutorial will be composed of five parts, which are briefly described in Sec. 1-5.

We study the problem of improving the user perceived quality of online games in which multiple players form a game session and exchange game-state updates over an overlay network. We propose an Indirect Relay System (IRS) to forward game-state updates over detour paths in order to reduce the round-trip time (RTT) among players. The IRS system efficiently identifies and ranks potential detour paths between any two players, and dynamically selects the most suitable one based on network and client conditions. To the best of our knowledge, this is the first system that directly reduced RTTs among players in online games, while previous works in the literature mitigate the network latency issue by either hiding it from players or preventing players with high RTTs from being in the same game session. We implement the proposed IRS system and deploy it on 500 PlanetLab nodes. The results from real experiments show that the IRS system improves the online gaming quality from several aspects, while incurring negligible network and processing overheads. We also deploy the IRS system on a number of residential computers with DSL and cable modem access links and we successfully found several detour paths among them. To evaluate the IRS system with wider ranges of system parameters we conduct extensive trace-driven simulations using a large number of real game client IPs. The experimental and simulation results show that the proposed IRS system: (i) significantly reduces RTTs among players, (ii) increases number of peers a player can connect to and maintain good gaming quality, (iii) imposes negligible network and processing overheads, and (iv) improves gaming quality and player performance.

We propose a general quality-power adaptation framework that controls the perceived video quality and the length of viewing time on battery-powered video receivers. The framework can be used for standalone video devices (e.g., DVD players and notebooks) as well as mobile receivers obtaining video signals from wireless networks (e.g., mobile TV and video streaming over WiMAX). Furthermore, the framework supports both live streams (e.g., live TV shows) and pre-encoded video streams (e.g., DVD movies). We present an adaptation algorithm for each mobile device to determine the optimal substream that can be received, decoded, and rendered to the user at the: (i) highest quality for a given viewing time, and (ii) longest viewing time for a given quality without exceeding the battery level constraint. We instantiate this framework and work out its details for mobile video broadcast networks. In particular, we propose a new video broadcast scheme that enables mobile video devices to efficiently adapt scalable video streams and achieve power saving proportional to the bit rates of the received streams. We implement the proposed framework in an actual mobile video streaming testbed and we conduct experiments using real video streams broadcast to mobile phones. These experiments show the practicality of the proposed framework and the possibility of achieving viewing time scalability. For example, on a mobile phone receiving and decoding the same video program, a viewing time in the range from 4 to 11 hours can be achieved by adaptively controlling the frame rate and visual quality of the video stream.

In peer-to-peer (P2P) mesh-based streaming systems, each video sequence is typically divided into segments, which are then streamed from multiple senders to a receiver. The receiver needs to coordinate the senders by specifying a transmission schedule for each of them. We consider the scheduling problem in both live and on-demand P2P streaming systems. We formulate the problem of scheduling segment transmission in order to maximize the perceived video quality of the receiver. We prove that this problem is NP-Complete. We present an integer linear programming (ILP) formulation for this problem, and we optimally solve it using an ILP solver. This optimal solution, however, is computationally expensive and is not suitable for real-time streaming systems. Thus, we propose a polynomial-time approximation algorithm, which yields transmission schedules with analytical guarantees on the worst-case performance. More precisely, we show that the approximation factor is at most 3, compared to the absolutely optimal solution as a benchmark. We implement the proposed approximation and optimal algorithms in a packet-level simulator for P2P streaming systems. We also implement two other scheduling algorithms proposed in the literature and used in popular P2P streaming systems. By simulating large P2P systems and streaming nine real video sequences with diverse visual and motion characteristics, we demonstrate that our proposed approximation algorithm: (i) produces near-optimal perceived video quality, (ii) can run in real time, and (iii) outperforms other algorithms in terms of perceived video quality, smoothness of the rendered videos, and balancing the load across sending peers. For example, our simulation results indicate that the proposed algorithm outperforms heuristic algorithms used in current systems by up to 8 dB in perceived video quality and up to 20% in continuity index.

Mobile TV broadcast networks have received significant attention from the industry and academia, as they have already been deployed in several countries around the world and their expected market potential is huge. In such networks, a base station broadcasts TV channels in bursts with bit rates much higher than the encoding bit rates of the videos. This enables mobile receivers to receive a burst of traffic and then turn off their receiving circuits till the next burst to conserve energy. The base station needs to construct a transmission schedule for all bursts of different TV channels. Constructing optimal (in terms of energy saving) transmission schedules has been shown to be an NP-complete problem when the TV channels carry video streams encoded at arbitrary and variable bit rates. In this paper, we propose a near-optimal approximation algorithm to solve this problem. We prove the correctness of the proposed algorithm and derive its approximation factor. We also conduct extensive evaluation of our algorithm using implementation in a real mobile TV testbed as well as simulations. Our experimental and simulation results show that the proposed algorithm: 1) is practical and produces correct burst schedules; 2) achieves near-optimal energy saving for mobile devices; and 3) runs efficiently in real time and scales to large scheduling problems.

Mobile TV networks have received significant attention from the industry and academia, as they have already been deployed in several countries and their expected market potential is huge. In such networks, a base station broadcasts TV channels in bursts with bit rates much higher than the encoding bit rates of the videos. This enables mobile receivers to receive a burst of traffic and then turn off their receiving circuit till the next burst to conserve energy. The base station needs to construct a transmission schedule for all bursts of different TV channels. Constructing optimal (in terms of energy saving) transmission schedules has been shown to be an NP-complete problem when the TV channels are encoded at arbitrary bit rates. In this paper, we propose a near-optimal approximation algorithm to solve this problem. We prove the correctness of the proposed algorithm and derive its approximation factor. We also conduct extensive evaluation of our algorithm using real implementation in a mobile TV testbed and simulations. Our experimental and simulation results show that the proposed algorithm: (i) is practical and produces correct burst schedules, (ii) achieves near-optimal energy saving for mobile devices, and (iii) runs efficiently in real time.

We consider the problem of broadcasting multiple variable-bit-rate (VBR) video streams from a base station to many mobile devices over a wireless network, so that: (i) perceived quality on mobile devices is maximized, (ii) bandwidth utilization is maximized, and (iii) energy consumption of mobile devices is minimized. We show that this problem is NP-Complete. We propose an approximation algorithm for the base station to statistically multiplex and transmit multiple VBR streams to achieve these objectives. We analytically analyze the performance of our algorithm and prove that it achieves optimal bandwidth utilization and near-optimal energy saving. Our algorithm frees network operators from the manual and error-prone bandwidth reservation process, which is usually used in practice for broadcasting VBR streams. We implement the proposed algorithm in a trace-driven simulator, and conduct extensive simulations. The simulation results show that our algorithm outperforms the existing algorithms in many aspects, including number of late frames, number of concurrently broadcast video streams, and energy saving of mobile devices. We also implement the proposed algorithm in a real testbed for video broadcasting as a proof of concept. The results from the testbed confirm that the proposed algorithm: (i) does not result in playout glitches, (ii) achieves high energy saving, and (iii) runs in real time.

We study the problem of broadcasting video streams over a WMAN to many mobile devices. We propose to form a cooperative network among mobile devices that receive the same video stream, and share received video data over a WLAN. We analytically show that the proposed system outperforms current systems in terms of energy consumption and channel switching delay. Our trace-based simulation results show that the proposed system: (i) achieves as high as 70% of energy saving gain, (ii) outperforms current systems with only two cooperative mobile devices, (iii) reduce channel switching delay by up to 98%, (iv) is robust under device failure and quickly reacts to network dynamics, and (v) uniformly distributes the load on all cooperative devices.

In peer-to-peer (P2P) systems, a receiver needs to be matched with multiple senders, because peers have limited capacity and reliability. Efficient peer matching can reduce the cost on Internet Service Providers (ISPs) for carrying the P2P traffic. We study the following peer-matching problem: given a set of potential senders, find the best subset of them that will minimize the transit cost on ISPs. This problem is fairly general and the proposed algorithms for solving it can be used in many P2P systems. We propose two ISP-friendly algorithms for solving this problem: ISPF and ISPF-Lite. These two matching algorithms leverage public available information, such as BGP tables, to infer the network topology, and to minimize the cost on ISPs. The inference algorithms, however, are fairly complex, and we propose optimization techniques to reduce the inference time and to lower the memory requirement. We use trace-driven simulations to show that the proposed algorithms outperform other popular matching algorithms by a large margin. Between the two proposed algorithms, ISPF results in better matching, but incurs higher complexity. Hence, we recommend ISPF if resources are not stringent, otherwise ISPF-Lite is recommended.

We present a complete, running, testbed for mobile TV networks that employ the Digital Video Broadcast - Handheld (DVB-H) open standard. DVB-H based networks have been deployed in several countries around the world and currently being pilot-tested in many others. Nevertheless, there exists no open-source testbed in the literature to enable researchers to analyze and optimize the performance of such networks; most testbeds are proprietary. Our testbed implements the complete stack of the DVB-H standard and it streams real videos to actual handheld devices. It integrates several off-the-shelf hardware components and devices with software components. Some of the software components are developed by us and others are leveraged (after bug fixes and modifications) from open-source projects. In addition, we present several experiments to: (i) evaluate and compare multiple energy-saving techniques recently proposed for mobile TV networks, and (ii) demonstrate a new method to reduce the channel switching delay.

Modern wireless mobile devices have evolved to small computers that can render multimedia content, while desktop/laptop computers have become more computationally powerful with faster Internet access. As these computing devices getting more popular, users demand for more and higher quality videos in many communication applications, where clients are heterogeneous in terms of network bandwidth and computing power. The goal of this thesis is to improve client perceived-quality of various video communication systems by adopting scalable video coding tools that enable efficient rate adaptation. We seek to understand scalable coding standards and design optimization and streaming algorithms to make the best possible use of them in practical systems. We consider practical problems of video communication systems in three different environments: Internet streaming systems, TV broadcast networks, and mobile video communication systems. We propose efficient algorithms to solve the considered problems. We evaluate the proposed algorithms using numerical methods and/or simulations. Most importantly, we design and implement testbeds to validate our algorithms. The expected results of applying our algorithms to video communication systems are better video quality and higher user satisfaction as well as better bandwidth utilization and lower processing overhead.

Traditional video servers partially cope with heterogeneous client populations by maintaining a few versions of the same stream with different bit rates. More recent video servers leverage multilayer scalable coding techniques to customize the quality for individual clients. In both cases, heuristic, error-prone, techniques are currently used by administrators to determine either the rate of each stream version, or the granularity and rate of each layer in a multilayer scalable stream. In this paper, we propose an algorithm to determine the optimal rate and encoding granularity of each layer in a scalable video stream that maximizes a system-defined utility function for a given client distribution. The proposed algorithm can be used to compute the optimal rates of multiversion streams as well. Our algorithm is general in the sense that it can employ arbitrary utility functions for clients. We implement our algorithm and verify its optimality, and we show how various structuring of scalable video streams affect the client utilities. To demonstrate the generality of our algorithm, we consider three utility functions in our experiments. These utility functions model various aspects of streaming systems, including the effective rate received by clients, the mismatch between client bandwidth and received stream rate, and the client-perceived quality in terms of PSNR. We compare our algorithm against a heuristic algorithm that has been used before in the literature, and we show that our algorithm outperforms it in all cases.

Fine-grained scalable (FGS) coding of video streams has been proposed in the literature to accommodate client heterogeneity. FGS streams are composed of two layers: a base layer, which provides basic quality, and a single enhancement layer that adds incremental quality refinements proportional to number of bits received. The base layer uses nonscalable coding which is more efficient in terms of compression ratio than scalable coding used in the enhancement layer. Thus for coding efficiency larger base layers are desired. Larger base layers, however, disqualify more clients from getting the stream. In this paper, we experimentally analyze this coding efficiency gap using diverse video sequences. For FGS sequences, we show that this gap is a non-increasing function of the base layer rate. We then formulate an optimization problem to determine the base layer rate of a single sequence to maximize the average quality for a given client bandwidth distribution. We design an optimal and efficient algorithm (called FGSOPT) to solve this problem. We extend our formulation to the multiple-sequence case, in which a bandwidth-limited server concurrently streams multiple FGS sequences to diverse sets of clients. We prove that this problem is NP-Complete. We design a branch-and-bound algorithm (called MFGSOPT) to compute the optimal solution. MFGSOPT runs fast for many typical cases because it intelligently cuts the search space. In the worst case, however, it has exponential time complexity. We also propose a heuristic algorithm (called MFGS) to solve the multiple-sequence problem. We experimentally show that MFGS produces near-optimal results and it scales to large problems: it terminates in less than 0.5 s for problems with more than 30 sequences. Therefore, MFGS can be used in dynamic systems, where the server periodically adjusts the structure of FGS streams to suit current client distributions.

We present optimal schemes for allocating bits of fine-grained scalable video sequences among multiple senders streaming to a single receiver. This allocation problem is critical in optimizing the perceived quality in peer-to-peer and distributed multi-server streaming environments. Senders in such environments are heterogeneous in their outgoing bandwidth and they hold different portions of the video stream. We first formulate and optimally solve the problem for individual frames, then we generalize to the multiple frame case. Specifically, we formulate the allocation problem as an optimization problem, which is nonlinear in general. We use rate-distortion models in the formulation to achieve the minimum distortion in the rendered video, constrained by the outgoing bandwidth of senders, availability of video data at senders, and incoming bandwidth of receiver. We show how the adopted rate-distortion models transform the nonlinear problem to an integer linear programming (ILP) problem. We then design a simple rounding scheme that transforms the ILP problem to a linear programming (LP) one, which can be solved efficiently using common optimization techniques such as the Simplex method. We prove that our rounding scheme always produces a feasible solution, and the solution is within a negligible margin from the optimal solution. We also propose a new algorithm (FGSAssign) for the single-frame allocation problem that runs in O ( n log n ) steps, where n is the number of senders. We prove that FGSAssign is optimal. Furthermore, we propose a heuristic algorithm (mFGSAssign) that produces near-optimal solutions for the multiple-frame case, and runs an order of magnitude faster than the optimal one. Because of its short running time, mFGSAssign can be used in real time. Our experimental study validates our analytical analysis and shows the effectiveness of our allocation algorithms in improving the video quality.

Rate-distortion (R-D) models are functions that describe the relationship between the bitrate and expected level of distortion in the reconstructed video stream. R-D models enable optimization of the received video quality in different network conditions. Several R-D models have been proposed for the increasingly popular fine-grained scalable video sequences. However, the models' relative performance has not been thoroughly analyzed. Moreover, the time complexity of each model is not known, nor is the range of bitrates in which the model produces valid results. This lack of quantitative performance analysis makes it difficult to select the model that best suits a target streaming system. In this article, we classify, analyze, and rigorously evaluate all R-D models proposed for FGS coders in the literature. We classify R-D models into three categories: analytic, empirical, and semi-analytic. We describe the characteristics of each category. We analyze the R-D models by following their mathematical derivations, scrutinizing the assumptions made, and explaining when the assumptions fail and why. In addition, we implement all R-D models, a total of eight, and evaluate them using a diverse set of video sequences. In our evaluation, we consider various source characteristics, diverse channel conditions, different encoding/decoding parameters, different frame types, and several performance metrics including accuracy, range of applicability, and time complexity of each model. We also present clear systematic ways (pseudo codes) for constructing various R-D models from a given video sequence. Based on our experimental results, we present a justified list of recommendations on selecting the best R-D models for video-on-demand, video conferencing, real-time, and peer-to-peer streaming systems.

Video coders, such as H.264/SVC, can encode a video stream into multiple layers, each with a different rate. Moreover, each layer can either be coarse-grained scalable (CGS) or fine-grained scalable (FGS). FGS layers support wider ranges of client bandwidth than CGS layers, but suffer from higher coding inefficiency. Currently there are no systematic ways in the literature to determine the optimal stream structure that renders the best average quality for all clients. In this paper, we formulate an optimization problem to determine the optimal rate and encoding granularity (CGS or FGS) of each layer in a scalable video stream that maximizes a system-defined utility function for a given client distribution. We design an efficient, yet optimal, algorithm to solve this optimization problem. Our algorithm is general in the sense that it can employ arbitrary utility functions for clients. We implement our algorithm and verify its optimality. We show how various structuring of scalable video streams affect individual client utilities. We compare our algorithm against a heuristic algorithm that has been used before in the literature, and we show that our algorithm outperforms the other one in all cases.