VR streaming and VR cloud gaming require high-resolution video streaming to provide users with high quality visual experience and maximize their interaction and immersion. Consequently, the video streams have a high bitrate and require a large amount of available bandwidth. Foveated video encoding (FVE) reduces bandwidth demand by selectively allocating higher quality to perceptually relevant regions based on human visual characteristics. However, scenes with high spatial detail or rapid motion introduce spatial and temporal frame complexities. Conventional video encoding assesses complexity and manages quality and bitrate through an internal rate-control mechanism. However, the SoTA FVE methods perform the quality allocation process after the rate control has already run. This may lead to rate violations and, consequently, under-or overutilization of available bandwidth, which in turn causes increased latency and/or reduced visual quality. In this paper, we present a real-time complexity-adaptive FVE method that minimizes computational latency using a GPU-accelerated compute shader. By prioritizing key spatial and temporal complexity variables, our weighted complexity estimation optimizes quality assignment. Our method outperforms the complexity-agnostic FVE benchmark with 44-78% greater bitrate stability, 21% lower latency, and 7% higher perceptual quality. It also surpasses the partial complexity-aware FVE benchmark, delivering 18-94% better network utilization alongside a 7% latency reduction and a 4-5% gain in quality. Our method also ensures generalizability across diverse scenarios.

VR cloud gaming promises immersive experiences, yet its realization is critically challenged by the trade-off between stringent latency requirements and high visual quality under unpredictable network conditions. Existing heuristic adaptive bitrate and foveation approaches lack adaptability to highly dynamic mobile networks. This results in a suboptimal trade-off between bandwidth usage and visual quality. While data-driven approaches (i.e., reinforcement learning, RL) have been successful in video streaming, their application to VR cloud gaming poses particular challenges. The stringent demands for high resolution and frame rate, and ultra-low latency are compounded by the necessity for fine-grained, per-frame inference to adapt to rapid changes in user gaze and network conditions. This work introduces FovRL, an RL framework for jointly optimizing foveation parameters and bitrate allocation in response to real-time network throughput. Our work pioneers the application of RL for real-time foveated encoding in immersive VR cloud gaming. Evaluations over real-world networks reveal that FovRL enhances bitrate adaptability to deliver superior perceptual visual quality, while maintaining latency comparable to the SoTA.

In cloud virtual reality (VR), delivering high perceptual visual quality under constrained wireless bandwidth remains a pivotal challenge. Foveated video encoding (FVE) copes with this by human vision-driven quality allocation, reducing bandwidth demand while preserving high visual fidelity for immersive VR streaming. However, existing FVE methods are content-independent, relying exclusively on gaze position and predefined quality degradation profiles. Thus, they fall short in modeling the intricate mechanisms of human attention. This leads to suboptimal quality distribution for visual saliency stimuli, causing perceptible artifacts and degraded perceptual quality. In this paper, we introduce a novel AI-driven streaming framework that incorporates visual saliency cues, defined as the innate capacity of scene elements to attract attention, into the video encoding pipeline. To meet the stringent low-latency demands of immersive VR, we propose a lightweight deep neural network for saliency inference, reducing computational complexity (FLOPs) by 48× compared to prior models while maintaining comparable accuracy. We integrate our pipeline into an open-source cloud VR gaming platform and conduct comprehensive experiments. Evaluation results demonstrate that our approach enhances perceptual visual quality by 22.98% compared to SoTA systems. Our IRB-approved user study shows that the saliency-guided FVE pipeline achieves superior visual quality and spatial smoothness while significantly reducing noticeable artifacts introduced by gaze-exclusive FVE. The project source code is available at https://github.com/WuZemyp/SaliencyFov.

Virtual reality (VR) cloud gaming is increasingly developing in the gaming industry. Yet, the performance of the congestion control algorithms on top of which these systems build remains under-explored. In this study, we implement two industry-standard network congestion control algorithms, Google Congestion Control (GCC) and Network-Assisted Dynamic Adaptation (NADA), according to their Requests for Comments (RFCs), and integrate them into an open-source VR gaming system (ALVR). Including ALVR's congestion control (ALVR-ABR), we conduct extensive experiments on real-world networks to evaluate each algorithm's frame latency, target-to-receiving bitrate gap, dropped frames, image quality, and fairness among heterogeneous competing flows. GCC decreases frame latency by 352ABR present significant gaps between the selected and received bitrate, causing substantial congestion-induced frame drops, while GCC has a minimal gap, resulting in minor frame drops, suggesting its suitability for game-player interaction. GCC exhibits a 2.7ABR, respectively, indicating slight immersion degradation. However, only NADA ensures a fair bandwidth share against loss-based flows due to its bitrate response to loss-induced congestion signals and lower sensitivity to delay gradients compared to GCC.

Remote rendering enables high-fidelity virtual reality (VR) experiences on standalone headsets by offloading intensive graphics workloads to remote servers. However, streaming high-quality VR graphics imposes substantial bandwidth and latency challenges. Spatial compression is a form of foveation which addresses this challenge by leveraging the human visual system's varying acuity, allocating higher visual quality around the user's gaze while reducing resolution in the periphery. In this work, we implement three gaze-adaptive foveation methods: Dynamic Axis-Aligned Distortion Transmission (D-AADT2 and D-AADT3) and Dynamic Foveated Radial Warp (D-FRW)) of which only D-AADT2 has been previously presented. These methods dynamically adapt spatial compression based on gaze-tracking input, ensuring optimal perceptual quality. We integrate these methods together with their static counterparts into the open-source Air Light VR (ALVR) remote-rendering framework, enabling native (72 FPS) framerates. We conclude a comprehensive objective evaluation across diverse VR games and demonstrate that the dynamic methods significantly outperform traditional static approaches in both encoding efficiency and perceptual quality metrics. A complementary subjective user study further validates these findings, confirming that dynamic gaze-adaptive foveation substantially enhances visual quality, immersion, and user interaction experience.

Accurate characterization of Head Mounted Display (HMD) pose in a virtual scene is essential for rendering immersive graphics in Extended Reality (XR). Remote rendering employs servers in the cloud or at the edge of the network to overcome the computational limitations of either standalone or tethered HMDs. Unfortunately, it increases the latency experienced by the user; for this reason, predicting HMD pose in advance is highly beneficial, as long as it achieves high accuracy. This work provides a thorough characterization of solutions that forecast HMD pose in remotely-rendered virtual reality (VR) by considering six degrees of freedom. Specifically, it provides an extensive evaluation of pose representations, forecasting methods, machine learning models, and the use of multiple modalities along with joint and separate training. In particular, a novel three-point representation of pose is introduced together with a data fusion scheme for long-term short-term memory (LSTM) neural networks. Our findings show that machine learning models benefit from using multiple modalities, even though simple statistical models perform surprisingly well. Moreover, joint training is comparable to separate training with carefully chosen pose representation and data fusion strategies.

Cloud Virtual Reality (VR) gaming offloads the computationally-intensive rendering tasks from resource-limited Head-Mounted Displays (HMDs) to cloud servers, which consume a staggering amount of bandwidth for high-quality gaming experiences. One way to cope with such high bandwidth demands is to capitalize on human vision systems by allocating a higher bitrate to the foveal region of HMD viewport, which is known as foveation in the literature. Although foveation was employed by remote VR gaming, existing open-source projects all adopt static foveation, in which the HMD gamer gaze position is assumed to be fixed at the viewport center. In this paper, we construct the very first cloud VR gaming system that supports dynamic foveation. That is, the real-time gaze positions of gamers are streamed from eye-trackers on HMDs to cloud servers, which in turn adjust the foveation parameters, such as foveal region size/location and peripheral region quality degradation, accordingly. Using our developed cloud VR gaming system, we design and carry out a user study using a game called Fruit Ninja VR 2 to find the foveation parameters in static and dynamic foveation for maximizing the gaming Quality of Experience (QoE) in Mean Opinion Score (MOS). With the chosen foveation parameters, we found that, compared to cloud VR gaming without foveation, static foveation leads to a MOS increase of 0.60 and a bitrate reduction of 8.71%. Furthermore, adopting dynamic foveation results in an additional 0.60 increase on MOS while saving 9.81% bitrate, compared to static foveation. Our findings demonstrate the potential of dynamic foveation in cloud VR gaming, which dictates both high visual quality and short response time. The optimization techniques developed in this and follow-up work could benefit other cloud-rendered applications that typically have less strict requirements than cloud VR gaming.

Depth maps are needed by various graphics rendering and processing operations. Depth map streaming is often necessary when such operations are performed in a distributed system and it requires in most cases fast performing compression, which is why video codecs are often used. Hardware implementations of standard video codecs enable relatively high resolution and frame rate combinations, even on resource constrained devices, but unfortunately those implementations do not currently support RGB+depth extensions. However, they can be used for depth compression by first packing the depth maps into RGB or YUV frames. We investigate depth map compression using a combination of depth map packing followed by encoding with a standard video codec. We show that the precision at which depth maps are packed has a large and nontrivial impact on the resulting error caused by the combination of the packing scheme and lossy compression when the bitrate is constrained. Consequently, we propose a variable precision packing scheme assisted by a neural network model that predicts the optimal precision for each depth map given a bitrate constraint. We demonstrate that the model yields near optimal predictions and that it can be integrated into a game engine with very low overhead using modern hardware.

Remote rendering systems comprise powerful servers that render graphics on behalf of low-end client devices and stream the graphics as compressed video, enabling high end gaming and Virtual Reality on those devices. One key challenge with them is the amount of bandwidth required for streaming high quality video. Humans have spatially non-uniform visual acuity: We have sharp central vision but our ability to discern details rapidly decreases with angular distance from the point of gaze. This phenomenon called foveation can be taken advantage of to reduce the need for bandwidth. In this paper, we study three different methods to produce a foveated video stream of real-time rendered graphics in a remote rendered system: 1) foveated shading as part of the rendering pipeline, 2) foveation as post processing step after rendering and before video encoding, 3) foveated video encoding. We report results from a number of experiments with these methods. They suggest that foveated rendering alone does not help save bandwidth. Instead, the two other methods decrease the resulting video bitrate significantly but they also have different quality per bit and latency profiles, which makes them desirable solutions in slightly different situations.

The availability of high bandwidth with low-latency communication in 5G mobile networks enables remote rendered real-time virtual reality (VR) applications. Remote rendering of VR graphics in a cloud removes the need for local personal computer for graphics rendering and augments weak graphics processing unit capacity of stand-alone VR headsets. However, to prevent the added network latency of remote rendering from ruining user experience, rendering a locally navigable viewport that is larger than the field of view of the HMD is necessary. The size of the viewport required depends on latency: Longer latency requires rendering a larger viewport and streaming more content. In this article, we aim to utilize multi-access edge computing to assist the backend cloud in such remote rendered interactive VR. Given the dependency between latency and amount and quality of the content streamed, our objective is to jointly optimize the tradeoff between average video quality and delivery latency. Formulating the problem as mixed integer nonlinear programming, we leverage the interpolation between client’s field of view frame size and overall latency to convert the problem to integer nonlinear programming model and then design efficient online algorithms to solve it. The results of our simulations supplemented by real-world user data reveal that enabling a desired balance between video quality and latency, our algorithm particularly achieves the improvements of on average about 22% and 12% in term of video delivery latency and 8% in term of video quality compared to respectively order-of-arrival, threshold-based, and random-location strategies.

Human conversation is characterized by brief pauses and so-called turn-taking behavior between the speakers. In the context of VoIP, this means that there are frequent periods where the microphone captures only background noise - or even silence whenever the microphone is muted. The bits transmitted from such silence periods introduce overhead in terms of data usage, energy consumption, and network infrastructure costs. In this paper, we contribute by shedding light on these costs for VoIP applications. We systematically measure the performance of six popular mobile VoIP applications with controlled human conversation and acoustic setup. Our analysis demonstrates that significant savings can indeed be achieved - with the best performing silence suppression technique being effective on 75% of silent pauses in the conversation in a quiet place. This results in 2-5 times data savings, and 50-90% lower energy consumption compared to the next best alternative. Even then, the effectiveness of silence suppression can be sensitive to the amount of background noise, underlying speech codec, and the device being used. The codec characteristics and performance do not depend on the network type. However, silence suppression makes VoIP traffic network friendly as much as VoLTE traffic. Our results provide new insights into VoIP performance and offer a motivation for further enhancements to a wide variety of voice assisted applications, such as home assistants and other IoT devices.

Humans have sharp central vision but low peripheral visual acuity. Prior work has taken advantage of this phenomenon in two ways: foveated rendering (FR) reduces the computational workload of rendering by producing lower visual quality for peripheral regions and foveated video encoding (FVE) reduces the bitrate of streamed video through heavier compression of peripheral regions. Remote rendering systems require both rendering and video encoding and the two techniques can be combined to reduce both computing and bandwidth consumption. We report early results from such a combination with remote VR rendering. The results highlight that FR causes large bitrate overhead when combined with normal video encoding but combining it with FVE can mitigate it.

Cloud gaming enables playing high-end games, originally designed for PC or game console setups, on low-end devices such as netbooks and smartphones, by offloading graphics rendering to GPU-powered cloud servers. However, transmitting the high-resolution video requires a large amount of network bandwidth, even though it is a compressed video stream. Foveated video encoding (FVE) reduces the bandwidth requirement by taking advantage of the non-uniform acuity of human visual system and by knowing where the user is looking. Based on a consumer-grade real-time eye tracker and an open source cloud gaming platform, we provide a cloud gaming FVE prototype that is game-agnostic and requires no modifications to the underlying game engine. In this article, we describe the prototype and its evaluation through measurements with representative games from different genres to understand the effect of parametrization of the FVE scheme on bandwidth requirements and to understand its feasibility from the latency perspective. We also present results from a user study on first-person shooter games. The results suggest that it is possible to find a “sweet spot” for the encoding parameters so the users hardly notice the presence of foveated encoding but at the same time the scheme yields most of the achievable bandwidth savings.

Air pollution is a major problem in urban areas, where high population density is accompanied with excess anthropomorphic emissions impacting the environment and increasing health effects. Highly accurate air quality monitoring stations have been used to monitor the severity of the problem and warn citizens. However, air quality can vary sharply even within the same city block, and pollution exposure can vary even 30% between individuals living in the same residence. Therefore, a dense deployment of air quality sensors is needed to detect these variations, and protect citizens from overexposure. Low-cost air quality sensors make it possible to densely instrument a city and detect hot spots as they happen. However, thus far limited information exists on their accuracy and practicability. In this paper, we conduct a 44-day measurement campaign to assess performance of low-cost air quality monitors under different environmental conditions. As practical use case, we consider pollution hot spot detection. Our results show that the mean error of low-cost sensors is small, but the variation in error is significantly larger than with reference sensors. We also show that the accuracy is sufficient for applications relying on variations in air quality index values, such as hot spot detection.

Poor indoor air quality is a significant burden to society that can cause health issues and decrease productivity. According to research, indoor air quality is intrinsically linked with human activity and mobility. Indeed, mobility is directly linked with transfer of small particles (e.g. PM2.5) and extent of activity affects production of CO2. Currently, however, estimation of indoor quality is difficult, requiring deployment of highly specialized sensing devices which need to be carefully placed and maintained. In this paper, we contribute by examining the suitability of infrastructure-based motion detectors for indoor air quality estimation. Such sensors are increasingly being deployed into smart environments, e.g., to control lighting and ventilation for energy management purposes. Being able to take advantage of these sensors would thus provide a cost-effective solution for indoor quality monitoring without need for deploying additional sensors. We perform a feasibility study considering measurements collected from a smart office environment having a dense deployment of motion detectors and correlating measurements obtained from motion detectors against air quality values. We consider two main pollutants, PM2.5and CO2, and demonstrate that there indeed is a connection between extent of movement and PM2.5concentration. However, for CO2, no relationship can be established, mostly due to difficulties in separating between people passing by and those residing long-term in the environment.

Real time interactive cloud-based mobile applications such as augmented reality and cloud gaming require low and stable latency, especially in urban areas. These conditions are difficult to meet with the traditional single carrier LTE network access and consolidated server deployment in a cloud. Yet, with multiple SIM/multiple radio devices, latency can be kept under a given threshold through dynamic selection among multiple carriers and server deployment at network edge. To this end, it is necessary to understand how mobile network latency changes over time during a session with different carriers and how the server placement affects the latencies. In this paper, we present results from a measurement study of mobile network latency and jitter in 4G networks of Hong Kong and Helsinki, two very different cities in terms of population density and mobile infrastructure. Based on the results, we introduce a lightweight carrier selection algorithm that displays latencies 10 to 20% lower than single carrier operation.

Multi-access edge computing (MEC) enables placing video content at the edge of a mobile network with the aim of reducing data traffic in the backhaul network. Direct device-to-device (D2D) communication can further alleviate load from the backhaul network. Both MEC and D2D have already been examined by prior work, but their combination applied to adaptive video streaming have not yet been explored in detail. In this paper, we analyze how enabling D2D jointly with edge computing affects the quality of experience (QoE) of video streaming clients and contributes to reducing the backhaul traffic. To this end, we formulate the problem of jointly maximizing the QoE of the clients and minimizing the backhaul traffic and edge processing as an integer non-linear programming (INLP) optimization model and propose a low-complexity algorithm using self-parameterization technique to solve the problem. The main takeaway from simulation results is that enabling D2D with edge computing reduces the backhaul traffic by approximately 18% and edge processing by 30% on average while maintaining roughly the same average video bitrate per client compared to edge computing without D2D. Our results provide a guideline for system designers to judge the effectiveness of enabling D2D into MEC in the next generation of 5G mobile networks.

Nearly all bitrate adaptive video content delivered today is streamed using protocols that run a purely client based adaptation logic. The resulting lack of coordination may lead to suboptimal user experience and resource utilization. As a response, approaches that include the network and servers in the adaptation process are emerging. In this article, we present an optimized solution for network assisted adaptation specifically targeted to mobile streaming in multi-access edge computing (MEC) environments. Due to NP-Hardness of the problem, we have designed a heuristic-based algorithm with minimum need for parameter tuning and having relatively low complexity. We then study the performance of this solution against two popular client-based solutions, namely Buffer-Based Adaptation (BBA) and Rate-Based Adaptation (RBA), as well as to another network assisted solution. Our objective is two fold: First, we want to demonstrate the efficiency of our solution and second to quantify the benefits of network-assisted adaptation over the client-based approaches in mobile edge computing scenarios. The results from our simulations reveal that the network assisted adaptation clearly outperforms the purely client-based DASH heuristics in some of the metrics, not all of them, particularly, in situations when the achievable throughput is moderately high or the link quality of the mobile clients does not differ from each other substantially.

High quality immersive Virtual Reality experience currently requires a PC setup with cable connected head mounted display, which is expensive and restricts user mobility. This paper presents CloudVR which is a system for cloud accelerated interactive mobile VR. It is designed to provide short rotation and interaction latencies through panoramic rendering and dynamic object placement. CloudVR also includes rendering optimizations to reduce server-side computational load and bandwidth requirements between the server and client. Performance measurements with a CloudVR prototype suggest that the optimizations make it possible to double the server's framerate and halve the amount of bandwidth required and that small objects can be quickly moved at run time to client device for rendering to provide shorter interaction latency. A small-scale user study indicates that CloudVR users do not notice small network latencies (20ms) and even much longer ones (100-200ms) become non-trivial to detect when they do not affect the interaction with objects. Finally, we present a design of CloudVR extension to multi-user scenarios.

We study performance characteristics of convolutional neural networks (CNN) for mobile computer vision systems. CNNs have proven to be a powerful and efficient approach to implement such systems. However, the system performance depends largely on the utilization of hardware accelerators, which are able to speed up the execution of the underlying mathematical operations tremendously through massive parallelism. Our contribution is performance characterization of multiple CNN-based models for object recognition and detection with several different hardware platforms and software frameworks, using both local (on-device) and remote (network-side server) computation. The measurements are conducted using real workloads and real processing platforms. On the platform side, we concentrate especially on TensorFlow and TensorRT. Our measurements include embedded processors found on mobile devices and high-performance processors that can be used on the network side of mobile systems. We show that there exists significant latency-throughput trade-offs but the behavior is very complex. We demonstrate and discuss several factors that affect the performance and yield this complex behavior.

Broadcasting live video directly from mobile devices is rapidly gaining popularity with applications like Periscope and Facebook Live. The quality of experience (QoE) provided by these services comprises many factors, such as quality of transmitted video, video playback stalling, end-to-end latency, and impact on battery life, and they are not yet well understood. In this article, we examine mainly the Periscope service through a comprehensive measurement study and compare it in some aspects to Facebook Live. We shed light on the usage of Periscope through analysis of crawled data and then investigate the aforementioned QoE factors through statistical analyses as well as controlled small-scale measurements using a couple of different smartphones and both versions, Android and iOS, of the two applications. We report a number of findings including the discrepancy in latency between the two most commonly used protocols, RTMP and HLS, surprising surges in bandwidth demand caused by the Periscope app’s chat feature, substantial variations in video quality, poor adaptation of video bitrate to available upstream bandwidth at the video broadcaster side, and significant power consumption caused by the applications.

Good user experience with interactive cloud-based multimedia applications, such as cloud gaming and cloud-based VR, requires low end-to-end latency and large amounts of downstream network bandwidth at the same time. In this paper, we present a foveated video streaming system for cloud gaming. The system adapts video stream quality by adjusting the encoding parameters on the fly to match the player's gaze position. We conduct measurements with a prototype that we developed for a cloud gaming system in conjunction with eye tracker hardware. Evaluation results suggest that such foveated streaming can reduce bandwidth requirements by even more than 50% depending on parametrization of the foveated video coding and that it is feasible from the latency perspective.

Cloud gaming is a relatively new paradigm in which the game is rendered in the cloud and is streamed to an end-user device through a thin client. Latency is a key challenge for cloud gaming. In order to optimize the end-to-end latency, it is first necessary to understand how the end-to-end latency builds up from the mobile device to the cloud gaming server. In this paper we dissect the delays occurring in the mobile device and measure access delays in various networks and network conditions. We also perform a Europe-wide latency measurement study to find the optimal server locations and see how the number of server locations affects the network delay. The results are compared to limits found for perceivable delays in recent human-computer interaction studies. We show that the limits can be achieved only with the latest mobile devices with specific control methods. In addition, we study the expected latency reduction by near future technological development and show that its potential impact is bigger on the end-to-end latency than that of replication of the service and server placement optimization.

MPEG has recently proposed Server and Network Assisted Dynamic Adaptive Streaming over HTTP (SANDDASH) for video streaming over the Internet. In contrast to the purely client-based video streaming in which each client makes its own decision to adjust its bitrate, SAND-DASH enables a group of simultaneous clients to select their bitrates in a coordinated fashion in order to improve resource utilization and quality of experience. In this paper, we study the performance of such an adaptation strategy compared to the traditional approach with large number of clients having mobile Internet access. We propose a multi-servers multi-coordinators (MSsMCs) framework to model groups of remote clients accessing video content replicated to spatially distributed edge servers. We then formulate an optimization problem to maximize jointly the QoE of individual clients, proportional fairness in allocating the limited resources of base stations as well as balancing the utilized resources among multiple serves. We then present an efficient heuristic-based solution to the problem and perform simulations in order to explore parameter space of the scheme as well as to compare the performance to purely client-based DASH.

Full charge capacity (FCC) refers to the amount of charge a battery can hold. It is the fundamental property of smartphone batteries that diminishes as the battery ages and is charged/discharged. We investigate the behavior of smartphone batteries while charging and demonstrate that battery voltage and charging rate information can together characterize the FCC of a battery. We propose a new method for accurately estimating FCC without exposing low-level system details or introducing new hardware or system modules. We further propose and implement a collaborative FCC estimation technique that builds on crowd-sourced battery data. The method finds the reference voltage curve and charging rate of a particular smartphone model from the data and then compares with those of an individual device. After analyzing a large data set towards a crowd-sourced rate versus FCC model, we report that 55 percent of all devices and at least one device in 330 out of 357 unique device models lost some of their FCC. For some old device models, the median capacity loss exceeded 20 percent. The models further enable debugging the performance of smartphone charging. We propose an algorithm, called BatterySense, which utilizes crowd-sourced rate to detect abnormal charging performance, estimate FCC of the device battery, and detect battery changes.

Sharing live multimedia content is becoming increasingly popular among mobile users. In this article, we study the problem of optimizing video quality in such a scenario using scalable video coding (SVC) and chunked video content. We consider using only standard stateless HTTP servers that do not need to perform additional processing of the video content. Our key contribution is to provide close to optimal algorithms for scheduling video chunk upload for multiple clients having different viewing delays. Given such a set of clients, the problem is to decide which chunks to upload and in which order to upload them so that the quality-delay tradeoff can be optimally balanced. We show by means of simulations that the proposed algorithms can achieve notably better performance than naive solutions in practical cases. Especially the heuristic-based greedy algorithm is a good candidate for deployment on mobile devices because it is not computationally intensive but it still delivers in most cases on-par video quality compared to the more complex local optimization algorithm. We also show that using shorter video segments and being able to predict bandwidth and video chunk properties improve the delivered video quality in certain cases.

Recent advances from wearables have significantly changed the way how humans communicate with the surrounding environment. To some extent, they have extended and augmented the capability of humans. For example, with a Google Glass, people can take pictures simply by winking eyes twice, which releases human hands from the cumbersome image-taking process. Thus, it enables new application scenarios that were not possible before. In this paper, we investigate utilizing vision-based techniques to provide a wearable positioning system. Specifically, we propose a Human-centric Positioning System (HoPS) that utilizes traffic signposts together with context information for real-time positioning. Towards that direction, we make three primary contributions: (1) we make several important observations that guide our design of HoPS system; for example, we find out that approximately 40 percent of traffic signposts monopolize a cell tower, and there are at most six signposts within the coverage of a single cell tower; (2) we investigate the impact factors of object detection success rate, and find its correlation with image quality, and resolution; and (3) we design and implement HoPS and an advanced version of HoPS based on additional context information from Wi-Fi network, which we name HoPS-WiFi. Experimental results demonstrate the effectiveness of HoPS, especially HoPS-WiFi, which can estimate the relevant location correctly within 1.3 seconds.

Live video streaming from mobile devices is quickly becoming popular through services such as Periscope, Meerkat, and Facebook Live. Little is known, however, about how such services tackle the challenges of the live mobile streaming scenario. This work addresses such gap by investigating in details the characteristics of the Periscope service. A large number of publicly available streams have been captured and analyzed in depth, in particular studying the characteristics of the encoded streams and the communication evolution over time. Such an investigation allows to get an insight into key performance parameters such as bandwidth, latency, buffer levels and freezes, as well as the limits and strategies adopted by Periscope to deal with this challenging application scenario.

Video streaming can drain a smartphone battery quickly. A large part of the energy consumed goes to wireless communication. In this article, we first study the energy efficiency of different video content delivery strategies used by service providers and identify a number of sources of energy inefficiency. Specifically, we find a fundamental tradeoff in energy waste between prefetching small and large chunks of video content: small chunks are bad because each download causes a fixed tail energy to be spent regardless of the amount of content downloaded, whereas large chunks increase the risk of downloading data that user will never view because of abandoning the video. Hence, the key to optimal strategy lies in the ability to predict when the user might abandon viewing prematurely. We then propose an algorithm called eSchedule that uses viewing statistics to predict viewer behavior and computes an energy optimal download strategy for a given mobile client. The algorithm also includes a mechanism for explicit control of traffic overhead, i.e., unnecessary download of content that the user will never watch. Our evaluation results suggest that the algorithm can cut the energy waste down to less than half compared to other strategies. We also present and experiment with an Android prototype that integrates eSchedule into a YouTube downloader.

Sharing of devices across separate networks is currently challenging. Users may want to give their friends remote access to their home entertainment devices, but a secure and easy-to-use solution has not yet emerged. While service discovery frameworks provide a uniform interface for different networked devices to interact, most of them, especially the widely adopted Universal Plug and Play, fail to connect devices in separate networks. This paper proposes a proxy-based architecture H-box that bridges different UPnP-based networks. We integrate a number of standardized protocols, namely XMPP, HIP and Teredo in a novel way to achieve our goal. The prototype runs on Raspberry Pi, bridging commodity UPnP devices located in remote networks. The performance evaluation shows that the approach is feasible even for media streaming between homes. Further, we envision the H-box architecture to be adopted for car entertainment systems and extended to support device sharing and collaboration between separate networks.

Data traffic in cellular networks increased tremendously over the past few years and this growth is predicted to continue over the next few years. Due to differences in access technology and user behavior, the characteristics of cellular traffic can differ from existing results for wireline traffic. In this study we focus on understanding the flow rates and on the relationship between the rates and other flow properties by analyzing packet level traces collected in a large cellular network. To understand the limiting factors of the flow rates, we further analyze the underlying causes behind the observed rates, e.g., network congestion, access link or end host configuration. Our study extends other related work by conducting the analysis from a unique dimension, the comparison with traffic in wired networks, to reveal the unique properties of cellular traffic. We find that they differ in variability and in the dominant rate limiting factors.

Smartphones are frequently used to shoot and share videos online and emerging applications, such as Augmented Reality, will increase the usage of the camera. Unfortunately, shooting and streaming video drains a modern smartphone's battery very quickly. We report results from a measurement study to dissect the smartphone energy consumption when using such an application. Our main findings are that the majority of power is drawn already when the camera is in focus mode and not yet recording. This power is drawn by the camera internal hardware and some other hardware of the smartphone related to the video processing, and none of this hardware seems to scale the power draw with the video resolution or bit rate. We also study the effectiveness of two simple optimization techniques, namely frame bundling to optimize the radio usage and more aggressive frequency and voltage scaling to reduce the computational power draw. We conclude that while the mechanisms are effective, their potential is overshadowed by the large power draw of other hardware.

Wireless data transmission consumes a significant part of the overall energy consumption of smartphones, due to the popularity of Internet applications. In this paper, we investigate the energy consumption characteristics of data transmission over Wi-Fi, focusing on the effect of Internet flow characteristics and network environment. We present deterministic models that describe the energy consumption of Wi-Fi data transmission with traffic burstiness, network performance metrics like throughput and retransmission rate, and parameters of the power saving mechanisms in use. Our models are practical because their inputs are easily available on mobile platforms without modifying low-level software or hardware components. We demonstrate the practice of model-based energy profiling on Maemo, Symbian, and Android phones, and evaluate the accuracy with physical power measurement of applications including file transfer, web browsing, video streaming, and instant messaging. Our experimental results show that our models are of adequate accuracy for energy profiling and are easy to apply.

This article proposes a novel energy-efficient multimedia delivery system called EStreamer. First, we study the relationship between buffer size at the client, burst-shaped TCP-based multimedia traffic, and energy consumption of wireless network interfaces in smartphones. Based on the study, we design and implement EStreamer for constant bit rate and rate-adaptive streaming. EStreamer can improve battery lifetime by 3x, 1.5x, and 2x while streaming over Wi-Fi, 3G, and 4G, respectively.

Video streaming on smartphones is one of the most popular but also most energy hungry services today. Using mobile video services results in two contradictory sources of energy waste for smartphones: i) energy waste because of excessively aggressive prefetching of content that the user will not watch because of abandoning the session, and ii) excessive amount of tail energy, which is energy wasted by keeping the wireless interface powered on after receiving a chunk of content; this is caused by prefetching chunks that are too small. To remedy this, we propose a novel download scheduling algorithm based on crowd-sourced video viewing statistics. Our algorithm judiciously evaluates the probability of a user interrupting a video viewing in order to perform the right amount of prefetching. In this way, the algorithm balances the amount of the two above-mentioned kinds of energy waste. By simulations, we show that our scheduler cuts the energy waste to half compared to existing download strategies. We have also developed an Android prototype that implements the download scheduler together with a novel downloader that speeds up the download by exploiting the Fast Start technique. The prototype exhibits the desired properties of the scheduler, and its faster downloading mechanism yields further energy savings of up to 80% compared to the default Android YouTube app.

Shaping constant bit rate traffic into bursts has been proposed earlier for UDP-based multimedia streaming to save Wi-Fi communication energy of mobile devices. The relationship between the burst size and energy consumption of wireless interfaces is such that the larger is the burst size, the lower is the energy consumption per bit received as long as there is no packet loss. However, the relationship between the burst size and energy in case of TCP traffic has not yet been fully uncovered. In this paper, we develop a power consumption model which describes this relationship in wireless multimedia streaming scenarios. Then, we implement a cross-layer stream delivery system, EStreamer. This system relies on a heuristic derived from the model and on client playback buffer status to determine a burst size and provides as small energy consumption as possible without jeopardizing smooth playback. The heuristic greatly simplifies the deployment of EStreamer compared to most existing solutions by ensuring energy savings regardless of the wireless interface being used. We show that in the best cases using EStreamer reduces energy consumption of a mobile device by 65%, 50-60% and 35% while streaming over Wi-Fi, LTE and 3G respectively. Compared with existing energy-aware applications energy consumption can be reduced by 10-55% further.

Multimedia streaming applications are among the most energy hungry applications in smartphones. The energy consumption mostly depends on the delivery techniques and on the power management techniques of wireless interfaces (Wi-Fi and 3G). In order to provide insights on what kind of streaming techniques exist, how they work on different mobile platforms, and what is their impact on the energy consumption of mobile phones, we have done a large set of active measurements with several smartphones having both Wi-Fi and cellular network access. Our analysis reveals five different techniques to deliver the content to the video players. The selection of a technique depends on the device, player, quality, and service. The results from our power measurements allow us to conclude that none of the identified techniques is optimal because they take none of the following facts into account: access technology used, user behaviour, and user preferences concerning data waste. However, we point out the techniques that provide the most attractive trade-offs in particular situations. Furthermore, we make several observations on the energy consumption of different players, containers, and video qualities that should be taken into consideration when optimizing the energy consumption.

Detecting and tracking the position of a mobile user is an increasingly important feature in many mobile applications. In this work we study how cheap and energy-efficient air pressure sensors measuring the altitude could be used, as a complement to the dominant GPS system. The cornerstone of our approach is that a huge amount of route data, collected with GPS devices, is available in various cloud services. The location detection and route tracking task thus becomes a question of matching the collected altitude traces with the altitude curves of stored data to find the best matching routes. Here we build a prototype system of crowd-sourced database containing only altitude data. How accurately this stored altitude data could be matched with the collected altitude traces is the key question of our study.

Offloading computation to cloud has been widely used for extending battery life of mobile devices. However, little effort has been invested in applying the offloading techniques to communication-related tasks. We propose SmartDiet, a toolkit to identify the constraints that reduce offloading opportunities and to calculate the energy-saving potential of offloading communication-related tasks. SmartDiet traces the method-level application execution and estimates the allocation of communication energy cost from traffic traces. We discuss key features of SmartDiet and show some preliminary results using a prototype implementation.

Energy consumption of wireless data transmission, a significant part of the overall energy consumption on a mobile device, is context-dependent—it depends on both internal and external contexts, such as application workload and wireless signal strength. In this paper, we propose an event-driven framework that can be used for efficient power management on mobile devices. The framework adapts the behavior of a device component or an application to the changes in contexts, defined as events, according to developer-specified event-condition-action (ECA) rules that describe the power management mechanism. In contrast to previous work, our framework supports complex event processing. By correlating events, complex event processing helps to discover complex events that are relevant to power consumption. Using our framework developers can implement and configure power management applications by editing event specifications and ECA rules through XML-based interfaces. We evaluate this framework with two applications in which the data transmission is adapted to traffic patterns and wireless link quality. These applications can save roughly 12 percent more energy compared to normal operation.

We study how much energy can be saved by reshaping audio streaming traffic before receiving at the mobile devices. The rationale is the following: Mobile network interfaces (WLAN and 3G) are in active mode when they transmit or receive data, otherwise they are in idle/sleep mode. To save energy, minimum possible time should be spent in active mode and maximum in idle/sleep mode. It is well known that by reshaping the usually constant bit rate multimedia traffic into bursts, it is possible to spend more time in idle/sleep mode leading to impressive energy savings. We propose a proxy-based solution that shapes an audio stream into bursts before relaying the traffic to the mobile device. The novelty of our work is an evaluation of the energy savings using such a proxy with different configurations for both WLAN access with standard 802.11 Power Saving Mode and 3G access. We conclude that for WLAN access, proxy causes power savings of 30%-65% depending on the audio stream rate, location of the proxy and amount of cross traffic. In the case of 3G, the effectiveness of our proxy seems to vary depending on the phone model and operator. In some cases, the energy savings are encouraging, while in other cases the proxy turns out to be ineffective due to abnormal delay variation and TCP flow control behavior.

We report results from a measurement study on the role of the most popular end-to-end security protocol Transport Layer Security (TLS) in the energy consumption of a mobile device. We measured energy consumed by TLS transactions between a Nokia N95 and several popular Web services over WLAN and 3G network interfaces. Our detailed analysis corroborates some earlier results but also reveals, contrary to earlier studies, that the transmission and I/O energy, both in the TLS handshake and the record protocol, far exceed the required computational energy by the actual cryptographic algorithms and that with transactions larger than 500KB, the energy required to transmit the actual data clearly outranks the TLS energy overhead. In addition, we note that the energy consumption varies remarkably between measured services.

Energy consumption caused by wireless transmission poses a big challenge to the battery lifetime of mobile devices. While the potential of using lossless compression for saving energy has been long acknowledged, no general solution has been proposed for applying lossless compression to energy adaptation for mobile services. We propose a proxy-based energy adaptation framework, in which the data to be transmitted is losslessly compressed on a proxy server according to context-aware policies. The context includes factors relevant to computational and communication cost, as well as the user's preferences. We showcase a context-aware policy which aims at minimizing clientside energy consumption caused by transmission and decompression. Using our framework, we implement an energy-aware mobile e-mail service, and present power measurement results that show significant energy savings.

This paper presents SLA (Segment Level Authentication), a transport segment level solution designed to prevent both of the intra-domain and inter-domain source spoofing. SLA is based on public key cryptography authentication. It enables intermediate network nodes the ability to validate the packet authenticity by verifying authentication information carried in packets. Although public key cryptography is computationally intensive and induces the traffic overhead, SLA leverages FPGA (Field Programmable Gate Array) based ECC (Elliptic Curve Cryptography) hardware cryptography accelerator to decrease the computation and traffic overhead. SLA provides incremental deployment and offers incentives for both of hosts and ASes. We find that the SLA is feasible for Gigabit links and can effectively mitigate source spoofing in both of intra-domain and inter-domain networks.

Using Mobile Ad-hoc Networks (MANETs) for audio and video transmission is very promising for application domains such as emergency and rescue. However, audio/video streaming services are not designed for such dynamic and unstable networks. The problems are even more important in so-called sparse MANETs where the node density is relatively low so that disconnections and network partitions are common. We have designed an architecture that combines MANET routing with caching and delay tolerant store-carry-forward operations in an overlay network to improve the quality of audio/video transmission over sparse MANETs. We have implemented a prototype to evaluate the architecture. The results from the experiments demonstrate that our system clearly outperforms simple client-server solutions when the network has temporal disconnections.

While the applications using the Internet have changed over time, TCP is still the dominating transport protocol that carries over 90% of the total traffic. Throughput is the key performance metric for long TCP connections. The achieved throughput results from the aggregate effects of the network path, the parameters of the TCP end points, and the application on top of TCP. Finding out which of these factors is limiting the throughput of a TCP connection -- referred to as TCP root cause analysis -- is important for end users that want to understand the origins of their problems, ISPs that need to troubleshoot their network, and application designers that need to know how to interpret the performance of the application. In this paper, we revisit TCP root cause analysis by first demonstrating the weaknesses of a previously proposed flight-based approach. We next discuss in detail the different possible limitations and highlight the need to account for the application behavior during the analysis process. The main contribution of this paper is a new approach based on the analysis of time series extracted from packet traces. These time series allow for a quantitative assessment of the different causes with respect to the resulting throughput. We demonstrate the interest of our approach on a large BitTorrent dataset.