We operate one of the largest short video streaming platforms in the world, serving billions of global users every day. Similar to other platforms, we employ Adaptive Bitrate (ABR) streaming, which adaptively selects an appropriate representation from a group of video representations forming a bitrate ladder. In this paper, instead of developing yet another ABR algorithm, we optimize the bitrate ladder configuration, which constitutes the decision space of ABR, by developing a system called TLadder.

Snow-covered forests are widely distributed in middle- and high-latitude regions of the Northern Hemisphere and have significant impacts on global climate change and albedo feedback. However, knowledge of the radiative transfer mechanism in snow–canopy–atmosphere systems is insufficient. Existing bidirectional reflectance models often oversimplify, assuming that snow only persists on the floor, and neglect the interactions between the atmosphere and snow-covered forests. In our previous snow-covered forest bidirectional reflectance (SFBR) model, we considered ground snow, the concentration of soot pollution, needle leaf characteristics, and discontinuous canopy distributions. Furthermore, this study proposed an improved snow-covered forest bidirectional reflectance (SFBR2) model by considering canopy-intercepted snow (CIS) and atmospheric effects. Specifically, the SFBR2 model was constructed by a series of analytical models for CIS, ground snow (asymptotic radiative transfer (ART) snow model), soil [brightness shape moisture (BSM)], needle leaf [leaf incorporating biochemistry exhibiting reflectance and transmittance yields (LIBERTY)], canopy [two-layer version of four-stream scattering by arbitrarily inclined leaves (4SAIL2)], and atmosphere [simplified method for atmospheric correction (SMAC)], in which the CIS is parameterized by snow optical properties and two-stream theory, and the interaction between the atmosphere and snow-covered forests is optimized by a four-stream theory. It makes the model able to simulate the reflectance at the top of the canopy/atmosphere (TOC/TOA). Validations against 3-D large-scale remote sensing data and image simulation framework over heterogeneous 3-D scenes (LESS) model, unmanned aerial vehicle (UAV) observations, and MODIS data indicated a good consistency with SFBR2 model in the reflectance simulation. The established model has the ability to simulate the bidirectional reflectance of different snow-covered forest scenes whether CIS exists or not. Potential applications include satellite signal simulation, radiation mechanism analysis, and parameter inversion in snow-covered forests.

Streaming of live 360-degree video allows users to follow a live event from any view point and has already been deployed on some commercial platforms. However, the current systems can only stream the video at relatively low-quality because the entire 360-degree video is delivered to the users under limited bandwidth. Streaming video falling into user field of view (FoV) can improve bandwidth efficiency of 360-degree video delivery. In this paper, we propose to use the idea of “flocking” to simultaneously improve the accuracy of user FoV prediction and video delivery efficiency for live 360-degree video streaming. By assigning variable playback latencies to users in a streaming session based on their network conditions, a “streaming flock” is formed and led by “strong” users with low playback latencies in the front of the flock. We propose a long short-term memory (LSTM) based collaborative FoV prediction scheme where the FoV traces of users in the front of the flock are utilized to predict the FoV of users behind them. Given a predicted FoV, we develop an optimal rate allocation strategy to maximize the perceptual quality. By conducting experiments using real-world user FoV traces and LTE/5 G network bandwidth traces, we evaluate the gains of the proposed strategies over several benchmarks. Our experimental results demonstrate that the proposed streaming system can increase the overall quality dramatically by about 10 dB compared with heuristic FoV prediction strategy. In addition, the network-aware flocking formation can further reduce the video freeze without influencing video quality.

Robot locomotion is a major challenge in robotics. Model-based approaches are vulnerable to model errors, and incur high computation overhead resulted from long control horizon. Model-free approaches are trained with a large number of training samples, which are expensive to obtain. In this paper, we develop a hybrid control and learning framework, called Reinforced iLQR (RiLQR), which combines the advantages of model-based iLQR control with model-free RL policy learning to simultaneously achieve high sample efficiency, low computation overhead, and high robustness against model errors in robot locomotion. Through extensive evaluation on the Mujoco platform, we demonstrate that RiLQR outperforms the state-of-the-art model-based and model-free baselines by big margins in a set of tasks with different complexities.

It is highly challenging to simultaneously achieve high-rate and low-latency in live video streaming. Chunk-based streaming and playback speed adaptation are two promising new trends to achieve high user Quality-of-Experience (QoE). To thoroughly understand their potentials, we develop a detailed chunk-level dynamic model that characterizes how video rate and playback speed jointly control the evolution of a live streaming session. Leveraging on the model, we first study the optimal joint video rate-playback speed adaptation as a non-linear optimal control problem. We further develop model-free joint adaptation strategies using deep reinforcement learning. Through extensive experiments, we demonstrate that our proposed joint adaptation algorithms significantly outperform rate-only adaptation algorithms and the recently proposed low-latency video streaming algorithms that separately adapt video rate and playback speed without joint optimization. In a wide-range of network conditions, the model-based and model-free algorithms can achieve close-to-optimal trade-offs tailored for users with different QoE preferences.

Low-latency is a critical user Quality-of-Experience (QoE) metric for live video streaming. It poses significant challenges for streaming over the Internet. In this paper, we explore the design space of low-latency live streaming by developing dynamic models and optimal adaptation strategies to establish QoE upper bounds as a function of the allowable end-to-end latency. We further develop practical live streaming algorithms within the iterative Linear Quadratic Regulator (iLQR) based Model Predictive Control and Deep Reinforcement Learning frameworks, namely MPC-Live and DRL-Live, to maximize user live streaming QoE by adapting the video bitrate while maintaining low end-to-end video latency in dynamic network environment. Through extensive experiments driven by real network traces, we demonstrate that our live streaming algorithms can achieve close-to-optimal performance within the latency range of two to five seconds.

Virtual Reality (VR) and Augmented Reality (AR) technologies have become popular in recent years. Encoding and transmitting the omni-directional or $360^\circ $ video is critical and challenging for those applications. The $360^\circ $ video requires much higher bandwidth than the traditional planar video. A premium quality $360^\circ $ video with 120 frames per second (fps) and 24K resolution can easily consume bandwidth in the range of Gigabits-per-second~\cite1. On the other hand, at any given time, a user only watches a small portion of the $360^\circ$ scope within her Field-of-View (FoV). An effective way to reduce the bandwidth requirement of $360^\circ $ video is through FoV-adaptive streaming, which codes and delivers the predicted FoV region at higher quality, and discards or codes at lower quality the remaining regions. Such strategy has been quite extensively studied for video-on-demand \citefov_adapt_2,fov_adapt_3,1,tile_based_3,qian2016optimizing and live video streaming applications\citelive_1,live_2,live_3, sun2020flocking. Interactive applications, such as conferencing, gaming, and remote collaboration, can also benefit from $360^\circ $ video by creating an immersive environment for participants to interact with each other citeinteractive_gamming \citevr_conferencing \citelee2015outatime. However, realtime coding and streaming of $360^\circ $ video with extremely low latency, required for interactive applications, has not been sufficiently addressed. This work focuses on developing low-latency and FoV-adaptive coding and streaming strategies for interactive $360^\circ$ video streaming. We assume the sender and the receiver are connected by a network path with dynamically varying throughput without short-latency guarantee. The sender is either the video source, or a proxy server relaying the source video. The receiver is either the end user device that directly renders the video, or a local edge server that renders the video and transmit to the end user \citeHou2017.

Streaming of live 360-degree video allows users to follow a live event from any view point and has already been deployed on some commercial platforms. However, the current systems can only stream the video at relatively low-quality because the entire 360-degree video is delivered to the users under limited bandwidth. In this paper, we propose to use the idea of "flocking" to improve the performance of both prediction of field of view (FoV) and caching on the edge servers for live 360-degree video streaming. By assigning variable playback latencies to all the users in a streaming session, a "streaming flock" is formed and led by low latency users in the front of the flock. We propose a collaborative FoV prediction scheme where the actual FoV information of users in the front of the flock are utilized to predict of users behind them. We further propose a network condition aware flocking strategy to reduce the video freeze and increase the chance for collaborative FoV prediction on all users. Flocking also facilitates caching as video tiles downloaded by the front users can be cached by an edge server to serve the users at the back of the flock, thereby reducing the traffic in the core network. We propose a latency-FoV based caching strategy and investigate the potential gain of applying transcoding on the edge server. We conduct experiments using real-world user FoV traces and WiGig network bandwidth traces to evaluate the gains of the proposed strategies over benchmarks. Our experimental results demonstrate that the proposed streaming system can roughly double the effective video rate, which is the video rate inside a user's actual FoV, compared to the prediction only based on the user's own past FoV trajectory, while reducing video freeze. Furthermore, edge caching can reduce the traffic in the core network by about 80%, which can be increased to 90% with transcoding on edge server.

Low-latency is a critical user Quality-of-Experience (QoE) metric for live video streaming. It poses significant challenges for streaming over the Internet. In this paper, we explore the design space of low-latency live video streaming by developing dynamic models and optimal control strategies. We further develop practical live video streaming algorithms within the Model Predictive Control (MPC) framework, namely MPC-Live, to maximize user QoE by adapting the video bitrate while maintaining low end-to-end video latency in dynamic network environment. Through extensive experiments driven by real network traces, we demonstrate that our live video streaming algorithms can improve the performance dramatically within latency range of two to five seconds.

360° video streaming is a key component of the emerging Virtual Reality (VR) and Augmented Reality (AR) applications. In 360° video streaming, a user may freely navigate through the captured 360° video scene by changing her desired Field-of-View. High-throughput and low-delay data transfers enabled by 5G wireless networks can potentially facilitate untethered 360° video streaming experience. Meanwhile, the high volatility of 5G wireless links present unprecedented challenges for smooth 360° video streaming. In this paper, novel multi-path multi-tier 360° video streaming solutions are developed to simultaneously address the dynamics in both network bandwidth and user viewing direction. We systematically investigate various design trade-offs on streaming quality and robustness. Through simulations driven by real 5G network bandwidth traces and user viewing direction traces, we demonstrate that the proposed 360° video streaming solutions can achieve a high-level of Quality-of-Experience (QoE) in the challenging 5G wireless network environment.

CitationBuxton and Myers (1986) compared a specialized document navigation device and a scrollbar. Their device tracked finger movements along two touch-sensitive areas: one for absolute movement and one for relative movement. They found that their system was faster when navigating a 5-page document. We identified finger-tracking issues with the Buxton and Myers device for larger documents (100 pages) and developed an alternative device employing a range sensor and rotary encoder to track finger movement. We ran a new experiment comparing the traditional scrollbar, the Buxton and Myers design, and our new design. Both the Buxton and Myers design and our new design were poorly received by users compared to the scrollbar. Our results indicate that in a large document, factors beyond finger tracking accuracy influence the performance of a device providing absolute movement. From these results, we identify possible improvements necessary to implement effective and practical absolute navigation devices.