Live volumetric video streaming enables immersive user experiences but poses significant challenges due to the high bandwidth requirement that 3D representations entail. Recent research has focused on reducing volumetric video bandwidth, but it struggles to effectively address temporal redundancy under real-time constraints, limiting its applicability for live streaming scenarios. To address these challenges, we present DeltaStream, a novel live volumetric video streaming system that efficiently encodes 3D point clouds by leveraging 2D information. By utilizing 2D RGB and depth frames, DeltaStream efficiently infers inter-frame changes to reduce the streaming bandwidth of volumetric video. Furthermore, DeltaStream introduces an adaptive block-based approach that can reduce the client-side decoding load. Through extensive evaluations, our results demonstrate that DeltaStream reduces bandwidth by up to 71% with 1.63× faster decoding speed while maintaining visual quality compared to state-of-the-art systems.

As the user count for video-related services continues to grow, ensuring high-quality service (QoS) for them will become even more crucial in the future. Many studies have been conducted to enhance the quality of on-demand video streaming using adaptive bitrate (ABR) algorithms and artificial intelligence (AI). This study addresses a more complex challenge than that of on-demand video streaming: enhancing service quality in multi-party, full-duplex communication scenarios, such as video conferences. We propose a deep reinforcement learning (DRL)-based video bitrate allocation framework for a media server in the video conferencing system. Our framework aims to increase overall QoS by applying an appropriate bitrate for each connection in a video conferencing call, considering the network conditions for users. We train the DRL model to maximize the aggregate QoS of users in a meeting by constructing a feedback loop between a media server and a DRL server. Our experimental results demonstrate that our framework can adaptively control the video bitrate according to changes in network conditions. As a result, it achieves higher video bitrates in the user application (approximately, 5% under stable network conditions and 35% over the highly dynamic network conditions) compared to the existing rule-based bandwidth allocation.

Many studies are applying reinforcement learning to real-world problems. However, this is a difficult problem, and its application in the real world requires solving many challenges. Therefore, in order to solve this problem well, it is necessary to understand the process of data collection in the real world and the data collected through this process. Video conferencing is also an example of the application of reinforcement learning in the real world, so it is necessary to understand the video conferencing system used and the data collected through it. To this end, this paper presents a process to collect network status information data from a video conferencing system and introduces data and model analysis methods for effective reinforcement learning environment settings and problem definition. In addition, among various problems in video conferencing, the video quality selection problem is set as the target problem, and we trained the model to perform the introduced feature importance analysis. We also included the performance evaluation results and correlation with data analysis results.

Many studies have applied machine learning to bitrate control to increase Quality of Experience (QoE) of video streaming services in highly dynamic networks. However, their solutions mainly focused on HTTP adaptive streaming with one-to-one connections. This paper studies video conferencing applications where multi-party, full-duplex communication happens among participants. In particular, we propose Muno, a Deep Reinforcement Learning (DRL)-based bandwidth prediction framework for multi-party video conferencing systems. Muno learns and predicts an appropriate bitrate for each connection in a multi-party conferencing call. We trained Muno to maximize the QoE of individual connections by constructing a feedback loop between a media server and DRL servers. Our experimental results show that Muno achieves a higher video streaming rate and lower delay compared to state-of-the-art rulebased algorithms.

While many studies have been conducted to apply reinforcement learning (RL) to real world problems beyond games such as Atari, video conferencing is also one of real world applications. In video conferencing, reinforcement learning is used to control the bitrate to improve the user's quality of experience (QoE). However, real world problems such as video conferencing have different characteristics compared to electronic games. Usually the rewards in real world problems are not clear or abstract, and this makes it difficult to design the RL model and training process of the model to maximize the cumulative reward. Therefore, in this paper, we present the method for stabilizing the training of the models that apply reinforcement learning to video conferencing. In addition, we established a simulation environment that can train deep RL models in 1-to-1 video conferencing. An evaluation is performed to analyze the difference between the baseline model and the models generated using the stabilization method in the simulation environment.

With the advancement of artificial intelligence(AI) technology, AI is being used in various industries such as factory automation and autonomous driving. Video conferencing systems have also added functions that use AI to overcome the limitations of existing algorithms, for example, super resolution and virtual background functions using image segmentation. However, web-based video conferencing limits the application of these features due to a limited web browser environment. In this paper, we introduce several approaches to apply deep learning in a web browser environment to provide the features that use deep learning models, and introduce image segmentation models used for virtual background functions in each method and evaluate their performance. Finally, we discuss areas that need to be considered to apply deep learning models to web-based video conferencing.

Scanning and acquiring a 3D indoor environment suffers from complex occlusions and misalignment errors. The reconstruction obtained from an RGB-D scanner contains holes in geometry and ghosting in texture. These are easily noticeable and cannot be considered as visually compelling VR content without further processing. On the other hand, the well-known Manhattan World priors successfully recreate relatively simple structures. In this article, we would like to push the limit of planar representation in indoor environments. Given an initial 3D reconstruction captured by an RGB-D sensor, we use planes not only to represent the environment geometrically but also to solve an inverse rendering problem considering texture and light. The complex process of shape inference and intrinsic imaging is greatly simplified with the help of detected planes and yet produces a realistic 3D indoor environment. The generated content can adequately represent the spatial arrangements for various AR/VR applications and can be readily composited with virtual objects possessing plausible lighting and texture.