Multi-party real-time video conferencing has become an indispensable service in industrial production and daily life. However, the current dynamic and limited network resources can no longer meet the growing service demands of users, resulting lagging and low visual quality. The emerging semantic transmission, together with the network-wide redundant computation capacity, provides new opportunities towards a new paradigm of semantic video conferencing. The key challenge of such fusion lies in the interplay of traditional streaming adaptation and the new semantic processing, calling for a holistic mechanism to optimize the service provision with compatibility and efficiency. In this paper, we for the first time address this challenge, and propose SemConf, a novel framework that integrate the semantic transmission into the video conferencing towards optimal user QoE. Our extensive evaluations, against state-of-the-art baselines, reveal that SemConf achieves a substantial improvement in QoE, with up to 33.6% enhancement in bandwidth-constrained environment. Overall, this work highlights the critical role of the coordination algorithm in balancing computational load and network throughput, showcasing SemConf as a transformative approach in the realm of semantic video conferencing.

Video conferencing has become indispensable in human communication. Researchers are exploring immersive capabilities to enhance video conferencing experiences by delivering realistic interactions. However, existing methods have stringent and extra hardware beyond a typical video conference, including multiple depth cameras, large screens, and headsets, which pose obstacles to the widespread adoption due to high costs and complex setups. Thus, there is an urgent demand for light-weight systems using only on-hand devices including single RGB camera and standard screen, without additional hardware. We propose DVCO, a novel 3D video conferencing system via on-hand devices. With DVCO, users can experience lifelike virtual conferencing that includes natural contact and interactive features. To achieve this, DVCO has two main components. Virtual Camera Transformation (VCT) and New View Generator (NVG). VCT computes a downscaled sender image from tracking to determine viewpoint and gaze vector, enhancing virtual presence on standard screens. NVG takes an input frame and desired view angle to produce an output reflecting the new view from a single RGB camera. Together, these provide an affordable, easy-to-integrate enhancement for current video conferencing systems without expensive upgrades. Through a user study, it has been demonstrated that DVCO offers an exceptional level of immersion when compared to traditional systems. Experiments are conducted to showcase the superior performance of VCT and NVG in comparison to baseline methods.

Virtual Reality (VR) has emerged as a transformative platform for remote collaboration, but its adoption for video conferencing is hindered by challenges related to facial expression reconstruction and computational resource constraints, especially on economical mobile VR headsets. This paper introduces a novel system for VR video conferencing that addresses these challenges through two key modules: Predictive Generation and Collaborative Computation. Predictive Generation leverages multimodal inputs, including voice, head motion, and eye blinks, to synthesize realistic facial animations with low latency, eliminating the need for high-precision hardware. Collaborative Computation enhances computational efficiency by employing a game-theoretic framework for resource sharing among users. Experimental evaluations demonstrate that our system delivers immersive and realistic VR video conferencing experiences with superior facial expression reconstruction and efficient resource utilization. Our approach makes VR video conferencing more accessible and practical for a broader audience across mobile headsets.

Recently, content-aware methods have been employed to reduce bandwidth and enhance the quality of Internet video delivery. These methods involve training distinct content-aware super-resolution (SR) models for each video chunk on the server, subsequently streaming the low-resolution (LR) video chunks with the SR models to the client. Prior research has incorporated additional partial parameters to customize the models for individual video chunks. However, this leads to parameter accumulation and can fail to adapt appropriately as video lengths increase, resulting in increased delivery costs and reduced performance. In this paper, we introduce RepCaM++, an innovative framework based on a novel Re- parameterization Content-aware Modulation (RepCaM) module that uniformly modulates video chunks. The RepCaM framework integrates extra parallel-cascade parameters during training to accommodate multiple chunks, subsequently eliminating these additional parameters through re- parameterization during inference. Furthermore, to enhance RepCaM's performance, we propose the Transparent Visual Prompt (TVP), which includes a minimal set of zero-initialized image-level parameters (e.g., less than 0.1%) to capture fine details within video chunks. We conduct extensive experiments on the VSD4K dataset, encompassing six different video scenes, and achieve state-of-the-art results in video restoration quality and delivery bandwidth compression.

Virtual Reality (VR) has become increasingly popular for remote collaboration, but video conferencing poses challenges when the user's face is covered by the headset. Existing solutions have limitations in terms of accessibility. In this paper, we propose HeadsetOff, a novel system that achieves photorealistic video conferencing on economical VR headsets by leveraging voice-driven face reconstruction. HeadsetOff consists of three main components: a multimodal predictor, a generator, and an adaptive controller. The predictor effectively predicts user future behavior based on different modalities. The generator employs voice, head motion, and eye blink to animate the human face. The adaptive controller dynamically selects the appropriate generator model based on the trade-off between video quality and delay. Experimental results demonstrate the effectiveness of HeadsetOff in achieving high-quality, low-latency video conferencing on economical VR headsets.