Emerging live immersive viewing systems require streaming large 360 videos to users via limited wireless bandwidth. Neural-enhanced video streaming offers a promising solution by streaming down-scaled videos and enhancing them by client computation. However, prior systems treated video downscaling and enhancement separately, overlaying existing enhancement techniques onto current video infrastructure to accommodate legacy downscaling methods. This supplemental client design has led to spatial information loss and prohibitive model overheads in 360 video streaming systems. This paper presents Orbis, a redesigned, holistic neural-enhanced video streaming framework that integrates complementary down-scaling and enhancement for live immersive viewing. Orbis is empowered by an enhancement-driven interleaved downscaling approach, an inpainting-based enhancement model tailored to interleaved data, and a multi-scale tile adaptation scheme that optimizes immersive viewing experience in dynamic environments. Experimental results show that Orbis improves viewing experience by up to 60% and reduces wireless bandwidth by up to 49% compared to the best-performing baseline.

Recent advances in computer vision algorithms and video streaming technologies have facilitated the development of edge-server-based video analytics systems, enabling them to process sophisticated real-world tasks, such as traffic surveillance and workspace monitoring. Meanwhile, due to their omnidirectional recording capability, 360-degree cameras have been proposed to replace traditional cameras in video analytics systems to offer enhanced situational awareness. Yet, we found that providing an efficient 360-degree video analytics framework is a non-trivial task. Due to the higher resolution and geometric distortion in 360-degree videos, existing video analytics pipelines fail to meet the performance requirements for end-to-end latency and query accuracy. To address these challenges, we introduce the innovative ST-360 framework specifically designed for 360-degree video analytics. This framework features a spatial–temporal filtering algorithm that optimizes both data transmission and computational workloads. Evaluation of the ST-360 framework on a unique dataset of 360-degree first-responders videos reveals that it yields accurate query results with a 50% reduction in end-to-end latency compared to state-of-the-art methods.

It is well-recognized that correctly detecting shadow boundaries is much more difficult than shadow interior pixels in real-world situations due to the rapid intensity and color transitions near shadow boundaries and the complex background textures. This paper presents a novel boundary-aware shadow detection network to enhance the overall detection performance, especially on challenging shadow boundaries. To enable the network aware of shadow boundaries and balance the supervision between image pixels on the boundary and inside of shadows, the shadow mask is decoupled into a body mask and boundary mask. The feature correction module is designed after feature interaction to eliminate various shadow inferences further. Experimental results show that the proposed network outperforms the state-of-the-art shadow detection methods on four shadow detection benchmark datasets.

First responders operate in hazardous working conditions with unpredictable risks. To better prepare for demands of the job, first responder trainees conduct training exercises that are being recorded and reviewed by the instructors, who check for objects indicating risks within the video recordings (e.g., firefighter with an unfastened gas mask). However, the traditional reviewing process is inefficient due to unanalyzed video recordings and limited situational awareness. For better reviewing experience, a latency-aware Viewing and Query Service (VQS) should be provided. The VQS should support object searching, which can be achieved using the video object detection algorithms. Meanwhile, the application of 360-degree cameras facilitates an unlimited field of view of the training environment. Yet, this medium represents a major challenge because low-latency high-accuracy 360-degree object detection is difficult due to higher resolution and geometric distortion. In this paper, we present the Responders-360 system architecture designed for 360-degree object detection. We propose a Dynamic Selection algorithm that optimizes computation resources while yielding accurate 360-degree object inference. The results, using a unique dataset collected from a firefighting training institute, show that the Responders-360 framework achieves 4x speedup and 25% memory usage reduction compared with the state-of-the-art methods.

Today's mainstream shadow detection methods are manually designed via a case-by-case approach. Accordingly, these methods may only be able to detect shadows for specific scenes. Given the complex and diverse shadow scenes in reality, none of the existing methods can provide a one-size-fits-all solution with satisfactory performance. To address this problem, this paper introduces a new concept, named shadow detection confidence, which can be used to evaluate the effect of any shadow detection method for any given scene. The best detection effect for a scene is achieved by combining prediction results by multiple methods. To measure the shadow detection confidence characteristics of an image, a novel relative confidence map prediction network (RCMPNet) is proposed. Experimental results show that the proposed method outperforms multiple state-of-the-art shadow detection methods on four shadow detection benchmark datasets.