The eye-tracking video saliency prediction (VSP) task and video salient object detection (VSOD) task both focus on the most attractive objects in video and show the result in the form of predictive heatmaps and pixel-level saliency masks, respectively. In practical applications, eye tracker annotations are more readily obtainable and align closely with the authentic visual patterns of human eyes. Therefore, this paper aims to introduce fixation information to assist the detection of video salient objects under weak supervision. On the one hand, we ponder how to better explore and utilize the information provided by fixation, and then propose a Position and Semantic Embedding (PSE) module to provide location and semantic guidance during the feature learning process. On the other hand, we achieve spatiotemporal feature modeling under weak supervision from the aspects of feature selection and feature contrast. A Semantics and Locality Query (SLQ) Competitor with semantic and locality constraints is designed to effectively select the most matching and accurate object query for spatiotemporal modeling. In addition, an Intra-Inter Mixed Contrastive (IIMC) model improves the spatiotemporal modeling capabilities under weak supervision by forming an intra-video and inter-video contrastive learning paradigm. Experimental results on five popular VSOD benchmarks indicate that our model outperforms other competitors on various evaluation metrics.

Video saliency prediction aims to identify the regions in a video that attract human attention and gaze, driven by bottom-up features from the video and top-down processes like memory and cognition. Among these top-down influences, language plays a crucial role in guiding attention by shaping how visual information is interpreted. Existing methods primarily focus on modeling perceptual information while neglecting the reasoning process facilitated by language, where ranking cues are crucial outcomes of this process and practical guidance for saliency prediction. In this paper, we propose CaRDiff (Caption, Rank, and generate with Diffusion), a framework that imitates the process by integrating multimodal large language model (MLLM), a grounding module, and a diffusion model, to enhance video saliency prediction. Specifically, we introduce a novel prompting method VSOR-CoT (Video Slient Object Ranking Chain of Thought), which utilizes an MLLM with a grounding module to caption video content and infer salient objects along with their rankings and positions. This process derives ranking maps that can be sufficiently leveraged by the diffusion model to accurately decode the saliency maps for the given video. Extensive experiments showcase the effectiveness of VSOR-CoT in improving the performance of video saliency prediction. The proposed CaRDiff performs better than state-of-the-art models on the MVS dataset and demonstrates cross-dataset capabilities on the DHF1k dataset through zero-shot evaluation.

We operate one of the largest short video streaming platforms in the world, serving billions of global users every day. Similar to other platforms, we employ Adaptive Bitrate (ABR) streaming, which adaptively selects an appropriate representation from a group of video representations forming a bitrate ladder. In this paper, instead of developing yet another ABR algorithm, we optimize the bitrate ladder configuration, which constitutes the decision space of ABR, by developing a system called TLadder.

Blind video quality assessment (BVQA) plays a pivotal role in evaluating and improving the viewing experience of end-users across a wide range of video-based platforms and services. Contemporary deep learning-based models primarily analyze video content in its aggressively subsampled format, while being blind to the impact of the actual spatial resolution and frame rate on video quality. In this paper, we propose a modular BVQA model and a method of training it to improve its modularity. Our model comprises a base quality predictor, a spatial rectifier, and a temporal rectifier, responding to the visual content and distortion, spatial resolution, and frame rate changes on video quality, respectively. During training, spatial and temporal rectifiers are dropped out with some probabilities to render the base quality predictor a standalone BVQA model, which should work better with the rectifiers. Extensive experiments on both professionally-generated content and user-generated content video databases show that our quality model achieves superior or comparable performance to current methods. Additionally, the modularity of our model offers an opportunity to analyze existing video quality databases in terms of their spatial and temporal complexity.

User Generated Content (UGC) refers to media generated by users for end-consumers that represent most of the media exchange on social media. UGC is subject to acquisition and transmission limitations that disable access to the pristine, i.e., perfect source content. Evaluating their quality, especially with current pre- and post-processing algorithms or filters, is a major issue for most off-the-shelf full-reference quality metrics. We propose to conduct a benchmark on existing full-reference, non-reference, and aesthetic quality metrics for UGC with special effects. We aim to identify the challenges posed by both UGC and filtering. We then propose a new combination of metrics tailored to enhanced and filtered UGC, which reaches a trade-off between complexity and accuracy.

On a platform of user-generated content (UGC), the uploaded videos need to be encoded again before distribution. For this specific encoding scenario, we propose a novel dataset and a corresponding learning-based scheme that is able to achieve significant bitrate saving without decreasing perceptual quality. In the dataset, each video's label indicates whether it can be encoded with a much lower bitrate while still keeps the same perceptual quality. Models trained on this dataset can then be used to classify the input video and adjust its final encoding parameters accordingly. With enough classification accuracy, more than 20% average bitrate saving can be obtained through the proposed scheme. The dataset will be further expanded to facilitate the study on this problem.

Supporting video transmission over error-prone mobile ad-hoc networks is becoming increasingly important as these networks become more widely deployed. We propose a routing-aware multiple description video coding approach to support video transmission over mobile ad-hoc networks with multiple path transport. We build a statistical model to estimate the packet loss probability of each packet transmitted over the network based on the standard ad-hoc routing messages and network parameters. We then estimate the frame loss probability and dynamically select reference frames in order to alleviate error propagation caused by the packet losses. We conduct experiments using the QualNet simulator that accounts for node mobility, channel properties, MAC operation, multipath routing, and traffic type. The results demonstrate that our proposed method provides 0.7-2.3 dB gains in PSNR for different video sequences under different network settings and guarantees better video quality for a selectably high number of users of the network. Furthermore, we examine the estimation accuracy of our proposed estimation model and show that our model works effectively under various network settings.

Supporting video transmission over error-prone wireless ad-hoc networks is becoming increasingly important as these networks become more widely deployed. In this paper, we propose a routing-aware multiple description video coding approach to support video transmission over wireless ad-hoc networks with path diversity. Our method uses the standard ad-hoc routing messages to estimate the possible packet losses in the networks and dynamically selects reference frames in order to alleviate error propagation caused by packet losses. We conducted experiments using the QualNet simulator that accounts for node mobility, channel properties, MAC operation, multipath routing, and traffic type. The results demonstrate that our proposed method provides up to 2.3 dB gains in PSNR and significantly improves the perceptual video quality for multiple users.

Video coding standards, such as H.264/MPEG-4 AVC, have adopted the hybrid codec paradigm that consists of intra and inter prediction, followed by transform and quantization of the prediction residual. Inter prediction uses block-based motion compensation (BMC) to predict samples in the current picture. BMC, however, suffers from blocking artifacts and discontinuities. Along with deblocking, overlapped block motion compensation (OBMC) has been proposed in the past, primarily to suppress blocking artifacts. However, issues that include decoding complexity and implementation have been impediments to practical applications of OBMC. Here1, we propose a new decoder- and encoder-friendly architecture for inter prediction with OBMC. Furthermore, we show preliminary experimental results that point to coding efficiency gains over the BMC architecture of H.264/MPEG-4 AVC.

Video communications over wireless networks suffers various patterns of losses, including burst losses that cause great degradation in video quality. In this paper, we propose an algorithm based on recursive optimal per-pixel estimate (ROPE) to accurately estimate the overall distortion accounting for the loss pattern. The estimated distortion is applied to the rate-distortion (RD)-based mode selection to provide the optimal tradeoff between intra and inter coding. Simulation results show that in lossy networks, the proposed extended ROPE algorithm achieves gains in average PSNR and PSNRr,fby up to 0.8 dB and 3.6 dB respectively. This shows that the proposed algorithm can enhance error resilience for video communications over networks with burst losses.