Video frame interpolation and prediction aim to synthesize frames in-between and subsequent to existing frames, respectively. Despite being closely-related, these two tasks are traditionally studied with different model architectures, or same architecture but individually trained weights. Furthermore, while arbitrary-time interpolation has been extensively studied, the value of arbitrary-time prediction has been largely overlooked. In this work, we present uniVIP - unified arbitrary-time Video Interpolation and Prediction. Technically, we firstly extend an interpolation-only network for arbitrary-time interpolation and prediction, with a special input channel for task (interpolation or prediction) encoding. Then, we show how to train a unified model on common triplet frames. Our uniVIP provides competitive results for video interpolation, and outperforms existing state-of-the-arts for video prediction. Codes will be available at: https://github.com/srcn-ivl/uniVIP

In the research of video quality assessment (VQA), two-branch network [1] has emerged as a promising solution. It decouples VQA with separate technical and aesthetic branches to measure the perception of low-level distortions and high-level semantics respectively. However, we argue that while technical and aesthetic perspectives are complementary, the technical perspective itself should be measured in semantic-aware manner. We hypothesize that existing technical branch struggles to perceive the semantics of high-resolution videos, as it is trained on local mini-patches sampled from videos. This issue can be hidden by apparently good results on low-resolution videos, but indeed becomes critical for high-resolution VQA. This work introduces SiamVQA, a simple but effective Siamese network for high-resolution VQA. SiamVQA shares weights between technical and aesthetic branches, enhancing the semantic perception ability of technical branch to facilitate technical-quality representation learning. Furthermore, it integrates a dual cross-attention layer for fusing technical and aesthetic features. SiamVQA achieves state-of-the-art accuracy on high-resolution benchmarks, and competitive results on lower-resolution benchmarks. Codes will be available at: https://github.com/srcn-ivl/SiamVQA

Recent advances in deep learning have greatly improved No-Reference Video Quality Assessment (NR-VQA), with fragment-based approaches significantly reducing computational complexity through random patch sampling. However, challenges remain due to the reliance on random sampling and the uniform weighting of patches, which limit global observation effectiveness. To address these issues, we propose Saliency Fragments No-Reference Video Quality Assessment (SF-VQA), a novel method prioritizing salient regions. SF-VQA employs Saliency Grid Mini Sampling (SGMS) to select patches near areas of interest and utilizes the Saliency Grid Score (SGS) to compute a weighted quality score based on grid relevance. Experimental results demonstrate that SF-VQA achieves superior performance in Spearman’s Rank Correlation Coefficient (SRCC) and Pearson’s Linear Correlation Coefficient (PLCC) with minimal parameter increase, outperforming existing NR-VQA methods. This demonstrates SF-VQA’s efficiency and effectiveness in advancing NR-VQA.

Video frame interpolation (VFI) has witnessed great progress in recent years. However, existing VFI models still struggle to achieve a good trade-off between accuracy and efficiency. Accurate VFI models typically rely on heavy compute to process all samples, ignoring the fact that easy samples with small motion or clear texture can be well addressed by a fast VFI model and do not require such heavy compute. In this paper, we present a dynamic VFI pipeline with integrated pre-assessment of interpolation difficulty. Specifically, it leverages a difficulty pre-assessment model to measure the difficulty level of interpolating input frames, and then dynamically selects an accurate or a fast VFI model for frame interpolation. Furthermore, we contribute a large-scale annotated dataset to train our VFI difficulty pre-assessment model. Extensive experiments show that our dynamic VFI pipeline can achieve an excellent trade-off between accuracy and efficiency, by feeding hard samples to accurate model, and passing easy samples through fast model.

Neural Radiance Fields (NeRF) have shown remarkable success in representing 3D scenes and generating novel views. However, they often struggle with aliasing artifacts, especially when rendering images from different camera distances from the training views. To address the issue, Mip-NeRF proposed using volumetric frustums to render a pixel and suggested integrated positional encoding (IPE). While effective, this approach requires long training times due to its reliance on MLP architecture. In this work, we propose a novel antialiasing technique that utilizes grid-based representations, usually showing significantly faster training time. In addition, we exploit frequency-domain representation to handle the aliasing problem inspired by the sampling theorem. The proposed method, FreqMipAA, utilizes scale-specific low-pass filtering (LPF) and learnable frequency masks. Scale-specific low-pass filters (LPF) prevent aliasing and prioritize important image details, and learnable masks effectively remove problematic high-frequency elements while retaining essential information. By employing a scale-specific LPF and trainable masks, FreqMipAA can effectively eliminate the aliasing factor while retaining important details. We validated the proposed technique by incorporating it into a widely used grid-based method. The experimental results have shown that the FreqMipAA effectively resolved the aliasing issues and achieved state-of-the-art results in the multi-scale Blender dataset. Our code is available at https://github.com/yi0109/FreqMipAA.

Flow-guided synthesis provides a common framework for frame interpolation, where optical flow is estimated to guide the synthesis of intermediate frames between consecutive inputs. In this paper, we present UPR-Net, a novel Unified Pyramid Recurrent Network for frame interpolation. Cast in a flexible pyramid framework, UPR-Net exploits lightweight recurrent modules for both bi-directional flow estimation and intermediate frame synthesis. At each pyramid level, it leverages estimated bi-directional flow to generate forward-warped representations for frame synthesis; across pyramid levels, it enables iterative refinement for both optical flow and intermediate frame. In particular, we show that our iterative synthesis strategy can significantly improve the robustness of frame interpolation on large motion cases. Despite being extremely lightweight (1.7M parameters), our base version of UPR-Net achieves excellent performance on a large range of benchmarks. Code and trained models of our UPR-Net series are available at: https://github.com/srcn-iv1/UPR-Net.

No-Reference Image Quality Assessment (NR-IQA) techniques have shown improved performance with the help of deep-learning but lightweight architectures have not received attention. In this paper, we propose an NR-IQA network named Lightweight Non-uniform Weighting-based NR-IQA (LiNuIQA) that adopts an efficient network as a feature extractor for a resource constraint environment and harnesses non-uniformly self-weighted local (from each patch) and global information (from all patches) to overcome the inherent problem of low performance stemming from use of lightweight feature extractor. This non-uniform weighting technique is designed to utilize combinations of local and global information with very low resources unlike conventional weighting techniques. The experimental results show that our network outperforms several recently popular NR-IQA networks in terms of both PLCC and SRCC while having the smallest number of parameters and multiply-adds (MAdd) operations. In addition, it can be seen from our experiments that appropriate weighting method plays an important role in IQA and can be implemented with extremely low resources.

We propose a simple yet effective algorithm for motion-based video frame interpolation. Existing motion-based interpolation methods typically rely on an off-the-shelf optical flow model or a U-Net based pyramid network for motion estimation, which either suffer from large model size or limited capacity in handling various challenging motion cases. In this work, we present a novel compact model to simultaneously estimate the bi-directional motions between input frames. It is designed by carefully adapting the ingredients (e.g., warping, correlation) in optical flow research for simultaneous bi-directional motion estimation within a flexible pyramid recurrent framework. Our motion estimator is extremely lightweight (15x smaller than PWC-Net), yet enables reliable handling of large and complex motion cases. Based on estimated bi-directional motions, we employ a synthesis network to fuse forward-warped representations and predict the intermediate frame. Our method achieves excellent performance on a broad range of frame interpolation benchmarks. Code and trained models are available at https://github.com/srcn-ivl/EBME.

Deep Neural Networks have enabled remarkable progress in the field of single image super resolution (SR). However, these models are often large and complex to be applied for real-world applications with limited resources as in mobile and embedded systems. We investigate whether the typical low latency models as MobileNet can be expected of comparable efficiency at SR tasks with recently reported SR performance in the literature. To this end, a moderate and effective architecture, Bottleneck-In-Bottleneck (BIB), is introduced in this paper. The BIB uses multiple expansion factors of the residual blocks in the form of a bottleneck, reducing computation complexity while utilizing advantageous factors of large feature dimensions. We also propose BIBNet with multiple BIB blocks, which can easily adjust its size and computational cost to create a variety of efficient and high-performance models. Extensive experiments show that, with fewer parameters and computations, BIBNet achieves highly competitive performance compared to other conventional SR methods with more complex architectures.

A smart TV system-on-chip (SoC) has very heavy computation and memory demands that must be met with low-cost components. As a result, there is potentially an extremely high utilization of the channel between the SoC and its memory chip. This paper presents the design of a new memory access scheduler customized for the type of memory traffic typically encountered with smart TVs. This includes special accumulated hard real-time graphics requirements, user response-sensitive soft real-time requirements, and the need to provide high memory throughput and priority-handling capabilities even under extremely heavy memory traffic conditions. The simulation results show that the proposed memory access scheduler is able to achieve up to 98% of the ideal upper bound memory throughput when faced with extremely heavy memory traffic-this is a significant improvement over previous schedulers. Novel future prediction and light-handed priority handling methods are used to achieve these results while satisfying the unique real-time requirements of smart TVs.