The reference frame memory accesses in inter prediction result in high DRAM bandwidth requirement and power consumption. This problem is more intensive by the adoption of intra block copy (IBC), a new coding tool in the screen content coding (SCC) extension to High Efficiency Video Coding (HEVC). In this paper, we propose a lossless recompression scheme that compresses the reference blocks in intra prediction, i.e., intra block copy, before storing them into DRAM to alleviate this problem. The proposal performs pixel-wise texture analysis with an edge-based adaptive prediction method yet no signaling for direction in bitstreams, thus achieves a high gain for compression. Experimental results demonstrate that the proposed scheme shows a 72% data reduction rate on average, which solves the memory bandwidth problem.

In this paper, we propose a learning-based method to compose a video-story from a group of video clips that describe an activity or experience. We learn the coherence between video clips from real videos via the Recurrent Neural Network (RNN) that jointly incorporates the spatial-temporal semantics and motion dynamics to generate smooth and relevant compositions. We further rearrange the results generated by the RNN to make the overall video-story compatible with the storyline structure via a submodular ranking optimization process. Experimental results on the video-story dataset show that the proposed algorithm outperforms the state-of-the-art approach.

Despite rapid advances in face recognition, there remains a clear gap between the performance of still image-based face recognition and video-based face recognition, due to the vast difference in visual quality between the domains and the difficulty of curating diverse large-scale video datasets. This paper addresses both of those challenges, through an image to video feature-level domain adaptation approach, to learn discriminative video frame representations. The framework utilizes large-scale unlabeled video data to reduce the gap between different domains while transferring discriminative knowledge from large-scale labeled still images. Given a face recognition network that is pretrained in the image domain, the adaptation is achieved by (i) distilling knowledge from the network to a video adaptation network through feature matching, (ii) performing feature restoration through synthetic data augmentation and (iii) learning a domain-invariant feature through a domain adversarial discriminator. We further improve performance through a discriminator-guided feature fusion that boosts high-quality frames while eliminating those degraded by video domain-specific factors. Experiments on the YouTube Faces and IJB-A datasets demonstrate that each module contributes to our feature-level domain adaptation framework and substantially improves video face recognition performance to achieve state-of-the-art accuracy. We demonstrate qualitatively that the network learns to suppress diverse artifacts in videos such as pose, illumination or occlusion without being explicitly trained for them.

In this paper, we propose a spatial propagation networks for learning affinity matrix. We show that by constructing a row/column linear propagation model, the spatially variant transformation matrix constitutes an affinity matrix that models dense, global pairwise similarities of an image. Specifically, we develop a three-way connection for the linear propagation model, which (a) formulates a sparse transformation matrix where all elements can be the output from a deep CNN, but (b) results in a dense affinity matrix that is effective to model any task-specific pairwise similarity. Instead of designing the similarity kernels according to image features of two points, we can directly output all similarities in a pure data-driven manner. The spatial propagation network is a generic framework that can be applied to numerous tasks, which traditionally benefit from designed affinity, e.g., image matting, colorization, and guided filtering, to name a few. Furthermore, the model can also learn semantic-aware affinity for high-level vision tasks due to the learning capability of the deep model. We validate the proposed framework by refinement of object segmentation. Experiments on the HELEN face parsing and PASCAL VOC-2012 semantic segmentation tasks show that the spatial propagation network provides general, effective and efficient solutions for generating high-quality segmentation results.

Partial differential equations (PDEs) have been used to formulate image processing for several decades. Generally, a PDE system consists of two components: the governing equation and the boundary condition. In most previous work, both of them are generally designed by people using mathematical skills. However, in real world visual analysis tasks, such predefined and fixed-form PDEs may not be able to describe the complex structure of the visual data. More importantly, it is hard to incorporate the labeling information and the discriminative distribution priors into these PDEs. To address above issues, we propose a new PDE framework, named learning to diffuse (LTD), to adaptively design the governing equation and the boundary condition of a diffusion PDE system for various vision tasks on different types of visual data. To our best knowledge, the problems considered in this paper (i.e., saliency detection and object tracking) have never been addressed by PDE models before. Experimental results on various challenging benchmark databases show the superiority of LTD against existing state-of-the-art methods for all the tested visual analysis tasks.

Visual tracking is a fundamental task in computer vision. Although many efforts have been made in the past decades, it is still challenging to handle the complex factors in real world tracking scenarios. Ranking methods have shown their power on different data analysis tasks. However, we can not directly utilize this technique on sequential data for tracking. This is because a single ranking model cannot simultaneously reveal both the spatial and the temporal information. In this paper, we propose a novel discriminative sequential ranking (DSR) method to build appearance model for robust visual tracking. Our method can successfully handle both spatial and temporal variations by the coupled ranking processes. Specifically, the spatial process provides a target probability to reflects the intrinsic structure of the object at current frame. Meanwhile, the temporal process provides a background probability (guided by the sequential information) to stably describe the background appearance, which makes our tracker robust for background clutter. Experimental evaluations on the benchmark database with 50 challenging videos confirm that our method outperforms many other state-of-the-art tracking algorithms.

Visual tracking is a fundamental task in computer vision. Although many efforts have been made in the past decades, it is still challenging to handle the complex factors in real world tracking scenarios. Ranking methods have shown their power on different data analysis tasks. However, we can not directly utilize this technique on sequential data for tracking. This is because a single ranking model cannot simultaneously reveal both the spatial and the temporal information. In this paper, we propose a novel discriminative sequential ranking (DSR) method to build appearance model for robust visual tracking. Our method can successfully handle both spatial and temporal variations by the coupled ranking processes. Specifically, the spatial process provides a target probability to reflects the intrinsic structure of the object at current frame. Meanwhile, the temporal process provides a background probability (guided by the sequential information) to stably describe the background appearance, which makes our tracker robust for background clutter. Experimental evaluations on the benchmark database with 50 challenging videos confirm that our method outperforms many other state-of-the-art tracking algorithms.

Conventional saliency detection approaches are human fixation detection and single dominant region detection. However, real-world photographs usually consist of multiple dominant regions. We propose a saliency detection method with the aim to highlight objects as a whole and distinguish objects with different saliency levels. It combines the bottom-up approach and top-down approach via two nested levels of hierarchical segmentations - the coarse level objects and fine level details. We first calculate a preliminary saliency on the fine patches with a random walk model. Then a location cue and an object-level cue are fused to refine the preliminary saliency to emphasize the objects against the background. At last, the object-level saliency map is synthesized via a heat diffusion process restricted by the coarse level patches to enhance object saliency and distinguish saliency between different objects. Extensive evaluation on a publicly available database verifies that our method outperforms the state-of-the-art algorithms.