Visual Relationship Forecasting (VRF) in video aims to anticipate relations among objects without observing future visual content. The task relies on capturing and modeling the semantic coherence in object interactions, as it underpins the evolution of events and scenes in videos. However, existing VRF datasets provide limited support for learning such coherence. Their noisy annotations fail to reflect distinct action or scene changes, and weak correlations exist between different actions and relational transitions in subject-object pairs. Furthermore, existing methods struggle to distinguish similar relationships and overfit to unchanging relationships in consecutive frames rather than reflecting the semantic coherence of object interactions. To address these challenges, we present SemCoBench, a benchmark that emphasizes semantic coherence for visual relationship forecasting in video. Based on action labels and short-term subject-object pairs, SemCoBench decomposes relationship categories and dynamics by cleaning and reorganizing video datasets to ensure predicting semantic coherence in object interactions. In addition, we propose the Semantic Coherent Transformer (SemCoFormer), which consists of a Relationship Augmented Module (RAM) and a Coherent Reasoning Module (CRM). The RAM bidirectionally enhances cross-modal features to better distinguish similar relationships, while the CRM performs sparse encoding of cross-frame relation transitions to model dynamic relational changes. The experimental results on SemCoBench demonstrate that modeling the semantic coherence is a key step toward reasonable, fine-grained, and diverse visual relationship forecasting, contributing to a more comprehensive understanding of video scenes. Our project page: https://lyao-61.github.io/ SemCo/.

Rate control (RC) is a critical component in learned image compression (LIC), particularly in the emerging JPEG-AI standard, which enables adaptive bitrate achievement to meet diverse bandwidth constraints. JPEG-AI default RC employs an iterative optimization process, wherein a pre-trained RC model is selected and the (generated) latent representations are adjusted based on the mismatch between actual and target bitrates. Despite satisfactory results, such a trial-and-error paradigm necessitates multiple processing cycles, resulting in inevitable computational overhead. We propose an efficient neural rate control framework for JPEG-AI to address this limitation. Our idea is to train a ResNet-based neural control (NRC) to learn the mapping from the input images and target bitrates to the optimal coding parameters. The trained NRC can then be applied to predict the coding parameters based on the new input images and target bitrates directly. Experimental results on DIV2K and MSCOCO datasets show that our NRC achieves comparable rate-distortion performance while reducing encoding time by about 5× compared to JPEG-AI default RC.

Autonomous driving progress relies on large-scale annotated datasets. In this work, we explore the potential of generative models to produce vast quantities of freely-labeled data for autonomous driving applications and present SubjectDrive, the first model proven to scale generative data production in a way that could continuously improve autonomous driving applications. We investigate the impact of scaling up the quantity of generative data on the performance of downstream perception models and find that enhancing data diversity plays a crucial role in effectively scaling generative data production. Therefore, we have developed a novel model equipped with a subject control mechanism, which allows the generative model to leverage diverse external data sources for producing varied and useful data. Extensive evaluations confirm SubjectDrive's efficacy in generating scalable autonomous driving training data, marking a significant step toward revolutionizing data production methods in this field.

In this paper, we present HomoGen, an enhanced video inpainting method based on homography propagation and diffusion models. HomoGen leverages homography registration to propagate contextual pixels as priors for generating missing content in corrupted videos. Unlike previous flow-based propagation methods, which introduce local distortions due to point-to-point optical flows, homography-induced artifacts are typically global structural distortions that preserve semantic integrity. To effectively utilize these priors for generation, we employ a video diffusion model that inherently prioritizes semantic information within the priors over pixel-level details. A content-adaptive control mechanism is proposed to scale and inject the priors into intermediate video latents during iterative denoising. In contrast to existing transformer-based networks that often suffer from artifacts within priors, leading to error accumulation and unrealistic results, our denoising diffusion network can smooth out artifacts and ensure natural outputs. Extensive experiments demonstrate the effectiveness of the proposed method qualitatively and quantitatively.

Neural lossless image compression methods have recently achieved impressive compression ratios by fitting neural networks to represent data distributions of large datasets. However, these methods often require complex networks to capture intricate data distributions effectively, resulting in high decoding complexity. In this paper, we present a novel approach named Fitted Neural Lossless Image Compression (FNLIC) that enhances efficiency through a two-phase fitting process. For each image, a latent variable model is overfitted to optimize the representation of the individual image’s probability distribution, which is inherently simpler than the distribution of an entire dataset and requires less complex neural networks. Additionally, we pre-fit a lightweight autoregressive model on a comprehensive dataset to learn a beneficial prior for overfitted models. To improve coordination between the pre-fitting and overfitting phases, we introduce independent fitting for the pre-fitter and the adaptive prior transformation for the overfitted model. Extensive experimental results on high-resolution datasets show that FNLIC achieves competitive compression ratios compared to both traditional and neural lossless image compression methods, with decoding complexity significantly lower than other neural methods of similar performance. The code is at https://github.com/ZZ022/FNLIC.

Space-time video resampling aims to conduct both spatial-temporal downsampling and upsampling processes to achieve high-quality video reconstruction. Although there has been much progress, some major challenges still exist, such as how to preserve motion information during temporal resampling while avoiding blurring artifacts, and how to achieve flexible temporal and spatial resampling factors. In this paper, we introduce an Invertible Motion Steganography Module (IMSM), designed to embed motion information from high-frame-rate videos into downsampled frames with lower frame rates in a visually imperceptible manner. Its reversible nature allows the motion information to be recovered, facilitating the reconstruction of high-frame-rate videos. Furthermore, we propose a 3D implicit feature modulation technique that enables continuous spatiotemporal resampling. With tailored training strategies, our method supports flexible frame rate conversions, including non-integer changes like 30 FPS to 24 FPS and vice versa. Extensive experiments show that our method significantly outperforms existing solutions across multiple datasets in various video resampling tasks with high flexibility. Codes will be made available at the URL https://github.com/hahazh/CSTVR.

Employing neural networks (NN) for rate-distortion (R-D) modeling in image/video coding has received growing attention. Typically, NNs are used to capture the R-D relationship through the rate-quantizer (R-Q) and distortion-quantizer (D-Q) models in various works. However, existing NN-based schemes do not consider the monotonicity of the R-D relationship, resulting in inferior distortion predictions. We design a monotonic neural network-based scheme to tackle this issue. We first adopt linear scaling to ground truth to make the R-Q and D-Q relationship always monotonic increasing. We then train an NN to learn the mapping from the latent features and quantizer to the rate/distortion while enforcing the derivative of NN predicted R/D w.r.t the QP is strictly positive. As such, the monotonic R-D relationship can be maintained. Simulation results on the Vimeo dataset with H.266/VVC show that the proposed approach is better than the state-of-the-art NN-based scheme in prediction accuracy and maintaining monotonicity.

Spatio-Temporal Video Grounding (STVG) aims to localize spatio-temporal tubes of specific objects or actions within videos based on textual queries. Despite significant progress, existing methods struggle to generalize effectively to real-world scenarios due to the limited quantity and diversity of annotated data. In this paper, we introduce RealVG, a robust and training-free pipeline that leverages powerful Multimodal Large Language Models (MLLMs) through question-answering to tackle STVG in the wild. To address the challenges posed by complex real-world videos and queries, we propose a spatio-temporal decoupling module and a query-guided visual token filter to decompose intricate scenes and refine target-oriented perception, enhancing the robustness and adaptability of MLLMs. Specifically, the spatio-temporal decoupling module breaks down videos and queries into simpler sub-scenes and sub-queries, reducing complexity and promoting a precise understanding of static visual elements. Meanwhile, the query-guided visual token filter eliminates irrelevant tokens, sharpening focus on the target object and improving short-range action perception. Experimental results demonstrate that RealVG achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings, despite requiring no STVG data for training.

Large Multimodal Models (LMMs) have demonstrated impressive performance in short video understanding tasks but face great challenges when applied to long video understanding. In contrast, Large Language Models (LLMs) exhibit outstanding capabilities in modeling long texts. Existing work attempts to address this issue by introducing long video-text pairs during training. However, these approaches require substantial computational and data resources. In this paper, we tackle the challenge of long video understanding from the perspective of context windows, aiming to apply LMMs to long video tasks without retraining on long video datasets. We first conduct an in-depth analysis of why pretrained LMMs struggle to understand lengthy video content, identifying that discrepancies between visual and language modalities lead to different context windows for visual and language tokens, making it difficult to directly extend the visual tokens to match the language context window. Based on this, we propose to adapt LMMs for long video understanding tasks by extending the visual context window, eliminating the need for retraining on large-scale long video datasets. To further mitigate the significant memory consumption caused by long sequences, we introduce a progressive pooling inference strategy that selectively adjusts the spatial resolution of frame embeddings, reducing the number of visual tokens while retaining important spatial information. Across multiple long video understanding benchmarks, our method consistently improves the performance as the number of video frames increases. On the MLVU benchmark, our method outperforms GPT-4o, even though our model size is only 7B. Additionally, in the 256-frame setting, our method reduces memory usage by approximately 45% compared to the baseline, without introducing any performance loss.

While deep learning has demonstrated impressive results in joint demosaicing and denoising (JDD), its computational and memory requirements hinder practical deployment. To address this, we propose an asymmetric look-up table (LUT)-based method, AsLUT, which performs asymmetric processing across different color channels and relative positions according to the mosaic pattern. First, Channel-wise Asymmetric Indexing is proposed to fully exploit intra-channel information under constrained input dimensions, obtaining initial full-resolution RGB images. Additionally, since the element fidelity varies periodically across spatial positions according to the distribution of known elements in the raw images, Mosaic Point-wise LUTs (MP-LUTs) and Mosaic Modulation (MM) are proposed to explicitly adapt to this fidelity variation. The former assigns distinct color mapping operators to elements at different relative positions, and the latter applies position-specific scalar weights to adjust pixel values in the image domain. Furthermore, Pattern-Guided Channel Fusion (PGCF) exploits spatial and channel correlations by performing fusion guided by mosaic pattern. Extensive experiments demonstrate that AsLUT achieves superior performance compared to state-of-the-art methods both quantitatively and qualitatively.

In this work, we first propose DiffVC-OSD, a One-Step Diffusion-based Perceptual Neural Video Compression framework. Unlike conventional multi-step diffusion-based methods, DiffVC-OSD feeds the reconstructed latent representation directly into a One-Step Diffusion Model, enhancing perceptual quality through a single diffusion step guided by both temporal context and the latent itself. To better leverage temporal dependencies, we design a Temporal Context Adapter that encodes conditional inputs into multi-level features, offering more fine-grained guidance for the Denoising Unet. Additionally, we employ an End-to-End Finetuning strategy to improve overall compression performance. Extensive experiments demonstrate that DiffVC-OSD achieves state-of-the-art perceptual compression performance, offers about 20× faster decoding and an 86.92% bitrate reduction compared to the corresponding multistep diffusion-based variant.

Learned image compression methods have shown remarkable performance and expansion potential compared to traditional codecs. Currently, there are two mainstream image compression frameworks: one uses stacked convolution and other uses window-based self-attention for transform coding, most of which aggregate valuable dependencies in a fixed spatial range. In this paper, we focus on extending content-adaptive aggregation capability and propose a dynamic kernel-based transform coding. The proposed adaptive aggregation generates kernel offsets to capture valuable information with dynamic sampling convolution to help transform. With the adaptive aggregation strategy and the sharing weights mechanism, our method can achieve promising transform capability with acceptable model complexity. Besides, considering the coarse hyper prior, the channel-wise, and the spatial context, we formulate a generalized entropy model. Based on it, we introduce dynamic kernel in hyper-prior to generate more expressive side information context. Furthermore, we propose an asymmetric sparse entropy model according to the investigation of the spatial and variance characteristics of the grouped latents. The proposed entropy model can facilitate entropy coding to reduce statistical redundancy while maintaining inference efficiency. Experimental results demonstrate that our method achieves superior rate–distortion performance on three benchmarks compared to the state-of-the-art learning-based methods.

Efficient image compression is crucial for remote sensing satellite systems, as it determines the performance of machine vision applications analyzing the downlinked image data at ground stations. However, existing conventional or learning-based image compression approaches exhibit limitations in either high complexity or undesirable vision task performance. This letter investigates an efficient neural image compression standard, JPEG-AI-based self-supervised remote sensing image compression approach, SS-JPEG-AI, for semantic segmentation tasks. Our approach maintains the low-complexity advantages of JPEG-AI while incorporating (1) a computationally efficient transformer-based attention mechanism that enhances reconstruction quality without increasing encoder complexity for onboard systems and (2) a contrastive learning strategy that improves feature discriminability and sharpens inter-category decision boundaries for segmentation tasks. Compared to the state-of-the-art image compression methods, SS-JPEG-AI achieves better Bjøntegaard Delta-rate (BD-rate) performance across the mean Intersection over Union (mIoU) and mean F-score (mFscore) while maintaining up to30 timessmaller computational complexity.

Most existing single-image super-resolution (SISR) methods focus on addressing predefined uniform degradations, such as bicubic. However, these methods often perform poorly in real-world scenarios due to complicated and varying realistic degradations. In this paper, we propose a novel information bottleneck-based self-distillation method (IBSD) to boost lightweight networks for real-world image super-resolution. The proposed IBSD leverages the principle of information bottleneck to guide SR networks to learn invariant correlations from low-resolution (LR) to high-resolution (HR) across various degradations, thereby improving their generalization capacity. Specifically, the target super-resolution network (i.e., student) is interpreted as a Markov chain, and the distillation process is carried out through two modules. Mutual information (MI) estimation networks are used to quantify the mutual information between adjacent nodes within the Markov chain. To enhance robustness against blur and noise in real-world scenarios, an auxiliary loss with a progressive soft target is employed to better identify what is effective for reconstruction in the high-frequency domain. Minimizing the mutual information while preserving task-relevant features can help remove information that reflects spurious correlations between specific degradations and reconstructed targets. Experiments conducted on real-world image super-resolution datasets demonstrate that our proposed method can significantly improve the performance of recent lightweight SR models without adding any extra inference complexity, and it outperforms existing self-distillation approaches. Code is publicly available athttps://github.com/hanzhu1121/IBSD.

Variations in scene complexity and image quality across remote sensing images lead to inconsistent performance when applying pre-trained semantic segmentation models. To ensure quality control for downstream tasks, quality assessment on interpretation results becomes a challenging task, primarily due to the limited availability of reference annotations. To address this issue, we propose RS-SQA, a remote sensing semantic segmentation quality assessment model based on the vision language model (VLM) for images without ground truth. RS-SQA uses a dual-branch architecture that leverages a RS VLM for semantic understanding and utilizes intermediate features from segmentation methods to extract implicit information. We fuse these two feature streams via a semantic-guided module, yielding accurate quality predictions. Specifically, we introduce CLIP-RS, a large-scale pre-trained VLM trained on 10M remote sensing images with purified captions. Feature visualizations confirm that CLIP-RS can effectively differentiate between various levels of segmentation quality. To facilitate the development of RS semantic segmentation quality assessment, we present RS-SQED, a new dedicated dataset sampled from 4 major RS semantic segmentation datasets and annotated with segmentation accuracy scores from 8 representative segmentation methods. Experiments on the established dataset demonstrate that RS-SQA outperforms state-of-the-art quality assessment models and aids in selecting the most suitable pre-trained models for remote sensing semantic segmentation by comparing predicted accuracies across different models.

This paper addresses the task of space-time video super-resolution (STVSR). Existing methods generally suffer from inaccurate motion estimation and motion compensation (MEMC) problems for large motions. Inspired by recent progress in physics-informed neural networks, we model the challenges of MEMC in STVSR as a mapping between two continuous function spaces. Specifically, our approach transforms independent low-resolution representations in the coarse-grained continuous function space into refined representations with enriched spatiotemporal details in the fine-grained continuous function space. To achieve efficient and accurate MEMC, we design a Galerkin-type attention function to perform frame alignment and temporal interpolation. Due to the linear complexity of the Galerkin-type attention mechanism, our model avoids patch partitioning and offers global receptive fields, enabling precise estimation of large motions. The experimental results show that the proposed method surpasses state-of-the-art techniques in both fixed-size and continuous space-time video super-resolution tasks. Code is publicly available at the URL https://github.com/hahazh/STVSR-NO.

POI recommendation plays an important role in many applications, such as mobility prediction and location-based advertisements. Existing POI recommendation methods mainly capture the observed patterns in user visits for recommendations, without a comprehensive consideration of the underlying reasons behind the visits. Therefore, different causes of a visit, i.e., users’ interest and geographical context, are entangled. When the underlying causes change (e.g., when a user moves to a new place), the robustness of the recommendations cannot be guaranteed. To address the above challenges, we propose DUIG, a novel user interest and geographical influences disentanglement framework for POI recommendations. We first design a personalized disentanglement strategy to divide check-ins through geographical influence. Specifically, the colliding effect of causality is leveraged to the divide cause-specific check-ins, such that user interest and geographical influence can be properly disentangled in user and POI embeddings. Through this mechanism, even if the underlying reasons that affect a user’s preference change, intervention can be conducted upon the causes to make recommendations generalized to the new scenario. In addition, a geographical-aware negative sampling strategy is proposed to utilize hard negatives to regularize the embedding and disentanglement in the latent space, where a larger sampling probability is introduced for negative samples containing more geographic information. Extensive experiments on two real-world POI recommendation datasets demonstrate the superior performance of DUIG.

As a video understanding task, activity parsing aims at encompassing actions into multiple levels of activity components, including activity, sub-activity and atomic action, enabling understanding of complex video scenes within multimedia systems. Existing methods form activity parsing as a multi-task learning problem to predict multi-granular activity labels simultaneously, which ignores modeling the hierarchical structure and the fine-grained transitions of activity components at different levels. In this paper, we propose a Hierarchical Adaptive Reasoning Graph (HARG) to model the hierarchical structure (i.e., object level$\rightarrow$atomic action level$\rightarrow$activity level) dynamically and precisely. To achieve that, an object reasoning graph (ORG) and an atomic action reasoning graph (ARG) are designed to reason fine-grained information transitions between multiple actors at different levels. In addition, an adaptive segmentation module (ASM) is investigated for bridging the gap among different levels, permitting step-by-step reasoning from the object level to the atomic action level. Experimental results show our method outperforms state-of-the-art methods on two activity parsing datasets, achieving hierarchical modeling and fine-grained reasoning for activity understanding. The code is available on GitHub:https://github.com/whuoyj/HARG.

Recently, foundational diffusion models have attracted considerable attention in image compression tasks, whereas their application to video compression remains largely unexplored. In this article, we introduce DiffVC, a diffusion-based perceptual neural video compression framework that effectively integrates foundational diffusion model with the video conditional coding paradigm. This framework uses temporal context from previously decoded frame and the reconstructed latent representation of the current frame to guide the diffusion model in generating high-quality results. To accelerate the iterative inference process of diffusion model, we propose the Temporal Diffusion Information Reuse (TDIR) strategy, which significantly enhances inference efficiency with minimal performance loss by reusing the diffusion information from previous frames. Additionally, to address the challenges posed by distortion differences across various bitrates, we propose the Quantization Parameter-based Prompting (QPP) mechanism, which utilizes quantization parameters as prompts fed into the foundational diffusion model to explicitly modulate intermediate features, thereby enabling a robust variable bitrate diffusion-based neural compression framework. Experimental results demonstrate that our proposed solution delivers excellent performance in both perception metrics and visual quality.

Pretrained Visual Language Models (PVLMs) have demonstrated impressive zero-shot abilities in image captioning when accompanied by prompt prefixes. However, PVLMs using fixed prompt prefixes may suffer from mode collapse when dealing with multiple target domains. Some studies have attempted to mitigate this issue by constructing instruction datasets and injecting domain prior knowledge through hand-crafted prompts. However, this approach inevitably incurs high manual and time costs. In this article, we propose an unsupervised prompt learning method to improve domain Generalization for Image Captioning (GeneIC), which learns a domain-specific prompt vector for the target domain without requiring annotated data. GeneIC aligns visual and language modalities with a pre-trained CLIP model, thus optimizing the domain-specific prompt vector from two aspects: attribute and semantic consistency. Specifically, GeneIC first generates attribute-transferred images with differing attributes while retaining semantic similarity with the original images. Then, GeneIC uses CLIP to measure the similarity between the images and the generated sentences. By exploring the variable and invariant features in the original and attribute-transferred images, attribute consistency constrains the attribute change direction of both images and sentences to learn domain-specific knowledge. The semantic consistency directly measures the similarity between the generated sentences and images to ensure the accuracy and comprehensiveness of the generated sentences. Consequently, GeneIC only optimizes the prompt vectors, which effectively retains the knowledge in the large model and introduces domain-specific knowledge. Experiments show that GeneIC exhibits superior generalization performance compared to state-of-the-art methods on multiple target domain datasets.

Autoregressive Initial Bits (ArIB), a framework that combines subimage autoregression and latent variable models, has shown its advantages in lossless image compression. However, in current methods, the image splitting makes the information of latent variables being uniformly distributed in each subimage, and causes inadequate use of latent variables in addition to posterior collapse. To tackle these issues, we introduce Bit Plane Slicing (BPS), splitting images in the bit plane dimension with the considerations on different importance for latent variables. Thus, BPS provides a more effective representation by arranging subimages with decreasing importance for latent variables. To solve the problem of the increased number of dimensions caused by BPS, we further propose a dimension-tailored autoregressive model that tailors autoregression methods for each dimension based on their characteristics, efficiently capturing the dependencies in plane, space, and color dimensions. As shown in the extensive experimental results, our method demonstrates the superior compression performance with comparable inference speed, when compared to the state-of-the-art normalizing-flow-based methods. The code is at https://github.com/ZZ022/ArIB-BPS.

In this paper, we extend our prior research named DKIC [1] and propose the perceptual-oriented learned image compression method, PO-DKIC, which is shown in Figure 1 . Specifically, DKIC adopts a dynamic kernel-based dynamic residual block group to enhance the transform coding and an asymmetric space-channel context entropy model to facilitate the estimation of Gaussian parameters. Based on DKIC, PO-DKIC introduces PatchGAN and LPIPS loss to enhance visual quality. Furthermore, to maximize the overall perceptual quality under a rate constraint, we formulate this challenge into a constrained programming problem and use the Linear Integer Programming method for resolution. The experiments demonstrate that our proposed method can generate realistic images with richer textures and finer details when compared to state-of-the-art image compression techniques.

With the rapid evolution of advanced image compression, DNN-based learned image compression has emerged as the promising approach for transmitting images in many security-critical applications, such as cloud-based face recognition and autonomous driving, due to its superior performance over traditional compression. There is a pressing need to fully investigate the robustness of a classification system post-processed by learned image compression. To bridge this research gap, we explore the adversarial attack on Learned Image Compression Classification System (LICCS) that targets image classification models that utilize learned image compressors as preprocessing modules. To perform an adversarial attack on an image within the LICCS, the goal is to introduce the adversarial perturbation δ to the source image X that causes the reconstructed adversarial examples gs(Q(ga(X+δ))) to be misclassified by the classification model, which can be formulated as follows:\begin{equation*}\begin{array}{ll} {\mathop {\arg \max }\limits_i f{{\left({{g_s}\left({Q\left({{g_a}\left({{\mathbf{X + \delta }}}\right)}\right)}\right)}\right)}_i} \ne y,}&{{\text{s}}{\text{.t}}{\text{.}}\parallel \delta {\parallel _p} \leq \varepsilon .} \end{array}\tag{1}\end{equation*}

In this paper, we introduce an imperceptible adversarial attack approach designed to effectively degrade the reconstruction quality of LIC, resulting in the reconstructed image being severely disrupted by noise where identifying any object in the reconstructed image is virtually impossible. More specifically, we generate adversarial examples by introducing a Frobenius norm-based loss function to maximize the discrepancy between original images and reconstructed images from adversarial examples in order to corrupt the reconstructed image severely.

Domain shift significantly hinders crowd counting performance in unseen domains. Domain adaptation methods tackle this issue using target domain images but falter when acquiring these images is difficult. Moreover, they demand additional training time for fine-tuning. To address this issue, we propose an Uncertainty-Guided Style Diversity Augmentation (UGSDA) method, enabling the models to be trained solely on the source domain and directly generalized to various target domains. It is achieved by generating sufficiently diverse and realistic samples during the training process. Specifically, our UGSDA method incorporates three tailor-designed components: the Global Styling Elements Extraction (GSEE) module, the Local Uncertainty Perturbations (LUP) module, and the Density Distribution Consistency (DDC) loss. The GSEE extracts global style elements from the feature space of the whole source domain. The LUP aims to obtain uncertainty perturbations from the batch-level input to form style distributions beyond the source domain, which used to generate diversified stylized samples together with global style elements. To regulate the extent of perturbations, the DDC loss imposes constraints between the source samples and the stylized samples, ensuring the stylized samples maintain a higher degree of realism and reliability. Comprehensive experiments validate the superiority of our approach, demonstrating its strong generalization capabilities across various datasets and models. Code is available at https://github.com/gcding/UGSDA-pytorch.

Joint demoisaicking and denoising (JDD) serves as an initial step of image signal processing (ISP), whose performance significantly influences the subsequent operations like image processing and compression. Although deep learning-based methods have demonstrated remarkable performance in JDD, they suffer from heavy computational cost and memory occupation, hindering their deployment of resource-constrained devices. To reach a compromise between performance and complexity, we propose a novel knowledge distillation method named Mutual Guidance Distillation (MGD). It aims at promoting the accuracy of lightweight JDD networks (student) by imitating the mutual guidance procedure between color components from a cumbersome network (teacher). The procedure is achieved by computing spatial correlation between representations of red, blue and green components from different layers. The higher the correlation, the greater the influences of each component representation on the restoration of other components. Then the single-branch student network is trained to mimic the correlation of a multi-branch teacher network. Since the multi-branch teacher derives advantages from component mutual guidance and achieves outstanding performance, student networks can be enhanced under the instruction of the teacher. Experimental results on various joint demosaicking and denoising datasets demonstrate that MGD outperforms several state-of-the-art distillation methods quantitatively and qualitatively, and visual results illustrate that MGD effectively mitigates color artifacts, even on hard cases from MIT moiré dataset.

As an emerging video coding standard, H.266NVC is widely recognized for efficiently reducing bit rate and im-proving compression ratio. However, adopting a block-based hybrid coding framework, VVC still encounters the challenge of compression artifacts, which cause a serious impact on both subjective and objective quality. Recently, the emergence of deep learning techniques has boosted the development of neural network-based in-loop filter methods, most of which adopt convolutional neural networks (CNNs) as their backbone. However, the local receptive field of CNN limits its ability to extract global features, which leads to a performance bottleneck in the CNN-base in-loop filter. In contrast, this paper offers a swin transformer-based in-loop filter for VVC, which has a more flexible and global receptive field than CNN. Specifically, we first introduce and optimize the swin transformer-based network for the video compression task. In addition, to ensure the rate-distortion performance, a coding tree unit-level flag is designed to involve the network in the RDO decision-making process at the block level. The experimental results show that compared with VTM-ll.0, the proposed swin transformer-based in-loop filter method can achieve an average of 6.77%, 14.62%, 14.97% and 6.88%, 17.24%, 17.50% Bjontegaard Delta (BD)-Bitrate savings under All-intra (AI) configurations for Y, U and V components when using PSNR and MS-SSIM as the quality metric, respectively.

In this paper, we propose a multi-agent reinforcement learning based bit allocation method towards quality stability for gaming video coding in Versatile Video Coding (VVC). The bits allocated to regions-of-interests (ROI) are critical to obtain a subjectively optimal visual quality but also constrained subject to the frame-level bit budgets. A multi-objective partially observable stochastic game is formulated by combining the frame-level and ROI-level bit allocation process, which optimizes both the quality and fluctuation simultaneously. The proposed method is implemented in VVC and verified with gaming video. A multi-agent reinforcement learning method is utilized for training the agents and obtaining reasonable bit allocation actions. In comparison to the reference methods, the proposed method achieves a more consistent quality at both the frame-level and ROI-level, while improving the quality of ROI.

Recent advancements in learned image compression methods have demonstrated superior rate-distortion performance and remarkable potential compared to traditional compression techniques. However, the core operation of quantization, inherent to lossy image compression, introduces errors that can degrade the quality of the reconstructed image. To address this challenge, we propose a novel Quantization Error Compensator (QEC), which leverages spatial context within latent representations and hyperprior information to effectively mitigate the impact of quantization error. Moreover, we propose a tailored quantization error optimization training strategy to further improve rate-distortion performance. Notably, QEC serves as a lightweight, plug-and-play module, offering high flexibility and seamless integration into various learned image compression methods. Extensive experimental results consistently demonstrate significant coding efficiency improvements achievable by incorporating the proposed QEC into state-of-the-art methods, with a slight increase in runtime.

In video coding, inter-prediction leverages neigh-boring frames to reduce temporal redundancy. The quality of these reference frames is essential for effective inter-prediction. Although many neural network-based methods have been proposed to improve the quality of reference frames, there is still room for the performance and efficiency trade-off. In this paper, we propose an interpolation diverse branch block (InterDBB) suitable for lightweight frame interpolation networks, which optimizes deep reference frame interpolation networks to improve performance without sacrificing speed and increasing complexity. Specifically, we propose a multi-branch structural reparameterization block without batch normalization. This straightforward yet effective modification ensures training stability and performance improvement. Moreover, we propose a parameterized motion estimation strategy based on different input resolution, to achieve a better trade-off between performance and computational complexity. Experimental results demonstrate that our method achieves -2.01%/-2.87%/-2.44% coding efficiency improvements for Y/U/V components under random access (RA) configuration compared to VTM-11.0_NNVC-5.0.

Remote sensing image (RSI) super-resolution (SR) demands lightweight and efficient methods due to required rapid response in practical applications. Integrating RSIs with different resolutions for diverse applications also requires arbitrary-scale SR, making fix-scaled SR scale inflexible. Therefore, a lightweight SR algorithm capable of arbitrary-scale is necessary for RSIs. To address the above issue, a super-scale feature-based lightweight arbitrary-scale (SFLA) SR network is proposed in this paper. The network consists of two modules: 1) A super-scale feature extraction (SSFE) module that extracts features at both the initial low-resolution (LR) and an integer super-scale resolution, 2) A self-attention implicit function reconstruction (SIFR) module that utilizes multi-layer perceptron (MLP) network and self-attention mechanism for pixel-wise feature mapping to achieve superior SR results. Comparative experiments and ablation results demonstrate that the proposed SFLA algorithm effectively strikes a good balance between performance and complexity.

In this work, we propose a Unified framework of Sequential Search and Recommendation (UnifiedSSR) for joint learning of user behavior history in both search and recommendation scenarios. Specifically, we consider user-interacted products in the recommendation scenario, as well as user-interacted products and user-issued queries in the search scenario as three distinct types of user behaviors. We propose a dual-branch network to encode the pair of interacted product history and issued query history in the search scenario in parallel. This allows for cross-scenario modeling by deactivating the query branch for the recommendation scenario. Through the parameter sharing between dual branches, as well as between product branches in two scenarios, we incorporate cross-view and cross-scenario associations of user behaviors, providing a comprehensive understanding of user behavior patterns. To further enhance user behavior modeling by capturing the underlying dynamic intent, an Intent-oriented Session Modeling module is designed for inferring intent-oriented semantic sessions from the contextual information in behavior sequences. In particular, we consider self-supervised learning signals from two perspectives for intent-oriented semantic session locating, which encourage session discrimination within each behavior sequence and session alignment between dual behavior sequences. Extensive experiments on three public datasets demonstrate that UnifiedSSR consistently outperforms state-of-the-art methods for both search and recommendation.

Recent works on Video Anomaly Detection (VAD) have made advancements in the unsupervised setting, known as Unsupervised VAD (UVAD), which brings it closer to practical applications. Unlike the classic VAD task that requires a clean training set with only normal events, UVAD aims to identify abnormal frames without any labeled normal/abnormal training data . Many existing UVAD methods employ handcrafted surrogate tasks, such as frame reconstruction, to address this challenge. However, we argue that these surrogate tasks are sub-optimal solutions, inconsistent with the essence of anomaly detection. In this paper, we propose a novel approach for UVAD that directly detects anomalies based on similarities between events in videos. Our method generates representations for events while simultaneously capturing prototypical normality patterns, and detects anomalies based on whether an event’s representation matches the captured patterns. The proposed model comprises a memory module to capture normality patterns, and a representation learning network to obtain representations matching the memory module for normal events. A pseudo-label generation module as well as an anomalous event generation module for negative learning are further designed to assist the model to work under the strictly unsupervised setting. Experimental results demonstrate that the proposed method outperforms existing UVAD methods and achieves competitive performance compared with classic VAD methods.

Learning based single image super-resolution (SISR) for real-world images has been an active research topic yet a challenging task, due to the lack of paired low-resolution (LR) and high-resolution (HR) training images. Most of the existing unsupervised real-world SISR methods adopt a two-stage training strategy by synthesizing realistic LR images from their HR counterparts first, then training the super-resolution (SR) models in a supervised manner. However, the training of image degradation and SR models in this strategy are separate, ignoring the inherent mutual dependency between downscaling and its inverse upscaling process. Additionally, the ill-posed nature of image degradation is not fully considered. In this paper, we propose an image downscaling and SR model dubbed as SDFlow, which simultaneously learns a bidirectional many-to-many mapping between real-world LR and HR images unsupervisedly. The main idea of SDFlow is to decouple image content and degradation information in the latent space, where content information distribution of LR and HR images is matched in a common latent space. Degradation information of the LR images and the high-frequency information of the HR images are fitted to an easy-to-sample conditional distribution. Experimental results on real-world image SR datasets indicate that SDFlow can generate diverse realistic LR and SR images both quantitatively and qualitatively.

Recently, Video Coding for Machines (VCM) has gained more and more attention due to its efforts in machine vision tasks. As a crucial track in VCM, feature compression preserves and transmits critical feature information for machine vision. Most existing studies employ dimensionality reduction to the raw multi-scale feature before compression. However, feature sparsity is left insufficiently considered in removing redundancy in compressed features. In this letter, we propose a novel framework for image feature compression for machines, where the multi-scale feature is hierarchically transformed into a sparse representation for compression. The multi-scale feature is first fused by convolutional neural networks and the attention mechanism. To introduce sparsity into the fused feature, informative channels are identified by a channel-wise binary mask where activated elements are sampled from the importance distribution of channels learned from feature content. Then, the fused feature is masked to generate a sparse representation for compression. Experiments conducted on two machine tasks show significant improvements in our model over state-of-the-art methods.

In video coding, inter prediction aims to reduce temporal redundancy by using previously encoded frames as references. The quality of reference frames is crucial to the performance of inter prediction. This paper presents a deep reference frame generation method to optimize the inter prediction in Versatile Video Coding (VVC). Specifically, reconstructed frames are sent to a well-designed frame generation network to synthesize a picture similar to the current encoding frame. The synthesized picture serves as an additional reference frame inserted into the reference picture list (RPL) to provide a more reliable reference for subsequent motion estimation (ME) and motion compensation (MC). The frame generation network employs optical flow to predict motion precisely. Moreover, an optical flow reorganization strategy is proposed to enable bi-directional and uni-directional predictions with only a single network architecture. To reasonably apply our method to VVC, we further introduce a normative modification of the temporal motion vector prediction (TMVP). Integrated into the VVC reference software VTM-15.0, the deep reference frame generation method achieves coding efficiency improvements of 5.22%, 3.61%, and 3.83% for the Y component under random access (RA), low delay B (LDB), and low delay P (LDP) configurations, respectively. The proposed method has been discussed in Joint Video Exploration Team (JVET) meeting and is currently part of Exploration Experiments (EE) for further study.

Low-light image enhancement (LLIE) aims to improve the illuminance of images due to insufficient light exposure. Recently, various lightweight learning-based LLIE methods have been proposed to handle the challenges of unfavorable prevailing low contrast, low brightness, etc. In this paper, we have streamlined the architecture of the network to the utmost degree. By utilizing the effective structural re-parameterization technique, a single convolutional layer model (SCLM) is proposed that provides global low-light enhancement as the coarsely enhanced results. In addition, we introduce a local adaptation module that learns a set of shared parameters to accomplish local illumination correction to address the issue of varied exposure levels in different image regions. Experimental results demonstrate that the proposed method performs favorably against the state-of-the-art LLIE methods in both objective metrics and subjective visual effects. Additionally, our method has fewer parameters and lower inference complexity compared to other learning-based schemes. Code will be made publicly available at the URL https://gitee.com/zhanghahaxixi/SCLM.

Object counting and localization for remote sensing images are effective means to solve large-scale object analysis problems. Nowadays, most counting methods obtain the number of objects by employing convolutional neural network (CNN) to regress a density map of objects. Even if these leading methods have achieved impressive performances, they simply focus on estimating the number of single-class objects, without providing location information and cannot support multiclass objects. To tackle these problems, a point-based network named Dense Object Prediction Network (DOPNet) is proposed for multiclass object counting and localization for remote sensing images. DOPNet differs from the conventional approach of predicting multiple density maps by incorporating category attributes into the predicted objects, enabling the accurate counting and localization of multiclass objects. Specifically, DOPNet adopts a multiscale architecture (MS) to provide dense predictions of object proposals. A scale adaptive feature enhancement module (SAFEM) is designed to predict scales of objects for the suppression of duplicate proposals. Given only point level annotations for training, a pseudo-box generation algorithm is designed to find the most suitable pseudo-box of each annotated object for the supervision of scale learning. Comprehensive experiments prove that DOPNet can achieve preferable performance on challenging benchmarks of counting while providing object locations. Code and pre-trained models are available athttps://github.com/Ceoilmp/DOPNet.

Due to the huge data volume of high-resolution remote sensing imagery (RSI) and limited transmission bandwidth, RSIs are typically compressed for efficient transmission and storage. However, most of the existing compression algorithms are developed based on optimizing for the human perceptual that are not suitable for remote sensing image applications where RSIs are usually used for machine interpretation tasks, such as semantic segmentation for ground–object recognition. In this article, we propose an image coding for machines (ICMs) paradigm based on contrastive learning in a fully supervised manner to boost semantic segmentation of compressed RSIs. Specifically, we build an end-to-end compression framework to make full use of the global semantic information by clustering intracategory projected embeddings and spacing intercategory embeddings apart, to compensate for the loss of feature discriminability during the compression process and reconstruct the decision boundaries between different categories. Compared to the state-of-the-art image compression methods, our proposed method significantly improves the performance of semantic segmentation on the remote sensing labeling benchmark datasets.

JPEG compression adopts the quantization of Discrete Cosine Transform (DCT) coefficients for effective bit-rate reduction, whilst the quantization could lead to a significant loss of important image details. Recovering compressed JPEG images in the frequency domain has recently garnered increasing interest, complementing the multitude of restoration techniques established in the pixel domain. However, existing DCT domain methods typically suffer from limited effectiveness in handling a wide range of compression quality factors or fall short in recovering sparse quantized coefficients and the components across different colorspaces. To address these challenges, we propose a DCT domain spatial-frequential Transformer, namely DCTransformer, for JPEG quantized coefficient recovery. Specifically, a dual-branch architecture is designed to capture both spatial and frequential correlations within the collocated DCT coefficients. Moreover, we incorporate the operation of quantization matrix embedding, which effectively allows our single model to handle a wide range of quality factors, and a luminance-chrominance alignment head that produces a unified feature map to align different-sized luminance and chrominance components. Our proposed DCTransformer outperforms the current state-of-the-art JPEG artifact removal techniques, as demonstrated by our extensive experiments.

Learned video compression methods have gained various interests in the video coding community. Most existing algorithms focus on exploring short-range temporal information and developing strong motion compensation. Still, the ignorance of long-range temporal information utilization constrains the potential of compression. In this paper, we are dedicated to exploiting both long- and short-range temporal information to enhance video compression performance. Specifically, for long-range temporal information exploration, we propose a temporal prior that can be continuously supplemented and updated during compression within the group of pictures (GOP). With the updating scheme, the temporal prior can provide richer mutual information between the overall prior and the current frame for the entropy model, thus facilitating Gaussian parameter prediction. As for the short-range temporal information, we propose a progressive guided motion compensation to achieve robust and accurate compensation. In particular, we design a hierarchical structure to build multi-scale compensation, and by employing optical flow guidance, we generate pixel offsets as motion information at each scale. Additionally, the compensation results at each scale will guide the next scale's compensation, forming a flow-to-kernel and scale-by-scale stable guiding strategy. Extensive experimental results demonstrate that our method can obtain advanced rate-distortion performance compared to the state-of-the-art learned video compression approaches and the latest standard reference software in terms of PSNR and MS-SSIM. The codes are publicly available on: https://github.com/Huairui/LSTVC.

Object-oriented micro-video background music recommendation is a complicated task where the matching degree between videos and background music is a major issue. However, music selections in user-generated content (UGC) are prone to selection bias caused by historical preferences of uploaders. Since historical preferences are not fully reliable and may reflect obsolete behaviors, over-reliance on them should be avoided as knowledge and interests dynamically evolve. In this article, we propose a Deconfounded Cross-Modal matching model to mitigate such bias. Specifically, uploaders’ personal preferences of music genres are identified as confounders that spuriously correlate music embeddings and background music selections, causing the learned system to over-recommend music from majority groups. To resolve such confounders, backdoor adjustment is utilized to deconfound the spurious correlation between music embeddings and prediction scores. We further utilize Monte Carlo estimator with batch-level average as the approximations to avoid integrating the entire confounder space calculated by the adjustment. Furthermore, we design a teacher–student network to utilize the matching of music videos, which is professionally generated content (PGC) with specialized matching, to better recommend content-matching background music. The PGC data are modeled by a teacher network to guide the matching of uploader-selected UGC data of student network by Kullback–Leibler–based knowledge transfer. Extensive experiments on the TT-150k-genre dataset demonstrate the effectiveness of the proposed method. The code is publicly available on https://github.com/jing-1/DecCM

Traditional recommender systems aim to estimate a user’s rating to an item based on observed ratings from the population. As with all observational studies, hidden confounders, which are factors that affect both item exposures and user ratings, lead to a systematic bias in the estimation. Consequently, causal inference has been introduced in recommendations to address the influence of unobserved confounders. Observing that confounders in recommendations are usually shared among items and are therefore multi-cause confounders, we model the recommendation as a multi-cause multi-outcome (MCMO) inference problem. Specifically, to remedy the confounding bias, we estimate user-specific latent variables that render the item exposures independent Bernoulli trials. The generative distribution is parameterized by a DNN with factorized logistic likelihood and the intractable posteriors are estimated by variational inference. Controlling these factors as substitute confounders, under mild assumptions, can eliminate the bias incurred by multi-cause confounders. Furthermore, we show that MCMO modeling may lead to high variance due to scarce observations associated with the high-dimensional treatment space. Therefore, we theoretically demonstrate that controlling user features as pre-treatment variables can substantially improve sample efficiency and alleviate overfitting. Empirical studies on both simulated and real-world datasets demonstrate that the proposed deep causal recommender shows more robustness to unobserved confounders than state-of-the-art causal recommenders. Codes and datasets are released at https://github.com/yaochenzhu/Deep-Deconf.

While saliency detection for 3D meshes has been extensively studied in the past decades, only a little work considers color information, and most of existing 3D mesh saliency databases are collected using meshes without color information. The lack of publicly available 3D colored mesh saliency database hinders the research progress in 3D colored mesh saliency detection. In this article, we established a novel 3D colored mesh saliency database (3DCMS) based on an eye-tracking experiment and investigated subjects' visual attention behavior towards 3D colored meshes. Based on the investigations, a novel 3D colored mesh saliency detection framework is proposed which takes both color and geometric features into consideration. To evaluate the performance of the proposed algorithm, we compare it with several relevant methods and apply it to 3D mesh simplification task. The quantitative and qualitative evaluation results demonstrate the superior performance of the proposed framework. The proposed 3DCMS database will be made publicly available.1

Automatic video generation is a challenging research topic, attracting interests from different perspectives, including Image-to-Video generation (I2V), Video-to-Video generation (V2V), and Text-to-Video generation (T2V). To pursue more controllable and fine-grained video generation, a novel video generation task, named Text-Image-to-Video generation (TI2V), and a corresponding baseline solution, named Motion Anchor-based video Generator (MAGE), were proposed. However, two other factors, namely clean datasets and reliable evaluation metrics, also play important roles in the success of the TI2V task. In this article, we present a complete benchmark for the TI2V task which includes synthetic video-text paired datasets, a baseline method, and two evaluation metrics. More specifically: (1) Two versions of synthetic datasets are built based on CATER containing rich combinations of objects and actions, as well as the resulting changes of brightness and shadow. We also provide both explicit and ambiguous text descriptions to support deterministic and diverse video generation, respectively. (2) A refined version of MAGE, dubbed MAGE+, is proposed with an innovative motion anchor structure to store appearance-motion aligned representation, which can be further injected with explicit condition and implicit randomness to model the uncertainty in data distribution. (3) To evaluate the quality of generated video especially given ambiguous description, we introduce action precision and referring expression precision to assess the quality of motion based on captioning-and-matching method. Experiments conducted on proposed datasets, as well as relevant datasets, verify the effectiveness of our baseline and show appealing potentials of TI2V task.

Learned video compression has recently emerged as an essential research topic in developing advanced video compression technologies, where motion compensation is considered one of the most challenging issues. In this article, we propose a learned video compression framework via heterogeneous deformable compensation strategy (HDCVC) to tackle the problems of unstable compression performance caused by single-size deformable kernels in downsampled feature domain. More specifically, instead of utilizing optical flow warping or single-size-kernel deformable alignment, the proposed algorithm extracts features from the two adjacent frames to estimate content-adaptive heterogeneous deformable (HetDeform) kernel offsets. Then we align the features extracted from the reference frames with the HetDeform convolution to accomplish motion compensation. Moreover, we design a Spatial-Neighborhood-Conditioned Divisive Normalization (SNCDN) to reduce spatial statistic dependencies and achieve more effective data Gaussianization combined with the Generalized Divisive Normalization. Furthermore, we propose a multi-frame enhanced reconstruction module for exploiting context and temporal information for final quality enhancement. Experimental results indicate that HDCVC achieves superior performance than the recent state-of-the-art learned video compression approaches.

Multi-modal data presents a promising opportunity for improving multimedia recommendation models, but it also introduces task-irrelevant noise that can reduce model robustness. In this paper, we propose a robust multi-modal recommendation approach that accounts for different levels of task-irrelevant noise across modalities. We explicitly consider the uncertainty associated with each modality and perform stochastic sampling-based fusion according to the precision of different modalsities, which serves as a measure of uncertainty. The influence of noisy modalities with high uncertainty is removed, filtering out task-irrelevant noise, and therefore a noise-robust multi-modal recommendation is achieved. Moreover, the stochastic sampling strategy intrinsically considers and simulates scenarios with absent modalities during multi-modal fusion. Consequently, it incorporates additional randomness into the training process, which enables the model to handle the problem of modality missing. Furthermore, the proposed fusion approach integrates the noise robustness of the Product-of-Experts (PoE) framework when modeling with Gaussian distributions, along with the flexibility of the Mixture-of-Experts (MoE) technique to represent diverse distributions of latent variables. This integration allows the proposed approach to achieve noise-robust modeling with non-Gaussian variables. Specifically, we derive a solvable evidence lower bound for the proposed variational mixture of stochastic experts (VMoSE) auto-encoder, where both Gaussian and Student-T distributions are used to model the latent variables. Constraints are added to match the similarities between the ID embeddings and the multi-modal joint embeddings by utilizing an Expectation maximization (EM)-style algorithm for better model optimization. Extensive experiments demonstrate the effectiveness of the proposed method in multi-modal fusion and the robustness to modality noise and modality missing.

Next point-of-interest (POI) recommendation is a critical task in location-based social networks, yet remains challenging due to a high degree of variation and personalization exhibited in user movements. In this work, we explore the latent hierarchical structure composed of multi-granularity short-term structural patterns in user check-in sequences. We propose a Spatio-Temporal context AggRegated Hierarchical Transformer (STAR-HiT) for next POI recommendation, which employs stacked hierarchical encoders to recursively encode the spatio-temporal context and explicitly locate subsequences of different granularities. More specifically, in each encoder, the global attention layer captures the spatio-temporal context of the sequence, while the local attention layer performed within each subsequence enhances subsequence modeling using the local context. The sequence partition layer infers positions and lengths of subsequences from the global context adaptively, such that semantics in subsequences can be well preserved. Finally, the subsequence aggregation layer fuses representations within each subsequence to form the corresponding subsequence representation, thereby generating a new sequence of higher-level granularity. The stacking of hierarchical encoders captures the latent hierarchical structure of the check-in sequence, which is used to predict the next visiting POI. Extensive experiments on three public datasets demonstrate that the proposed model achieves superior performance while providing explanations for recommendations.

In recent years, learned video compression methods have improved substantially. However, most existing algorithms focus on exploring short-term temporal information, thus constraining the compression capability. In this paper, to further boost video compression performance, we exploit both long-and short-range temporal information and consider bidirectional temporal information. For long- and short-range temporal information exploration, we adopt temporal prior and progressive guided motion compensation. Specifically, with the continuously updating strategy, the temporal prior can provide rich mutual information between the overall prior and the current frame, facilitating Gaussian parameter prediction in the entropy model. Besides, the progressive guided motion compensation utilizes flow-to-kernel and scale-by-scale stable guiding strategy, thus achieving robust and effective inter coding. Furthermore, existing low-latency-oriented methods often suffer from strong error propagation, so we extend the framework with a bidirectional prediction scheme and propose the bidirectional temporal prior. Extensive experimental results demonstrate that our method can obtain competitive performance compared to the state-of-the-art learned video compression approaches and the standard reference software HM-16.22.

In this paper, we propose a deep reference frame generation method that aims to enhance bi-direction inter prediction under random access configuration in the latest video coding standard, Versatile Video Coding. Specifically, a pair of neighboring reconstructed frames are selected from decoded picture buffer and put into an optical-flow-based interpolation network to synthesize a new frame, similar to the current to-be-coded frame. Subsequently, this synthesized frame is incorporated into two-sided picture reference lists as additional reference frames. The proposed method is employed in both the encoding and decoding processes to eliminate bitstream signaling for supplementary information. The Small Ad-hoc Deep-Learning Library is utilized for implementing the proposed method. Experimental results demonstrate 3.67%/7.34%/6.51% coding efficiency improvements for Y/U/V components under the random access configuration when compared to the Versatile Video Coding NNVC reference software VTM-11_NNVC-5.0.

Deep neural network (DNN)-based methods have demonstrated enormous potential for Versatile Video Coding (VVC) inter prediction enhancement. However, due to their typically high computational complexity, implementing them in practical applications can be challenging. In this paper, we propose a lightweight deep reference frame interpolation network to enhance bi-prediction with low complexity. Specifically, given a pair of bi-directional reconstructed frames, first, we down-sample input frames to reduce the complexity before feeding them into the optical flow estimation network. Then the optical flows are utilized to warp extracted features at three different levels. The warped features are fused to generate the output intermediate frame. The additional reference frame is inserted into the reference picture lists to provide an additional reliable reference candidate. In contrast to previous efforts, the proposed method aims at achieving the trade-off between performance and complexity while maintaining a complexity of about 64 kMACs/pix. Experimental results demonstrate that our method achieves 1.82%/2.43%/2.02% coding efficiency improvements for Y/U/V components under random access (RA) configuration compared to the latest NNVC standard software VTM-11.0_NNVC-5.0.

Recently, quality assessment for user-generated content (UGC) videos has become a challenging task due to the absence of reference videos and the presence of complex distortions. Prior methods has highlighted the effectiveness of semantic features for quality assessment. However, these models are incapable for real-time prediction and efficient computation in practical applications. In this paper, we design a lightweight no-reference video quality assessment model leveraging pretrained lightweight network for semantic understanding and utilizing a low-level CNN for distortion features. The temporal features and spatial features are extracted respectively in the semantic and low-level scales, and then they are multiplied and integrated to obtain the video-level score. Experiments on UGC video quality databases show that our proposed model achieves comparable accuracy to state-of-the-art benchmarks while providing real-time performance on GPU.

Exposure errors in images, including both underexposure and overexposure, significantly diminish images’ contrast and visual appeal. Existing deep learning-based exposure correction methods either require large networks or longer processing time for inference and are thus not applicable for embedded devices and real-time applications. To address these issues, a lightweight network is proposed in this paper to correct exposure errors with limited memory occupation and inference steps. It adopts the Laplacian pyramid to incrementally recover the color and details of the image through a layer-by-layer procedure. A structural re-parameterization structure is designed to both reduce model size for inference speed up and improve performance with a multi-branch learning structure. Extensive experiments demonstrate that our method achieves a better performance-efficiency trade-off than other exposure correction methods.

Learned image compression (LIC) has shown significant potential and better rate-distortion performance than traditional techniques. However, existing CNN-based approaches or window-based self-attention methods can only capture spatial information within fixed ranges. To tackle this limitation, we propose a novel method with dynamic spatial aggregation for transform coding. Our approach introduces enhanced adaptive aggregation, which generates kernel offsets to capture relevant information within content-dependent ranges, improving the transform process. Furthermore, we define a generalized coarse-to-fine entropy model that considers global context, channel-wise information, and spatial context in a coarse-to-fine manner. Additionally, our method takes into full consideration the model’s efficiency and complexity. By introducing the asymmetric entropy model structure and efficient heterogeneous convolution, our approach maintains lower coding complexity and higher decoding speed while ensuring performance. Experimental results demonstrate a substantial improvement in rate-distortion performance achieved by our method when compared to traditional compression methods like VTM and BPG, as well as some LIC methods.

In this letter, we propose a learned scale-arbitrary image downscaling method to downscale high-resolution (HR) images to a target low-resolution (LR) where it could be well upscaled by traditional simple upscaling methods. Specifically, the scale information is first fused into the feature extraction process of the input HR image by employing our proposed scale-adaptive feature enhancement module (FEM). Then the proposed scale-arbitrary feature resampler (SAFR) adaptively generates sampling weights which are applied to the resampled candidate feature maps to produce the downscaled LR image. Experimental results demonstrate that when compared to the existing learned image downscaling method for non-learnable upscaling, the reconstructed HR images upscaled by our proposed method receive better quality..

In this paper, we consider the task of space-time video super-resolution (ST-VSR), which can increase the spatial resolution and frame rate for a given video simultaneously. Despite the remarkable progress of recent methods, most of them still suffer from high computational costs and inefficient long-range information usage. To alleviate these problems, we propose a Bidirectional Recurrence Network (BRN) with the optical-flow-reuse strategy to better use temporal knowledge from long-range neighboring frames for high-efficiency reconstruction. Specifically, an efficient and memory-saving multi-frame motion utilization strategy is proposed by reusing the intermediate flow of adjacent frames, which considerably reduces the computation burden of frame alignment compared with traditional LSTM-based designs. In addition, the proposed hidden state in BRN is updated by the reused optical flow and refined by the Feature Refinement Module (FRM) for further optimization. Moreover, by utilizing intermediate flow estimation, the proposed method can inference non-linear motion and restore details better. Extensive experiments demonstrate that our optical-flow-reuse-based bidirectional recurrent network (OFR-BRN) is superior to state-of-the-art methods in accuracy and efficiency. Codes are available on URL:https://github.com/hahazh/OFR-BRN

Hybrid recommendations have recently attracted a lot of attention where user features are utilized as auxiliary information to address the sparsity problem caused by insufficient user-item interactions. However, extracted user features generally contain rich multimodal information, and most of them are irrelevant to the recommendation purpose. In this article, we propose a variational bandwidth auto-encoder (VBAE) for recommendations, aiming to address the sparsity and noise problems simultaneously. VBAE first encodes user collaborative and feature information into Gaussian latent variables via deep neural networks to capture non-linear user similarities. Moreover, by considering the fusion of collaborative and feature variables as a virtual communication channel from an information-theoretic perspective, we introduce a user-dependent channel to dynamically control the information allowed to be accessed from the feature embeddings. A quantum-inspired uncertainty measurement of the hidden rating embeddings is proposed accordingly to infer the channel bandwidth by disentangling the uncertainty information in the ratings from the semantic information. Through this mechanism, VBAE incorporates adequate auxiliary information from user features if collaborative information is insufficient, while avoiding excessive reliance on noisy user features to improve its generalization ability to new users. Extensive experiments conducted on three datasets demonstrate the effectiveness of the proposed method.

In this paper, we propose a Hierarchical Multimodal Variational Encoder-Decoder (HMMVED) to predict the popularity of micro-videos by comprehensively leveraging the user information and the micro-video content in a hierarchical fashion. In particular, the multimodal variational encoder-decoder framework encodes the input modalities to a lower dimensional stochastic embedding, from which the popularity of micro-videos can be decoded. Considering the leading role of the user’s social influence in social media for information dissemination, a user encoder-decoder is designed to learn the prior Gaussian embedding of the micro-video from the user information, which is informative about the coarse-grained popularity. In order to incorporate the fluctuation around the coarse-grained popularity caused by the diverse multimodal content, in the micro-video encoder-decoder, the refined posterior distribution of the micro-video embedding is encoded from the content features while encouraged to be close to the learned prior distribution. The fine-grained popularity of each micro-video is decoded from the posterior embedding of the micro-video. Based on the multimodal extension of variational information bottleneck theory, we show that the learned latent embeddings of micro-videos are maximally expressive about the popularity whilst maximally compressing the information from input modalities. Extensive experiments conducted on two real-world datasets demonstrate the effectiveness of the proposed method. Codes and datasets are available at:https://github.com/JennyXieJiayi/HMMVED.

In this paper, we propose a cross-modal variational auto-encoder (CMVAE) for content-based micro-video background music recommendation. CMVAE is a hierarchical Bayesian generative model that matches relevant background music to a micro-video by projecting these two multimodal inputs into a shared low-dimensional latent space, where the alignment of two corresponding embeddings of a matched video-music pair is achieved by cross-generation. Moreover, the multimodal information is fused by the product-of-experts (PoE) principle, where the semantic information in visual and textual modalities of the micro-video are weighted according to their variance estimations such that the modality with a lower noise level is given more weights. Therefore, the micro-video latent variables contain less irrelevant information that results in a more robust model generalization. Furthermore, we establish a large-scale content-based micro-video background music recommendation dataset, TT-150k, composed of extracted features from approximately 3,000 different background music clips associated to 150,000 micro-videos from different users. Extensive experiments on the established TT-150k dataset demonstrate the effectiveness of the proposed method. A qualitative assessment of CMVAE by visualizing some recommendation results is also included.

Recommending appropriate tags to items can facilitate content organization, retrieval, consumption, and other applications, where hybrid tag recommender systems have been utilized to integrate collaborative information and content information for better recommendations. In this article, we propose a multi-auxiliary augmented collaborative variational auto-encoder (MA-CVAE) for tag recommendation, which couples item collaborative information and item multi-auxiliary information, i.e., content and social graph, by defining a generative process. Specifically, the model learns deep latent embeddings from different item auxiliary information using variational auto-encoders (VAE), which could form a generative distribution over each auxiliary information by introducing a latent variable parameterized by deep neural network. Moreover, to recommend tags for new items, item multi-auxiliary latent embeddings are utilized as a surrogate through the item decoder for predicting recommendation probabilities of each tag, where reconstruction losses are added in the training phase to constrain the generation for feedback predictions via different auxiliary embeddings. In addition, an inductive variational graph auto-encoder is designed to infer latent embeddings of new items in the test phase, such that item social information could be exploited for new items. Extensive experiments on MovieLens and citeulike datasets demonstrate the effectiveness of our method.

Generating controllable videos conforming to user intentions is an appealing yet challenging topic in computer vision. To enable maneuverable control in line with user intentions, a novel video generation task, named Text-Image-to-Video generation (TI2V), is proposed. With both controllable appearance and motion, TI2V aims at generating videos from a static image and a text description. The key challenges of TI2V task lie both in aligning appearance and motion from different modalities, and in handling uncertainty in text descriptions. To address these challenges, we propose a Motion Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA) structure to store appearance-motion aligned representation. To model the uncertainty and increase the diversity, it further allows the injection of explicit condition and implicit randomness. Through three-dimensional axial transformers, MA is interacted with given image to generate next frames recursively with satisfying controllability and diversity. Accompanying the new task, we build two new video-text paired datasets based on MNIST and CATER for evaluation. Experiments conducted on these datasets verify the effectiveness of MAGE and show appealing potentials of TI2V task. Datasets are available at https://github.com/Youncy-Hu/MAGE.

Action recognition is a challenging task since the attributes of objects as well as their relationships change constantly in the video. Existing methods mainly use object-level graphs or scene graphs to represent the dynamics of objects and relationships, but ignore modeling the fine-grained relationship transitions directly. In this paper, we propose an Object-Relation Reasoning Graph (OR2G) for reasoning about action in videos. By combining an object-level graph (OG) and a relation-level graph (RG), the proposed OR2G catches the attribute transitions of objects and reasons about the relationship transitions between objects simultaneously. In addition, a graph aggregating module (GAM) is investigated by applying the multi-head edge-to-node message passing operation. GAM feeds back the information from the relation node to the object node and enhances the coupling between the object-level graph and the relation-level graph. Experiments in video action recognition demonstrate the effectiveness of our approach when compared with the state-of-the-art methods.

In this paper, we briefly summarize the first challenge on moving object detection and tracking in satellite videos (SatVideoDT). This challenge has three tracks related to satellite video analysis, including moving object detection (Track 1), single object tracking (Track 2), and multiple-object tracking (Track 3). 123, 89, and 70 participants successfully registered, while 37, 42, and 29 teams submitted their final results on the test datasets for Tracks 1-3, respectively. The top-performing methods and their results in each track are described with details. This challenge establishes a new benchmark for satellite video analysis.

In video coding, bi-directional inter prediction aims to remove temporal redundancy via two-sided previously coded frames as reference. High-quality reference frames are essential to reduce the prediction residuals and improve coding efficiency performance. In this paper, we propose a deep learning-based reference frame interpolation method to enhance bi-prediction by introducing a synthetic frame to reference picture lists. Specifically, reconstructed frames are fed into a well-designed interpolation and filtering network to synthesize a picture which can be regarded as an additional reference of to-be-coded frame. Then, the picture is inserted at the appropriate place in reference picture lists to provide a more reliable reference for subsequent motion estimation and motion compensation. Experimental results show that the proposed method achieves 2.03%/6.96%/6.40% coding efficiency improvements for Y/U/V components under random access configuration, when compared with the VVC reference software VTM-15.0.

A reliable video quality assessment (VQA) algorithm is essential for evaluating and optimizing video processing pipelines. In this paper, we propose a quality aggregation network (QAN) for full-reference VQA, which models the characteristics of human visual perception of video quality in both spatial and temporal domain. The proposed QAN is composed of two mod-ules, the spatial quality aggregation (SQA) network and the tem-poral quality aggregation (TQA) network. Specifically, the SQA network models the quality of video frames using 3D CNN, taking both spatial and temporal masking effects into consideration for the modeling of the perception of human visual system (HVS). In the TQA network, considering the memory effect of HVS facing the temporal variation of frame-level quality, an LSTM-based temporal quality pooling network is proposed to capture the nonlinearities and temporal dependencies involved in the process of quality evaluation. According to the experimental results on two well-established VQA databases, the proposed model could outperform the state-of-the-art metrics. The code of the proposed method is available at: https://github.com/lorenzowu/QAN.

Video cropping is a key research task in video processing field. In this paper, a spatio-temporal saliency based video cropping framework (SalCrop) is introduced including four core modules: video scene detection module, video saliency prediction module, adaptive cropping module, and video codec module. It can automatically reframe videos in the desired aspect ratios. In addition, a large-scale video cropping dataset (VCD) is built for training and testing. Experiments on the VCD test dataset show that our SalCrop outperforms the state-of-the-art algorithms with high efficiency. Besides, a FFmpeg video filter is developed based on the framework, which can be widely used in different scenarios. A demo is available at: https://mme.tencent.com/smartcontent/videoCrop (access token: test_token).

Space-time video super-resolution targets to increase a given video's frame rate and resolution simultaneously. Al-though existing approaches have made great progress, most of them still suffer from the inaccurate approximation of large motions or fail to generate temporal consistent motion trajectory. To alleviate these problems, we carefully review the characteris-tics of different optical flow warping strategies, integrating and enhancing them to achieve more robust capabilities for handling extreme motions and time-modulated interpolation. Specifically, we utilize enhanced backward warping to perform alignment, mine space-time information across low resolution input frames, and propose an enhanced forward warping strategy to interpolate arbitrary intermediate frames. Furthermore, the proposed model can be trained end-to-end and produce intermediate results at any time by merely supervising the center moment. Experimental results show that the proposed algorithm performs favorably against the state-of-the-art methods in objective metrics and subjective visual effects.

Recently, user-oriented auto-encoders (UAEs) have been widely used in recommender systems to learn semantic representations of users based on their historical ratings. However, since latent item variables are not modeled in UAE, it is difficult to utilize the widely available item content information when ratings are sparse. In addition, whenever new items arrive, we need to wait for collecting rating data for these items and retrain the UAE from scratch, which is inefficient in practice. Aiming to address the above two problems simultaneously, we propose a mutually-regularized dual collaborative variational auto-encoder (MD-CVAE) for recommendation. First, by replacing randomly initialized last layer weights of the vanilla UAE with stacked latent item embeddings, MD-CVAE integrates two heterogeneous information sources, i.e., item content and user ratings, into the same principled variational framework where the weights of UAE are regularized by item content such that convergence to a non-optima due to data sparsity can be avoided. In addition, the regularization is mutual in that user ratings can also help the dual item content module learn more recommendation-oriented item content embeddings. Finally, we propose a symmetric inference strategy for MD-CVAE where the first layer weights of the UAE encoder are tied to the latent item embeddings of the UAE decoder. Through this strategy, no retraining is required to recommend newly introduced items. Empirical studies show the effectiveness of MD-CVAE in both normal and cold-start scenarios. Codes are available at https://github.com/yaochenzhu/MD-CVAE.

With the advancement in remote sensing satellite technology, object tracking in satellite videos has become an emerging research field. However, due to small object size, little appearance features, and poor distinguishability between targets and the background, traditional trackers with handcraft visual features achieve poor results in satellite videos. Deep neural networks have shown powerful potential for object tracking in ordinary videos but remain developing in satellite videos. In this letter, a Siamese network and a motion regression network are adopted to form a two-stream deep neural network (SRN) for satellite object tracking, which simultaneously utilizes appearance and motion features. Besides, a trajectory fitting motion (TFM) model based on history trajectories is also employed to further alleviate model drift. Comprehensive experiments demonstrate that the proposed method performs favorably compared with the state-of-the-art tracking methods.

Lossy compression causes the degradation of the classification accuracy of remote sensing (RS) images due to the introduced distortion by compression. In this letter, a convolutional sparse coding (CSC)-based method is proposed to quantitatively measure such an effect. In detail, the filters used in CSC are learned by online convolutional dictionary learning (OCDL) to construct the dictionary. Thereafter, the sparse coefficient maps are obtained based on the alternating direction method of multipliers (ADMM) algorithm. In addition, multiple kernel learning (MKL) is used to estimate the corresponding classification accuracy. The experimental results demonstrate that our method performs better in predicting the classification accuracy of RS images compared with the other state-of-the-art algorithms.

Generating non-existing frames from a consecutive video sequence has been an interesting and challenging problem in the video processing field. Typical kernel-based interpolation methods predict pixels with a single convolution process that convolves source frames with spatially adaptive local kernels, which circumvents the time-consuming, explicit motion estimation in the form of optical flow. However, when scene motion is larger than the pre-defined kernel size, these methods are prone to yield less plausible results. In addition, they cannot directly generate a frame at an arbitrary temporal position because the learned kernels are tied to the midpoint in time between the input frames. In this paper, we try to solve these problems and propose a novel non-flow kernel-based approach that we refer to as enhanced deformable separable convolution (EDSC) to estimate not only adaptive kernels, but also offsets, masks and biases to make the network obtain information from non-local neighborhood. During the learning process, different intermediate time step can be involved as a control variable by means of an extension of coord-conv trick, allowing the estimated components to vary with different input temporal information. This makes our method capable to produce multiple in-between frames. Furthermore, we investigate the relationships between our method and other typical kernel- and flow-based methods. Experimental results show that our method performs favorably against the state-of-the-art methods across a broad range of datasets. Code will be publicly available on URL: https://github.com/Xianhang/EDSC-pytorch.

The establishment of large video affective content analysis datasets, such as LIRIS-ACCEDE, opens up the possibility of utilizing the massive representation power of deep neural networks (DNNs) to model the complex process of eliciting affective responses from video viewers. However, label noise in these datasets poses a considerable challenge to both the training and evaluation of DNNs. The optimization of DNNs requires stochastic gradient descent (SGD), but label noise in the training set leads to an inaccurate estimate of the gradient, which may cause the model to converge to a nonoptima. In addition, label noise in the test set renders the results of model evaluation untrustworthy. In this article, we propose a multimodal deep quality embedding network (MMDQEN) for affective video content analysis. Specifically, MMDQEN can infer the latent label and label quality from the noisy training samples so that cleaner supervision signals are provided to the DNN-based affective classifier, and a tractable objective for MMDQEN is derived with variational inference and conditional independence assumption. In addition, to avoid model evaluation bias incurred by the annotation noise in the test set, new test sets based on the original LIRIS-ACCEDE database, which we name LIRIS-ACCEDE-RANK, are established where the samples are ranked according to their label uncertainty level, with corresponding evaluation metrics introduced accordingly to further reveal the performance of different models. Experiments conducted on both the LIRIS-ACCEDE and the LIRIS-ACCEDE-RANK datasets demonstrate the effectiveness of the proposed method.

Image super-resolution (SR) task aims to recover high-resolution (HR) images from degraded low-resolution (LR) images, which has achieved great progress due to the recent advances of deep neural networks. Due to severe information loss of the LR images, it is more challenging to reconstruct high quality HR images at large scale factors, i. e., higher than 4× . Traditional reference image based SR methods usually perform patch matching to locate detailed texture from HR reference images which could provide fine details from similar image contents. But it suffers from difficulties in achieving good matching in the largely downscaled image space or feature space due to the ill-posed nature between LR and HR mapping. In this paper, we tackle this problem by exploiting fine details contained in reference HR images. Inspired by vector quantization (VQ), we propose a simple yet effective auto-encoder convolutional neural network (CNN) module to learn discrete representations of images. Furthermore, we propose to progressively learn pairs of cross-scale discrete feature representations using paired LR and HR reference images. The coarser scale of the discrete representation is responsible for encoding the global image structure while the paired finer scale of the discrete representation takes charge of capturing missing details in the finer image scale. During inference, continuous features of the test LR image are used as queries to retrieve finer scale discrete representations (value) by searching the nearest coarser scale discrete representations (key). Then, the queries and retrieved values are combined to progressively recover the HR image. Experimental results indicate that when compared with the state-of-the-art image SR models, the proposed method can achieve advanced performance in terms of both objective quality and subjective quality. The code will be available on URL: https://github.com/sunwj/refsr.

For a typical Scene Graph Generation (SGG) method in image understanding, there usually exists a large gap in the performance of the predicates’ head classes and tail classes. This phenomenon is mainly caused by the semantic overlap between different predicates as well as the long-tailed data distribution. In this paper, a Predicate Correlation Learning (PCL) method for SGG is proposed to address the above problems by taking the correlation between predicates into consideration. To measure the semantic overlap between highly correlated predicate classes, a Predicate Correlation Matrix (PCM) is defined to quantify the relationship between predicate pairs, which is dynamically updated to remove the matrix’s long-tailed bias. In addition, PCM is integrated into a predicate correlation loss function (LPC) to reduce discouraging gradients of unannotated classes. The proposed method is evaluated on several benchmarks, where the performance of the tail classes is significantly improved when built on existing methods.

Recently an impressive development in immersive technologies, such as Augmented Reality (AR), Virtual Reality (VR) and 360${^\circ }$video, has been witnessed. However, methods for quality assessment have not been keeping up. This paper studies quality assessment of 360${^\circ }$video from the cross-lab tests (involving ten laboratories and more than 300 participants) carried out by the Immersive Media Group (IMG) of the Video Quality Experts Group (VQEG). These tests were addressed to assess and validate subjective evaluation methodologies for 360${^\circ }$video. Audiovisual quality, simulator sickness symptoms, and exploration behavior were evaluated with short (from 10 seconds to 30 seconds) 360${^\circ }$sequences. The following factors’ influences were also analyzed: assessment methodology, sequence duration, Head-Mounted Display (HMD) device, uniform and non-uniform coding degradations, and simulator sickness assessment methods. The obtained results have demonstrated the validity of Absolute Category Rating (ACR) and Degradation Category Rating (DCR) for subjective tests with 360${^\circ }$videos, the possibility of using 10-second videos (with or without audio) when addressing quality evaluation of coding artifacts, as well as any commercial HMD (satisfying minimum requirements). Also, more efficient methods than the long Simulator Sickness Questionnaire (SSQ) have been proposed to evaluate related symptoms with 360${^\circ }$videos. These results have been instrumental for the development of the ITU-T Recommendation P.919. Finally, the annotated dataset from the tests is made publicly available for the research community.

Graph embedding based methods have been used in recommendation systems recently, owing to their advances in modeling nodes as embeddings in a low-dimensional space. By effective neighborhood aggregation, graph convolutional networks can exploit high-order connections of neighbors such that the learned embeddings could be more informative thus improve the recommendation performance. However, user and item representations learned by graph aggregation inherently contain uncertainty due to sparsity of user-item interactions and noise of item features. To address these challenges, we propose a multi-modal variational graph auto-encoder (MVGAE) method. Specifically, we design modality-specific variational encoders that learn a Gaussian variable for each node whereas the mean vector represents semantic information and the variance vector denotes the noise level of the corresponding modality. Moreover, with the conditional independence assumption, the modality-specific Gaussian node embeddings are fused according to the product-of-experts principle, where the semantic information in each modality is weighted based on the estimated uncertainty level. Extensive experiments on three public datasets, Amazon Movies, Amazon Electronics and AliShop-7 C, demonstrate that our proposed method achieves competitive performance when compared with the state-of-the-art algorithms.

As one of the most important parts of virtual reality applications, panoramic video has become very popular. Differing from the traditional plane video, panoramic video is projected onto the 2D plane for processing, while viewed in the spherical domain. Quality of the projected plane cannot represent the real visual quality in VR viewing. In this paper, a no-reference (NR) quality assessment method is proposed for panoramic video. The proposed method extracts spatial and temporal video features from spherical domain to better model the perceived visual quality, alleviating the influence of non-uniform projection. Experiments conducted on a subjective quality database for panoramic video show that the proposed method can achieve better performance compared with the NR method designed for traditional 2D video.

In this paper, we propose a two stage optimal bit allocation scheme for HEVC hierarchical coding structure. The two stage, i.e., the frame-level and the CTU-level bit allocation, are separately conducted in the traditional rate control methods. In our proposed method, the optimal allocation in the second stage is firstly considered, and then the allocation strategy in the second stage is deemed as a foreknowledge in the first stage and applied to guide the frame-level bit allocation. With the formulation, the two stage bit allocation problem can be converted to a joint optimization problem. By solving the formulated optimization problem, the two stage optimal bit allocation scheme is established, in which more appropriate number of bits can be allocated to each frame and each CTU. The experimental results show that our proposed method can bring higher coding efficiency while satisfying the constraint of bit rate precisely.

In this paper, we propose a reinforcement learning based region of interest (ROI) bit allocation method for gaming video coding in Versatile Video Coding (VVC). Most current ROI-based bit allocation methods rely on bit budgets based on frame-level empirical weight allocation. The restricted bit budgets influence the efficiency of ROI-based bit allocation and the stability of video quality. To address this issue, the bit allocation process of frame and ROI are combined and formulated as a Markov decision process (MDP). A deep reinforcement learning (RL) method is adopted to solve this problem and obtain the appropriate bits of frame and ROI. Our target is to improve the quality of ROI and reduce the frame-level quality fluctuation, whilst satisfying the bit budgets constraint. The RL-based ROI bit allocation method is implemented in the latest video coding standard and verified for gaming video coding. The experimental results demonstrate that the proposed method achieves a better quality of ROI while reducing the quality fluctuation compared to the reference methods.

In the video saliency prediction task, one of the key issues is the utilization of temporal contextual information of keyframes. In this paper, a deep reinforcement learning agent for video saliency prediction is proposed, designed to look around adjacent frames and adaptively generate a salient contextual window that contains the most correlated information of keyframe for saliency prediction. More specifically, an action set step by step decides whether to expand the window, meanwhile a state set and reward function evaluate the effectiveness of the current window. The deep Q-learning algorithm is followed to train the agent to learn a policy to achieve its goal. The proposed agent can be regarded as plug-and-play which is compatible with generic video saliency prediction models. Experimental results on various datasets demonstrate that our method can achieve an advanced performance.

Video captioning is considered to be challenging due to the combination of video understanding and text generation. Recent progress in video captioning has been made mainly using methods of visual feature extraction and sequential learning. However, the syntax structure and semantic consistency of generated captions are not fully explored. Thus, in our work, we propose a novel multimodal attention based framework with Part-of-Speech (POS) sequence guidance to generate more accu-rate video captions. In general, the word sequence generation and POS sequence prediction are hierarchically jointly modeled in the framework. Specifically, different modalities including visual, motion, object and syntactic features are adaptively weighted and fused with the POS guided attention mechanism when computing the probability distributions of prediction words. Experimental results on two benchmark datasets, i.e. MSVD and MSR-VTT, demonstrate that the proposed method can not only fully exploit the information from video and text content, but also focus on the decisive feature modality when generating a word with a certain POS type. Thus, our approach boosts the video captioning performance as well as generating idiomatic captions.

A visual scanpath represents the human eye movements when scanning the visual field for acquiring and receiving visual information. Predicting visual scanpaths when a certain stimulus is presented plays an important role in modeling overt human visual attention and search behavior. In this paper, we presented an 'Inhibition of Return - Region of Interest' (IOR-ROI) recurrent mixture density network based framework learning to produce human-like visual scanpaths under task-free viewing conditions. The proposed model simultaneously predicts a sequence of ordered fixation positions and their corresponding fixation durations. Our model integrates bottom-up features and semantic features extracted by convolutional neural networks. Then the integrated feature maps are fed into the IOR-ROI Long Short-Term Memory (LSTM) which is the core component of the proposed model. The IOR-ROI LSTM is a dual LSTM unit, i.e., the IOR-LSTM and the ROI-LSTM, capturing IOR dynamics and gaze shift behavior simultaneously. IOR-LSTM simulates the visual working memory to adaptively maintain and update visual information regarding previously fixated regions. ROI-LSTM is responsible for predicting the next possible ROIs given the spatially inhibited image feature maps on the feature-wise basis. Fixation duration is predicted by a regression neural network given the viewing history and image feature maps corresponding to currently fixated ROI. Considering the eye movement pattern variations among subjects, a mixture density network is adopted to model the next fixation distribution as Gaussian mixtures and the fixation duration is also modeled using Gaussian distribution. Our model is evaluated on the OSIE and MIT low resolution eye-tracking datasets and experimental results indicate that the proposed method can achieve superior performance in predicting visual scanpaths. The code will be publicly available on URL: https://github.com/sunwj/scanpath.

In this letter, a method called Semantic Information Oriented No-Reference (SIONR) video quality assessment model is developed, which can effectively represent quality degradation of video by taking the variations of semantic information into consideration. Specially, temporal variations of the semantic features between adjacent frames are calculated to consider the inconsistency of the static semantic information. Moreover, low-level features are also applied as a supplementary to take distortions related to local details into consideration. Experimental results demonstrate that our proposed method obtains competitive performance compared with state-of-the-art methods in the two databases. Also, our model achieves good generalization capability. The code is available at: https://github.com/lorenzowu/SIONR.

With the rapid development of video distribution and broadcasting, affective video content analysis has attracted a lot of research and development activities recently. Predicting emotional responses of movie audiences is a challenging task in affective computing, since the induced emotions can be considered relatively subjective. In this article, we propose a multimodal local-global attention network (MMLGAN) for affective video content analysis. Inspired by the multimodal integration effect, we extend the attention mechanism to multi-level fusion and design a multimodal fusion unit to obtain a global representation of affective video. The multimodal fusion unit selects key parts from multimodal local streams in the local attention stage and captures the information distribution across time in the global attention stage. Experiments on the LIRIS-ACCEDE dataset, the MediaEval 2015 and 2016 datasets, the FilmStim dataset, the DEAP dataset and the VideoEmotion dataset demonstrate the effectiveness of our approach when compared with the state-of-the-art methods.

Semantic segmentation of remote sensing images plays an important role in many applications. However, a remote sensing image typically comprises a complex and heterogenous urban landscape with objects in various sizes and materials, which causes challenges to the task. In this work, a novel adaptive fusion network (AFNet) is proposed to improve the performance of very high resolution (VHR) remote sensing image segmentation. To coherently label size-varied ground objects from different categories, we design multilevel architecture with the scale-feature attention module (SFAM). By SFAM, at the location of small objects, low-level features from the shallow layers of convolutional neural network (CNN) are enhanced, whilst for large objects, high-level features from deep layers are enhanced. Thus, the features of size-varied objects could be preserved during fusing features from different levels, which helps to label size-varied objects. As for labeling the category with high intra-class difference and varied scales, the multiscale structure with a scale-layer attention module (SLAM) is utilized to learn representative features, where an adjacent score map refinement module (ACSR) is employed as the classifier. By SLAM, when fusing multiscale features, based on the interested objects scale, feature map from appropriate scale is given greater weights. With such a scale-aware strategy, the learned features can be more representative, which is helpful to distinguish objects for semantic segmentation. Besides, the performance is further improved by introducing several nonlinear layers to the ACSR. Extensive experiments conducted on two well-known public high-resolution remote sensing image data sets show the effectiveness of our proposed model. Code and predictions are available at https://github.com/athauna/AFNet/

Constructing adversarial examples in a black-box threat model injures the original images by introducing visual distortion. In this paper, we propose a novel black-box attack approach that can directly minimize the induced distortion by learning the noise distribution of the adversarial example, assuming only loss-oracle access to the black-box network. To quantify visual distortion, the perceptual distance between the adversarial example and the original image, is introduced in our loss. We first approximate the gradient of the corresponding non-differentiable loss function by sampling noise from the learned noise distribution. Then the distribution is updated using the estimated gradient to reduce visual distortion. The learning continues until an adversarial example is found. We validate the effectiveness of our attack on ImageNet. Our attack results in much lower distortion when compared to the state-of-the-art black-box attacks and achieves 100% success rate on InceptionV3, ResNet50 and VGG16bn. Furthermore, we theoretically prove the convergence of our model. The code is publicly available at https://github.com/Alina-1997/visual-distortion-in-attack.

In this paper, a multi-objective optimization of quality based coding tree unit (CTU) level rate control method, named MORC, for low-delay video coding of Versatile Video Coding (VVC) is proposed. Unlike the traditional rate control algorithms, objectives of minimizing average distortion and minimizing quality fluctuation are introduced and considered simultaneously to obtain the optimal solution. Weighting method is then used to convert the multi-objective optimization problem into a simplistic single-objective problem. Accordingly, given the target rate constraint, rate allocation is formulated as a convex optimization problem to achieve better trade-off among three multiple objectives: minimizing average video distortion, meeting rate constraint and minimizing video quality fluctuation. Based on the D- λ model, a two-stage method is further proposed to solve the optimization problem. Experiments have been conducted on the test model of VVC, which is VTM 3.0. Compared with the state-of-the-art rate control algorithms, the proposed method can obtain the Pareto optimal solution, which dominates the solution of other methods and is able to achieve better trade-off among multiple objectives.

In this article, a new CTU-level bit allocation scheme aimed at subjectively optimized video coding for video conferencing applications is presented, in which the non-cooperative Stackelberg game is used for formulating and solving the bit allocation problem during the encoding process. Videos are divided into the Region of interests (ROI) which attracts people more and the non-ROI. The two regions are defined as the players in the game, where the ROI is the leader who takes the priority in strategy making and the non-ROI follows the leader's strategy. Based on the formulated game, the bit allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, the bit allocation strategy between the ROI and the non-ROI will be established. Then the bits will be allocated to each CTU by a Newton-method-based algorithm for encoding, in which a trade-off between the ROI's quality and the overall quality can be achieved. Both the objective and subjective experimental results show that our proposed bit allocation method can improve the quality of ROI significantly with an acceptable overall quality degradation, leading to a better visual experience.

In this paper, a recurrent neural network is designed for video saliency prediction considering spatial-temporal features. In our work, video frames are routed through the static network for spatial features and the dynamic network for temporal features. For the spatial-temporal feature integration, a novel select and re-weight fusion model is proposed which can learn and adjust the fusion weights based on the spatial and temporal features in different scenes automatically. Finally, an attention-aware convolutional long short term memory (ConvLSTM) network is developed to predict salient regions based on the features extracted from consecutive frames and generate the ultimate saliency map for each video frame. The proposed method is compared with state-of-the-art saliency models on five public video saliency benchmark datasets. The experimental results demonstrate that our model can achieve advanced performance on video saliency prediction.

In this paper, an attentive cross-modal fusion (ACMF) network is proposed for RGB-D salient object detection. The proposed method selectively fuses features in a cross-modal manner and uses a fusion refinement module to fuse output features from different resolutions. Our attentive cross-modal fusion network is built based on residual attention. In each level of ResNet output, both the RGB and depth features are turned into an identity map and a weighted attention map. The identity map is reweighted by the attention map of the paired modality. Moreover, the lower level features with higher resolution are adopted to refine the boundary of detected targets. The entire architecture can be trained end-to-end. The proposed ACMF is compared with state-of-the-art methods on eight recent datasets. The results demonstrate that our model can achieve advanced performance on RGB-D salient object detection.

In e-commerce search, vectorized matching is the most important approach besides lexical matching, where learning vector representations for entities (e.g., query, item, shop) plays a crucial role. In this work, we focus on vectorized search matching model for shop search in Taobao. Unlike item search, shop search is faced with serious behavior sparsity and long-tail problem. To tackle this, we take the first step to transfer knowledge from item search, i.e., leveraging items purchased under a query and the shops they belong to. Moreover, we propose a novel gated heterogeneous graph learning model (named GHL) to derive vector representations for entities. Both first-order and second-order proximity of queries and shops are exploited to fully mine the heterogeneous relationships. And to relieve long-tail phenomenon, we devise an innovative gated neighbor aggregation scheme where each type of entities (i.e., hot ones and long-tail ones) can benefit from the heterogeneous graph in an automatic way. Finally, the whole framework is jointly trained in an end-to-end fashion. Offline evaluation results on real-world data of Taobao shop search platform demonstrate that the proposed model outperforms existing graph based methods, and online A/B tests show that it is highly effective and achieves significant CTR improvements.

Rate control plays an important role in practical video coding. In this paper, we present a rate control method for Versatile Video Coding (VVC). We conduct some statistical analysis to investigate the influence of the skip block upon the rate-distortion parameter estimation and quality dependency among frames. Based on the analysis, we propose the rate-distortion parameter updating strategy and establish a quality dependency factor based frame-level bit allocation scheme. Experimental results show that the proposed method can achieve up to 7.31%/5.44% BD-rate improvements in Low-Delay/Random-Access configuration upon previous rate control method in VVC respectively. The presented rate control algorithm has already been adopted into VVC VTM platform by Joint Video Experts Team (JVET).

One of the key challenges in no-reference video quality assessment (NR-VQA) is the absence of the reference video to measure the similarity or difference between the distorted video and the original one. In this paper, an encoder-decoder model is proposed to predict pixel-by-pixel similarity maps from the distorted video. The model takes multiple frames as input since correlated pixels of adjacent frames can be exploited to recover the similarity map of the middle frame of the distorted video clip. In addition, to further exploit the temporal perception mechanism of the human visual system (HVS), which is relevant to the perceptual video distortion measurement, visual persistence and temporal memory effects are considered in the spatio-temporal pooling network design. Experimental results demonstrate that our proposed method outperforms state-of-the-art NR-VQA metrics.

Shop search has become an increasingly important service provided by Taobao, the China's largest e-commerce platform. By using shop search, a user can easily identify the desired shop that provides a full-scale of relevant items matching his information need. With the tremendous growth of users and shops, shop search faces several unique challenging problems: 1) many shop names do not fully express what they sell, i.e., the semantic gap between user query and shop name; 2) due to the lack of user interactions, it is difficult to deliver a good search result for the long-tail queries and retrieve long-tail shops that are highly relevant to a query.

In the hybrid video coding framework, inter prediction is an efficient tool to exploit temporal redundancy. Since the performance of inter prediction depends on the content of reference frames, coding efficiency can be significantly improved by having more effective reference frames. In this paper, we propose an enhanced inter coding scheme by generating artificial reference frames with deep neural network. Specifically, a new reference frame is interpolated from two-sided previously reconstructed frames, which can be regarded as the prediction of the to-be-coded frame. The synthesized frame is merged into reference picture list for motion estimation to further decrease the prediction residual. We integrate the proposed method into HM-16.20 under random access configuration. Experimental results show that the proposed method can significantly boost the coding performance, which provides 4.6% BD-rate reduction on average compared to HEVC baseline.

Images and videos suffer from undesirable visual artifacts at high compression ratios, which is due to the use of the discrete cosine transform and scalar quantization in the compression. To restore the quantized coefficients via producing the quantization error, we propose a coefficients restoration convolutional neural network in the frequency domain (FD-CRNet). Taking advantage of residual learning, the proposed FD-CRNet efficiently exploits the related distribution of different frequency components. The squeeze-and-excitation block (SE block) is adopted to reduce the computational complexity and better restoration performance. Experimental results show the quantized coefficients are recovered near the lossless coefficients effectively, which outperforms the existed coefficients restoration methods. In addition, the performance of methods in the spatial domain is significantly improved by FD-CRNet with more authentic details and sharper edges when removing the compression artifacts.

360-degree videos have drawn increasing attention from both industry and research communities with the popularity of virtual reality applications. The 360-degree video is originally represented as a sphere and able to provides omnidirectional view. Equirectangular projection is widely applied to convert the 360-degree video from the 3D sphere to a 2D plane for the purposes of compression and storage. However, the content of the projected video is distorted, which brings challenges for the conventional video coding methods. In this paper, a novel intra prediction algorithm is proposed for 360-degree video coding. The proposed algorithm tries to handle the distortion by adapting the process of intra prediction to spherical domain. The proposed algorithm is implemented in High Efficiency Video Coding (HEVC) test model HM16.16. Experimental results show that the proposed algorithm brings 0.2% BD-rate reduction on average for 360-degree video coding.

Video super resolution is a challenging task and has attracted the attention of many researchers in recent years. In this paper, we propose a multi-stage spatio-temporal feature fusion network. Different from existing methods that only aggregate features from temporal branch once at a specified s tage of network, the proposed network is organized in a multi-stage manner so that the temporal correlation in features at different stages of the network can be fully exploited. Furthermore, we propose the temporal encoding convLSTM to effectively capture the temporal information at the end of each stage. Experiments on vid4 and viemo-90K demonstrate the effectiveness of the proposed method.

Predicting the popularity of a micro-video is a challenging task, due to a number of factors impacting the distribution such as the diversity of the video content and user interests, complex online interactions, etc. In this paper, we propose a multimodal variational encoder-decoder (MMVED) framework that considers the uncertain factors as the randomness for the mapping from the multimodal features to the popularity. Specifically, the MMVED first encodes features from multiple modalities in the observation space into latent representations and learns their probability distributions based on variational inference, where only relevant features in the input modalities can be extracted into the latent representations. Then, the modality-specific hidden representations are fused through Bayesian reasoning such that the complementary information from all modalities is well utilized. Finally, a temporal decoder implemented as a recurrent neural network is designed to predict the popularity sequence of a certain micro-video. Experiments conducted on a real-world dataset demonstrate the effectiveness of our proposed model in the micro-video popularity prediction task.

Although video enhancement and evaluation have been studied for many years, they are still challenging due to the evolutions of video acquiring and processing techniques. While the development of deep learning is undoubtedly exciting and has demonstrated its superior performance in a variety of applications, it is important to further investigate advanced technologies and solutions to bring seemingly endless possibilities for video enhancement and evaluation. This Special Section intends to collect some recent solutions toward video enhancement and evaluation.

Just noticeable difference (JND) models reveal visibility limitation of human visual system (HVS) and exploit perceptual redundancy in the visual signal. Conventional just noticeable difference (JND) model supposes the visual acuity is homogeneous with or without considerations on the isoeccentric locations. Recent research on vision science indicates that the effect of eccentricity on visual sensitivity is not homogeneous across the visual field (VF). Relevant studies have found visual acuity at the isoeccentric locations is better along the horizontal meridian than along the vertical meridian, i.e., horizontal-vertical anisotropy (HVA). Along the vertical meridian, better performance has been reported for the inferior meridian compared to the superior meridian, i.e., vertical-meridian asymmetry (VMA). These properties of HVS should be taken into account to yield a better JND estimation, for better exploiting the perceptual redundancy. In this paper, we use two hemi-ellipses to represent the iso-acuity contour, which can better express the HVA and VMA effects. Moreover, we design psychophysical experiments under different test conditions, where the results show the hemi-ellipse model performs better when compared to the circular model. Finally, we build an asymmetric foveated JND model using two hemi-ellipses for approximation. When comparing with traditional JND models, experimental results demonstrate the effectiveness of the proposed model.

Given a video sequence, video frame interpolation aims to synthesize an in-between frame of two consecutive frames. In this paper, we propose a multi-scale position feature transform (MS-PFT) network for video frame interpolation where two parallel prediction networks and one optimization network are designed to predict the features of target frame and generate the final interpolation result, respectively. To increase the fidelity of the synthesised frames, we propose to apply a position feature transform (PFT) layer in the residual blocks of the prediction networks to estimate scaling factors which help evaluate different degrees of the importance of deep features around a target pixel. A PFT layer utilizes optical flow to extract and generate position features and then adjusts the learning process of our model. We further extend our model into a multi-scale structure in which each scale of the network shares the same parameters to maximise the efficiency of our network with model size unchanged. The experiments show that our method can handle the challenging scenarios like occlusion and large motion effectively and the proposed method outperforms those state-of-the-art approaches on different datasets.

Dictionary learning has emerged as a powerful tool for a range of image processing applications and a proper dictionary always plays a key issue to the final achievable performance. In this paper, a class-oriented discriminative dictionary learning (CODDL) method is presented for image classification applications. It takes a comprehensive consideration of multiple optimization objectives, emphasizing class discrimination of both dictionary atoms and representation coefficients. The atoms of the learned dictionary should be grouped into class level sub-dictionaries. Meanwhile, the sparse representation coefficients of an input sample should be concentrated on the sub-dictionary of the class it belongs to. Then, based on the learned class-oriented discriminative dictionary, the structured representation coefficients can thus be used for image classification with a simple and efficient classification scheme. The superior performance of the proposed algorithm is demonstrated through extensive experiments.

Deep learning has achieved great success in the past decade, especially in the fields of computer vision and image processing. After witnessing such success, video coding experts are motivated to consider whether deep learning can also benefit video coding, and if so, they seek to discover why and how. Indeed, a number of research studies have been conducted to explore deep learning for image and video coding, which has been an active and fast-growing research area especially since the year 2015. These prior arts can be divided into two categories: new coding schemes that are built solely upon deep networks (deep schemes), and deep network-based coding tools that are embedded into traditional coding schemes (deep tools). Moreover, in the responses to the joint call for proposals on video compression with capability beyond High Efficiency Video Coding (HEVC), a number of deep tools have been proposed, and some of them are further studied for the upcoming Versatile Video Coding (VVC). In this paper, we summarize the ongoing efforts in the Joint Video Experts Team about the proposed deep tools, and we discuss several promising tools in much detail, including neural network-based intra prediction, convolutional neural network (CNN) based in-loop filtering, and CNN-based block-adaptive-resolution coding. A series of experimental results are provided to demonstrate the capability of these tools in achieving higher compression efficiency than the VVC or HEVC anchor. These results shed light on the promising direction of deep learning-based future video coding, towards which a lot of open problems call for further study.

Monocular 3D object detection has the merit of low cost and can be served as an auxiliary module for autonomous driving system, becoming a growing concern in recent years. In this paper, we present a monocular 3D object detection method with feature enhancement networks, which we call MonoFENet. Specifically, with the estimated disparity from the input monocular image, the features of both the 2D and 3D streams can be enhanced and utilized for accurate 3D localization. For the 2D stream, the input image is used to generate 2D region proposals as well as to extract appearance features. For the 3D stream, the estimated disparity is transformed into 3D dense point cloud, which is then enhanced by the associated front view maps. With the RoI Mean Pooling layer, 3D geometric features of RoI point clouds are further enhanced by the proposed point feature enhancement (PointFE) network. The region-wise features of image and point cloud are fused for the final 2D and 3D bounding boxes regression. The experimental results on the KITTI benchmark reveal that our method can achieve state-of-the-art performance for monocular 3D object detection.

Deep convolutional neural network based image super-resolution (SR) models have shown superior performance in recovering the underlying high resolution (HR) images from low resolution (LR) images obtained from the predefined downscaling methods. In this paper, we propose a learned image downscaling method based on content adaptive resampler (CAR) with consideration on the upscaling process. The proposed resampler network generates content adaptive image resampling kernels that are applied to the original HR input to generate pixels on the downscaled image. Moreover, a differentiable upscaling (SR) module is employed to upscale the LR result into its underlying HR counterpart. By back-propagating the reconstruction error down to the original HR input across the entire framework to adjust model parameters, the proposed framework achieves a new state-of-the-art SR performance through upscaling guided image resamplers which adaptively preserve detailed information that is essential to the upscaling. Experimental results indicate that the quality of the generated LR image is comparable to that of the traditional interpolation based method and the significant SR performance gain is achieved by deep SR models trained jointly with the CAR model. The code is publicly available on: https://github.com/sunwj/CAR.

Reinforcement learning (RL) has shown its advantages in image captioning by optimizing the non-differentiable metric directly in the reward learning process. However, due to the reward hacking problem in RL, maximizing reward may not lead to better quality of the caption, especially from the aspects of propositional content and distinctiveness. In this work, we propose to use a new learning method, meta learning, to utilize supervision from the ground truth whilst optimizing the reward function in RL. To improve the propositional content and the distinctiveness of the generated captions, the proposed model provides the global optimal solution by taking different gradient steps towards the supervision task and the reinforcement task, simultaneously. Experimental results on MS COCO validate the effectiveness of our approach when compared with the state-of-the-art methods.

In this paper, we propose a two-stream refinement network for RGB-D saliency detection. A fusion refinement module is designed to fuse output features from different resolution and modals. The structure information from depth helps distinguish between foreground and background and the lower level features with higher resolution can be adopted to refine the boundary of detected targets. The proposed model predicts high-resolution saliency map and then use a propagation-based module to further refine object boundary. Experimental results demonstrate that the proposed method performs well against to the state of the art methods on the recent RGB-D salient object detection dataset.

Recent works use 3D convolutional neural networks to explore spatio-temporal information for human action recognition. However, they either ignore the correlation between spatial and temporal features or suffer from high computational cost by spatio-temporal features extraction. In this work, we propose a novel and efficient Mutually Reinforced Spatio-Temporal Convolutional Tube (MRST) for human action recognition. It decomposes 3D inputs into spatial and temporal representations, mutually enhances both of them by exploiting the interaction of spatial and temporal information and selectively emphasizes informative spatial appearance and temporal motion, meanwhile reducing the complexity of structure. Moreover, we design three types of MRSTs according to the different order of spatial and temporal information enhancement, each of which contains a spatio-temporal decomposition unit, a mutually reinforced unit and a spatio-temporal fusion unit. An end-to-end deep network, MRST-Net, is also proposed based on the MRSTs to better explore spatio-temporal information in human actions. Extensive experiments show MRST-Net yields the best performance, compared to state-of-the-art approaches.

The 3D convolutional neural networks recently have been applied to explore spatial-temporal content for video action recognition. However, they either suffer from high computational cost by spatial-temporal feature extraction or ignore the correlation between appearance and motion. In this work, we propose a novel Cross-Fiber Spatial-Temporal Co-enhanced (CFST) architecture aiming to reduce the number of parameters tremendously while achieve accurate recognition of actions. We slice the complex 3D convolutional network into a group of lightweight fibers that run through the whole network. Crossing separated fibers, we introduce the Cross-Fiber Recalibration unit which shares extracted features from each fiber and measures the interaction between fibers to emphasize informative ones. Within each fiber, the Spatial-Temporal Co-enhanced unit is put forward to co-enhance the learning of spatial and temporal features, leading to more discriminative spatial-temporal representation. An end-to-end deep network, CFST-Net, is also presented based on the proposed CFST architecture for video action recognition. Extensive experimental results show that our CFST-Net significantly boosts the performance of existing convolution networks and achieves state-of-the-art accuracy on three challenging benchmarks, i.e., UCF-101, HMDB-51 and Kinetics-400, with much fewer parameters and FLOPs.

In this paper, a Hierarchical Temporal Model (HTM) is proposed for the video captioning task, based on exploring the global and local temporal structure to better recognize fine-grained objects and actions. In our HTM, the encoder and decoder are hierarchically aligned according to different levels of features. The encoder applies two LSTM layers to construct temporal structures at both frame-level and object-level where the attention mechanism is applied to locate objects of interest, and the decoder uses corresponding LSTM layers to extract pivotal features from global to local through multi-level attention mechanism. Moreover, the local temporal structure is constructed implicitly from candidate object-oriented features under the guidance of global temporal-spatial representation, that could generate more accurate descriptions in handling shot-switching problems. Experiments on the widely used Microsoft Video Description Corpus (MSVD) and Charades datasets demonstrate the effectiveness of our proposed approach when compared to the state-of-the-art methods.

Affective video content analysis has attracted a lot of attention recently. However, it faces various challenges such as the gap between intrinsic visual-aural features and spontaneous human emotional response, as well as ubiquitously existed label noise in affective annotations. Therefore, it is difficult to get useful supervision signals to learn well-generalized patterns responsible for eliciting affective impact. Observing that label uncertainty severely obstacles the progress of affective video content analysis, a deep denoising framework is proposed to infer true latent labels and annotation qualities from heavy label noise, fully utilizing the multimodal information contained in videos. Specifically, a quality embedding network is adopted in a multimodal fashion, and corresponding stochastic gradient descent (SGD) optimization objective is derived with variational inference and conditional independence assumption. To better reflect the effectiveness of affective models, new test sets are established based on the widely used LIRIS-ACCEDE dataset where the training database is kept unchanged, and a ranking-based evaluation metric is introduced accordingly. Experiments conducted on both the original LIRIS-ACCEDE test dataset and the refined one demonstrate the effectiveness of the proposed method.

With the development of applications such as wireless displays and remote desktop, screen content coding (SCC) has been a topic of increasing interest. The state-of-the-art partition modes in AVS3 restrict a parent coding unit (CU) to partition into no more than four sub-CUs in each partition process. To obtain more than four smaller sub-CUs, a parent CU needs to partition recursively. In this paper, to simplify the recursive partition processes and improve SCC efficiency, a novel partition mode which allows a parent CU to be partitioned into several pre-defined smaller sub-CUs with the same shape and size directly is proposed. We make some pre-experiments to verify the validity of the proposed partition mode, and limit the application scope to four rectangular CU sizes based on the pre-experimental results. As demonstrated in the experimental results, the proposed partition mode can obviously improve SCC efficiency.

In signal compression, distortion information is significant for rate distortion optimization. In this paper, we propose a convolutional neural network (CNN) to predict distortion information for H.265/HEVC. With the strong representation power of CNN, structural similarity (SSIM) maps indicating distortion information can be predicted directly in an end-to-end, pixel-to-pixel way. Different from traditional CNNs which focus on learning one-to-one mappings from input to output, we show that our CNN model can predict SSIM maps conditioned on quantization parameters (QPs), realizing one-to-many mappings. To construct our CNN network, QP labels are designed as conditions to feed the CNN model. We also apply symmetrical network architecture and multi-level feature fusion method to ensure our network can utilize both high-level semantic features and low-level structure features. The experiments on MS COCO database demonstrate the effectiveness of our CNN-based method for SSIM prediction.

With the increasing popularity of micro-video sharing where people shoot short-videos effortlessly and share their daily stories on social media platforms, the micro-video recommendation has attracted extensive research efforts to provide users with micro-videos that interest them. In this paper, a hypothesis we explore is that, not only do users have multi-modal interest, but micro-videos have multi-modal targeted audience segments. As a result, we propose a novel framework User-Video Co-Attention Network (UVCAN), which can learn multi-modal information from both user and microvideo side using attention mechanism. In addition, UVCAN reasons about the attention in a stacked attention network fashion for both user and micro-video. Extensive experiments on two datasets collected from Toffee present superior results of our proposed UVCAN over the state-of-the-art recommendation methods, which demonstrate the effectiveness of the proposed framework.

Efficient compression of omnidirectional video is important for emerging virtual reality applications. To compress this kind of video, each frame is projected to a 2D plane [e.g., equirectangular projection (ERP) map] first, adapting to the input format of existing video coding systems. At the display side, an inverse projection is applied to the reconstructed video to restore signals in spherical domain. Such a projection, however, makes presentation and encoding in different domains. Thus, an encoder agnostic to the projection performs inefficiently. In this paper, we analyze how a projection influences the distortion measurements in different domains. Based on the analysis, we propose a scheme to optimize the encoding process based on signals' distortion in spherical domain. With the proposed optimization, an average 4.31% (up to 9.67%) luma BD-rate reduction is achieved for ERP in random access configuration. The corresponding bit saving is averagely 10.84% (up to 34.44%) when considering the viewing field being π/2 . The proposed method also benefits other projections and viewport settings, with a marginal complexity increase.

In contrast with traditional videos, omnidirectional videos enable spherical viewing direction with support for head-mounted displays, providing an interactive and immersive experience. Unfortunately, to the best of our knowledge, there are only a few visual quality assessment (VQA) methods, either subjective or objective, for omnidirectional video coding. This paper proposes both subjective and objective methods for assessing the quality loss in encoding an omnidirectional video. Specifically, we first present a new database, which includes the viewing direction data from several subjects watching omnidirectional video sequences. Then, from our database, we find a high consistency in viewing directions across different subjects. The viewing directions are normally distributed in the center of the front regions, but they sometimes fall into other regions, related to the video content. Given this finding, we present a subjective VQA method for measuring the difference mean opinion score (DMOS) of the whole and regional omnidirectional video, in terms of overall DMOS and vectorized DMOS, respectively. Moreover, we propose two objective VQA methods for the encoded omnidirectional video, in light of the human perception characteristics of the omnidirectional video. One method weighs the distortion of pixels with regard to their distances to the center of front regions, which considers human preference in a panorama. The other method predicts viewing directions according to the video content, and then the predicted viewing directions are leveraged to allocate weights to the distortion of each pixel in our objective VQA method. Finally, our experimental results verify that both the subjective and objective methods proposed in this paper advance the state-of-the-art VQA for omnidirectional videos.

In this paper, we propose a novel two-stream neural network for video saliency prediction. Unlike some traditional methods based on hand-crafted feature extraction and integration, our proposed method automatically learns saliency related spatiotemporal features from human fixations without any pre-processing, post-processing, or manual tuning. Video frames are routed through the spatial stream network to compute static or color saliency maps for each of them. And a new two-stage temporal stream network is proposed, which is composed of a pre-trained 2D-CNN model (SF-Net) to extract saliency related features and a shallow 3D-CNN model (Te-Net) to process these features, for temporal or dynamic saliency maps. It can reduce the requirement of video gaze data, improve training efficiency, and achieve high performance. A fusion network is adopted to combine the outputs of both streams and generate the final saliency maps. Besides, a convolutional Gaussian priors (CGP) layer is proposed to learn the bias phenomenon in viewing behavior to improve the performance of the video saliency prediction. The proposed method is compared with state-of-the-art saliency models on two public video saliency benchmark datasets. The results demonstrate that our model can achieve advanced performance on video saliency prediction.

Image compression is essential for remote sensing due to the large volume of produced remote sensing imagery and system's limited transmission or storage capacity. As one of the most important applications, classification might be affected due to the introduced distortion during compression. Hence, we perform a quantitative study on the effects of compression on remote sensing image classification and propose a method to estimate the remote sensing image classification accuracy based on fractal analysis. Multiscale feature extraction is performed and a multiple kernel learning approach is proposed accordingly. The experimental results on our established database indicate that the classification accuracy predicted by our method exhibits high consistency with the ground truth and our method shows its superiority when compared with other classical reference algorithms.

Spectral decorrelation has been considered an important approach in multispectral image compression to remove the redundancy among bands. In this paper, we propose a novel adaptive spectral decorrelation method based on clustering analysis for Moderate Resolution Imaging Spectroradiometer (MODIS) image compression. The remote sensing image bands are divided into different clusters by the method based on density peak clustering. Then, reversible Karhunen-Loeve transform and polynomial least square estimation are employed to reduce the band redundancy, achieving an effective spectral decorrelation. As shown by the experimental results, our method achieves remarkable bit-saving when compared to the state-of-the-art algorithms on MODIS image data set.

In this paper, an optimized Coding Tree Unit (CTU)-level rate control approach is proposed for low-delay high efficiency video coding (H.265/HEVC). Unlike the traditional explicitly estimated rate-distortion (R-D) model, the distributions of parameters of the estimated R-D model are considered. Accordingly, we formulate the CTU-level rate allocation for H.265/HEVC as the decision making problem, where the objective is to minimize the summation of CTUs' expected distortion under the given rate constraint. Moreover, a two-stage Bisection based method is proposed to solve the optimization problem, as the rate control algorithm for H.265/HEVC. The R-D performance improvements on Bjøntegaard delta bit rate (BD-BR) demonstrate the advances of the proposed method.

To promote the applications of semantic segmentation, quality evaluation is important to assess different algorithms and guide their development and optimization. In this paper, we establish a subjective semantic segmentation quality assessment database based on the stimulus-comparison method. Given that the database reflects the relative quality of semantic segmentation result pairs, we adopt a robust regression mapping model to explore the relationship between subjective assessment and objective distance. With the help of the regression model, we can examine whether objective metrics coincide with subjective judgement. In addition, we propose a novel relative quality prediction network (RQPN) based on Siamese CNN as a new objective metric. The metric is trained by our subjective assessment database and can be applied to evaluate the performances of semantic segmentation algorithms, even if the algorithms were not used to build the database. Experiments are conducted to show the advance and the reliability of our database and demonstrate that results predicted by RQPN are more consistent to subjective assessment than existing objective metrics.

Inspired by the characteristics of the human visual system, a novel method is proposed for detecting the visually salient regions on 3D point clouds. First, the local distinctness of each point is evaluated based on the difference with its local surroundings. Then, the point cloud is decomposed into small clusters, and the initial global rarity value of each cluster is calculated; a random walk ranking method is then used to introduce cluster-level global rarity refinement to each point in all the clusters. Finally, an optimization framework is proposed to integrate both the local distinctness and the global rarity values to obtain the final saliency detection result of the point cloud. We compare the proposed method with several relevant algorithms and apply it to some computer graphics applications, such as interest point detection, viewpoint selection, and mesh simplification. The experimental results demonstrate the superior performance of the proposed method.

A photographer's intention towards a photo evokes the viewer's attentional response. By analyzing the photographer's intention, we can have a better understanding of what the photographers want to convey to the viewers, thus helping to improve image analyzing and understanding. In this paper, a novel method is presented to improve saliency detection based on exploring the relationship between the photographer's intention and the viewer's attention. An intention rate is derived to quantify the intention and integrated with traditional saliency detection in a unified framework accordingly. We evaluate the proposed scheme with several classic saliency detection algorithms on different datasets. The experimental results demonstrate the superior improvements of our method.

Objective quality assessment plays a crucial role in the evaluation and optimization processes of Virtual Reality (VR) technologies, for which state-of-the-art objective quality evaluation metrics for omnidirectional video, i.e., 360 degree video, are typically derived from traditional MSE (or PSNR). Here we propose an objective omnidirectional video quality assessment method based on structural similarity (SSIM) in the spherical domain. Adopting the relationship of the structural similarity between the 2-D plane and sphere, the interference brought by the projection between the two domains can be well handled in the assessment process. The performance of the proposed spherical structural similarity (S-SSIM) index is evaluated with a subjective omnidirectional video quality assessment database. As demonstrated in the experimental results, the proposed S-SSIM outperforms state-of-the-art objective quality assessment metrics in omnidirectional video quality assessment.

Region of Interests (ROI) coding has shown advantages in subjectively optimized video coding. In this paper, we propose a new CTU-Ievel rate allocation scheme based on the Stackelberg Game model to enhance the visual quality of ROI which formulates the rate allocation process as a noncooperative Stackelberg Game between the ROI and the non-ROI. In this game, ROI is the leader who takes the priority. Based on the formulated game, the rate allocation problem can be expressed as a utility optimization problem. By solving the corresponding utility optimization problem, a novel CTU-level rate allocation strategy is established, in which a trade-off between the ROI's quality and the overall quality can be achieved. The experimental results show that our proposed scheme can improve the quality of ROI significantly with a negligible overall quality degradation.

In this paper, a novel image captioning approach is proposed to describe the content of images. Inspired by the visual processing of our cognitive system, we propose a visual-semantic LSTM model to locate the attention objects with their low-level features in the visual cell, and then successively extract high-level semantic features in the semantic cell. In addition, a state perturbation term is introduced to the word sampling strategy in the REINFORCE based method to explore proper vocabularies in the training process. Experimental results on MS COCO and Flickr30K validate the effectiveness of our approach when compared to the state-of-the-art methods.

Achieving constant video quality is an essential optimization task for video coding applications. In this paper, we propose a two-pass rate control algorithm for High Efficiency Video Coding (HEVC) to reduce the fluctuation of video quality. First, we present a GOP-level rate allocation scheme which aims at minimizing distortion variance among GOPs. To derive an appropriate quantization parameter (QP) for each frame, we establish the distortion-quantizer (D-Q) and the rate-quantizer (R-Q) model based on ν-support vector regression. Further, an R-D tradeoff is designed to adjust the frame QP considering the constant quality control and rate contraints. Extensive experimental results show that the proposed algorithm achieves more constant video quality.

In-loop filtering is a key technique in the state of art video coding standard that plays a significant role in suppressing compression artifacts. Motivated by the latest advances of deep learning, in this paper, we design a dense residual convolutional neural network (DRN) based in-loop filter for High Efficiency Video Coding (HEVC). Taking advantage of both dense shortcuts and residual learning, DRN efficiently exploits the multi-level features to restore the high quality image from the degraded one. Bottleneck layers are employed in DRN in order to adaptively fuse the hierarchical features and saving the computational resources at the same time. Experimental results show that the proposed DRN based in-loop filter can further boost the coding performance, which provides 6.9% BD-rate reduction on average compared to the HEVC baseline. In addition, the proposed DRN outperforms previous CNN based in-loop filters.

Thermal cameras can capture images invariant to illumination conditions. However, thermal facial images are difficult to be recognized by human examiners. In this letter, an end-to-end framework, which consists of a generative network and a detector network, is proposed to translate thermal facial images into visible ones. The generative network aims at generating visible images given the thermal ones. The detector can locate important facial landmarks on visible faces and help the generative network to generate more realistic images that are easier to be recognized. As demonstrated in the experiments, the faces generated by our method have good visual quality and maintain identity preserving features.

Emerging virtual reality (VR) applications bring much challenge to video coding for 360-degree videos. To compress this kind of video, each picture should be projected to a 2D plane (e.g. equirctangular projection map) first, adapting to the input of existing video coding systems. At the display side, an inverse projection is performed before viewport rendering. However, such a project introduces much different levels of distortions depending on the location, which makes the rate-distortion optimization process in video coding much inefficient. In this paper, we consider the distortion in spherical domain and analyse its influence to the rate-distortion optimization process. Then we derive the optimal rate-distortion relationship in spherical domain and present its optimal solution based on HEVC/H.265. Experimental results show that the proposed method can bring up to 11.5% bit-saving compared with the current HEVC/H.265 anchor for 360-degree video coding.

Eye movement reflects the shift of overt visual attention. Eye movement trajectories from a group of observers can be expressed by a representative scanpath. The representative scan-path can work as a baseline for studies on scanpath prediction as well as provide useful knowledge about group behavior in psychological studies. In this paper, we propose a new framework to summarize a representative scanpath from individual scanpaths, taking into account the spatial distribution of scan-paths rather than simply treating them as strings of characters. It consists of three steps: extract areas of interest (AOI), remove outliers and summarize scanpaths. In the last step, we develop an algorithm termed Candidate-constrained DTW Barycenter Algorithm (CDBA) by imposing 3 constraints: (1) the components of the representative scanpath must be chosen from candidates (extracted AOIs); (2) the occurrence count of each AOI in the representative scanpath cannot exceed its maximum occurrence count in individual scanpaths; (3) any two contiguous AOIs in the representative scanpath must be contiguous in at least one individual scanpath. The experiments demonstrate that the proposed method outperforms other state-of-the-art scanpath mining methods.

360-degree video has been gaining its popularity as virtual reality (VR) technology develops in recent years. To compress these videos using standard video encoders, most of them are converted to a 2D image planar format with equirectangular projection (ERP). Compared with other encoders, High Efficiency Video Coding (HEVC) achieves significant improvements in coding efficiency. However, exhaustive computational complexity of HEVC makes it too time-consuming to compress 360-degree videos of high resolution and high frame rate for VR application. In this paper, a fast intra prediction algorithm is proposed according to the characteristics of ERP format video. Compared with original reference software HM 16.15, experimental results show that the proposed algorithm can achieve about 24.5% encoder time saving on average in All-Intra configuration with negligible quality loss.

In this paper, we propose a convolutional neural network (CNN)-based rate-distortion (R-D) modeling method for H.265/HEVC. A fully convolutional neural network (CNN) is designed to learn end-to-end, pixels-to-pixels mappings from the original images to the structural similarity (SSIM) maps indicating distortion. The rate information is predicted through a CNN with fully connected layers as well. When compared to traditional CNN methods, the proposed mappings to the distortion or rate information. The experiments demonstrate the feasibility of our CNN-based framework for rate-distortion modeling.

Achieving a stable video quality is an important task in video compression. In this paper, we present a multi-layer quantization control method to compress video to a certain target video quality based on the new hierarchical partition structure of H.265/HEVC. We first model the rate-quality characteristics of different video sequences using ν-support vector regression with a Gaussian radial basis function as the kernel function. Then, we propose a Kalman filter-based multi-layer quantization control method for the quality-constrained video coding. The advantage of our proposed approach in H.265/HEVC is demonstrated through experimental results. Compared with the other H.265/HEVC rate control algorithms and the fixed-QP scheme, the proposed method achieves a more stable video quality.

Visual attention identification is crucial to human visual perception analysis and relevant applications. In this paper, we propose a comprehensive visual attention identification algorithm consisting of clustering and center identification. In the clustering process, a spatial-temporal affinity propagation method for accurate fixation clustering is proposed. For the identified clusters, the random walk based method is utilized to extract the center of each cluster, which presents the essential part of an area of interest (AOI). The proposed approach addresses the problem of fixation overlapping in eye movement analytics. Compared with the state-of-the-arts methods, the proposed method shows superior performance for different eye tracking experiments.

Saliency detection is widely used to extract regions of interest in images for various image processing applications. Recently, many saliency detection models have been proposed for video in uncompressed (pixel) domain. However, video over Internet is always stored in compressed domains, such as MPEG2, H.264, and MPEG4 Visual. In this paper, we propose a novel video saliency detection model based on feature contrast in compressed domain. Four types of features including luminance, color, texture, and motion are extracted from the discrete cosine transform coefficients and motion vectors in video bitstream. The static saliency map of unpredicted frames (I frames) is calculated on the basis of luminance, color, and texture features, while the motion saliency map of predicted frames (P and B frames) is computed by motion feature. A new fusion method is designed to combine the static saliency and motion saliency maps to get the final saliency map for each video frame. Due to the directly derived features in compressed domain, the proposed model can predict the salient regions efficiently for video frames. Experimental results on a public database show superior performance of the proposed video saliency detection model in compressed domain.

In this paper, we propose a novel computational model of visual attention based on the relevant characteristics of the Human Visual System (HVS). The input image is firstly divided into small image patches. Then the sparse features for each image patch are extracted based on the learned sparse coding basis. The human visual acuity is adopted in the calculation of the center-surround feature differences for saliency detection. In addition, the neighboring image patches for computing the saliency value of each center image patch are selected based on the characteristics of HVS. Experimental results show that the proposed saliency detection algorithm outperforms other existing schemes tested with a large public image database.

Video enhancement plays an important role in various video applications. In this paper, we propose a new intra-and-inter-constraint-based video enhancement approach aiming to: 1) achieve high intraframe quality of the entire picture where multiple regions-of-interest (ROIs) can be adaptively and simultaneously enhanced, and 2) guarantee the interframe quality consistencies among video frames. We first analyze features from different ROIs and create a piecewise tone mapping curve for the entire frame such that the intraframe quality can be enhanced. We further introduce new interframe constraints to improve the temporal quality consistency. Experimental results show that the proposed algorithm obviously outperforms the state-of-the-art algorithms.

In this paper, a new heat-map-based algorithm is proposed for group activity recognition. The proposed algorithm first models human trajectories as series of heat sources and then applies a thermal diffusion process to create a heat map (HM) for representing the group activities. Based on this HM, a new key-point-based (KPB) method is used for handling the alignments among HMs with different scales and rotations. A surface-fitting (SF) method is also proposed for recognizing group activities. Our proposed HM feature can efficiently embed the temporal motion information of the group activities while the proposed KPB and SF methods can effectively utilize the characteristics of the HM for activity recognition. Section IV demonstrates the effectiveness of our proposed algorithms.

Saliency detection is widely used to extract the regions of interest in images. Many saliency detection models have been proposed for videos in the uncompressed domain. However, videos are always stored in the compressed domain such as MPEG2, H.264, MPEG4 Visual, etc. In this study, we propose a video saliency detection model based on feature contrast in the compressed domain. Four features of luminance, color, texture and motion are extracted from DCT coefficients and motion vectors in the video bitstream. The static saliency map of video frames is calculated based on the luminance, color and texture features, while the motion saliency map for video frames is computed by motion feature. The final saliency map for video frames is obtained through combining the static saliency map and motion saliency map. Experimental results show good performance of the proposed video saliency detection model in the compressed domain.

Saliency detection plays important roles in many image processing applications, such as regions of interest extraction and image resizing. Existing saliency detection models are built in the uncompressed domain. Since most images over Internet are typically stored in the compressed domain such as joint photographic experts group (JPEG), we propose a novel saliency detection model in the compressed domain in this paper. The intensity, color, and texture features of the image are extracted from discrete cosine transform (DCT) coefficients in the JPEG bit-stream. Saliency value of each DCT block is obtained based on the Hausdorff distance calculation and feature map fusion. Based on the proposed saliency detection model, we further design an adaptive image retargeting algorithm in the compressed domain. The proposed image retargeting algorithm utilizes multioperator operation comprised of the block-based seam carving and the image scaling to resize images. A new definition of texture homogeneity is given to determine the amount of removal block-based seams. Thanks to the directly derived accurate saliency information from the compressed domain, the proposed image retargeting algorithm effectively preserves the visually important regions for images, efficiently removes the less crucial regions, and therefore significantly outperforms the relevant state-of-the-art algorithms, as demonstrated with the in-depth analysis in the extensive experiments.

With the wide applications of saliency information in visual signal processing, many saliency detection methods have been proposed. However, some key characteristics of the human visual system (HVS) are still neglected in building these saliency detection models. In this paper, we propose a new saliency detection model based on the human visual sensitivity and the amplitude spectrum of quaternion Fourier transform (QFT). We use the amplitude spectrum of QFT to represent the color, intensity, and orientation distributions for image patches. The saliency value for each image patch is calculated by not only the differences between the QFT amplitude spectrum of this patch and other patches in the whole image, but also the visual impacts for these differences determined by the human visual sensitivity. The experiment results show that the proposed saliency detection model outperforms the state-of-the-art detection models. In addition, we apply our proposed model in the application of image retargeting and achieve better performance over the conventional algorithms.

In this paper, we propose a novel image retargeting algorithm to resize images based on the extracted saliency information from the compressed domain. Firstly, we utilize DCT coefficients in JPEG bitstream to perform saliency detection with the consideration of the human visual sensitivity. The obtained saliency information is used to determine the relative visual importance of each 8 x 8 block for the image. Furthermore, we propose a new adaptive block-level seam removal operation for connected blocks to resize the image. Thanks to the directly derived saliency information from the compressed domain, the proposed image retargeting algorithm effectively preserves the objects of attention, efficiently removes the less crucial regions, and therefore significantly outperforms the relevant state-of-the-art algorithms, as demonstrated with the careful analysis and in the extensive experiments.

In this paper, we propose a novel algorithm for improving and visualizing image search results. The proposed algorithm improves user's image search experience by three steps: (1) re-rank the initial image search results by the random walk refinement based on visual consistency and saliency cues, (2) project the re-ranked images into a 2-dimentional panel according to their saliency information and correlations, (3) detect and extract the saliency regions in each image for final visualization. To evaluate the performance of our algorithm, user study has been conducted. Experimental results demonstrate that our visualization algorithm provides more pleasing image search experience than the conventional image search methods.

In this paper, we propose a novel image retargeting algorithm based on the sensitivity-tuned visual significance map which is composed of a saliency map and a gradient map. We develop a new saliency detection model based on the human visual sensitivity and amplitude spectrum of image patches. We use a coherent normalization based fusion method to combine the saliency map and the gradient map to generate the visual significance map. The seam carving technique is adopted for image retargeting, based on the sensitivity-tuned visual significance map. Experiment results show that the proposed algorithm outperforms the relevant state-of-the-arts image retargeting algorithms significantly.

Video enhancement plays an important role in various video applications. It is desirable to achieve high visual quality of the entire picture where multiple region-of-interests (ROIs) within the frame can be adaptively and simultaneously enhanced. In this paper, a new global-based video enhancement algorithm is proposed. The proposed algorithm first analyzes features from different ROIs. Then, a 'global' tone mapping curve is created for the entire picture which can adaptively enhance different regions at the same time. According to the statistics of ROIs, two fusion strategies, i.e., piecewise-based and factor-based fusions, are proposed for creating the global tone mapping curve. Experimental results show that the proposed algorithm can obtain more appealing perceptual quality than the state-of-the-art algorithms.

Motion estimation (ME) is one of the most time-consuming parts in video coding. The use of multiple partition sizes in H.264/AVC makes it even more complicated when compared to ME in conventional video coding standards. It is important to develop fast and effective sub-pixel ME algorithms since: 1) the computation overhead by sub-pixel ME has become relatively significant while the complexity of integer-pixel search has been greatly reduced by fast algorithms, and 2) reducing sub-pixel search points can greatly save the computation for sub-pixel interpolation. In this letter, a novel fast sub-pixel ME algorithm is proposed which performs a “rough” sub-pixel search before the partition selection, and performs a “precise” sub-pixel search for the best partition. By reducing the searching load for the large number of non-best partitions, the computation complexity for sub-pixel search can be greatly decreased. Experimental results show that our method can reduce the sub-pixel search points by more than 50% compared to existing fast sub-pixel ME methods with negligible quality degradation.

Investigation on the human perception can play an important role in video signal processing. Recently, there has been great interest in incorporating the human perception in video coding systems to enhance the perceptual quality of the represented visual signal. However, the limited understanding of the human visual system and high complexity of computational models of human visual system make it a challenging task. Furthermore, the hybrid video coding structure brings difficulties to integrate computational models with coding components to fulfill the requirements. In this paper, we review the physiological characteristics of human perception and address the most relevant aspects to video coding applications. Moreover, we discuss the computational models and metrics which guide the design and implementation of the video coding system, as well as the recent advances in perceptual video coding. To introduce this overview with the latest technologies and most promising directions in perceptual video coding, we focus on three key areas. Specifically, we cover 1) visual attention and sensitivity modeling, with which we concentrate on the computational models of bottom-up and top-down attention, contrast sensitivity functions and masking effects, and fovea based manipulations; 2) perceptual quality optimization for constrained video coding, with which we discuss how to achieve maximum perceptual quality whilst satisfying various constraints; and 3) the impact of the human perception on advanced video applications, including emerging immersive multimedia services, and compression of high dynamic range video content and 3D video. For each aspect, we discuss the major challenges, highlight significant approaches, and outline future research directions.

In this paper, we propose a constant distortion rate control algorithm for H.264/AVC high definition videos with scene change. With the rate and distortion information of each frame, the frame scene complexity is modeled and the scene change is detected in the first pass encoding. In the second-pass encoding, the expected distortion of each GOP under the constraint of target bits is derived based on the first-pass statistics. And the quantization step (Q-step) of each frame can be obtained with a linear distortion-quantizer (D-Q) model by considering the parameter update of this model at scene change. The experimental results show that our proposed algorithm can provide more constant quality among different video scenes, compared to traditional joint model (JM) rate control algorithm.

In this paper, we propose a statistical learning based approach to analyze the rate-distortion characteristics of multiscale binary shape coding. We employ the polynomial kernel function and incorporate rate-distortion related features for our support vector regression. ε-Insensitive loss function is chosen to improve the estimation robustness. The parameter tuning is also studied. Moreover, we discuss the feature selection which helps to improve the estimation accuracy. Comparing to the traditional method, our proposed framework provides better rate distortion estimation not only on simple shapes but also on complex shapes.

In this paper, we address the key problem in real-time video coding, the rate-distortion (R-D) tradeoff. As most video coding applications employ color images, we analyze the R-D modeling problem of the color image sequence. We investigate that the R-D characteristics of color video signal in traditional transform-based video coding systems should be modeled for the luminance and chrominance components separately. So we propose separable R-D models for color video coding. The feedback from the encoder buffer is analyzed by a control-theoretic adaptation approach to avoid buffer overflow and underflow. To achieve smooth video quality and satisfy the delay constraints in real-time applications, a novel R-D tradeoff controller is designed. In the proposed R-D tradeoff framework, both the quality variation and buffer safety are considered. Extensive simulation results demonstrate that the proposed approach can achieve smooth video quality without sacrificing overall quality while efficiently utilize the bandwidth without buffer overflow or underflow

This paper describes a unified framework of bit-rate control for binary and gray-level shape coding. An exponential rate-distortion model is proposed for binary shape and used in the bit-rate control. An approximation method is applied for finer distortion scale adjustment by adjusting the conversion ratio at the binary alpha block level. Since the gray-level shape is encoded as its support information and the alpha plane information, the rate-quantizer relationship of the alpha plane is modeled. The bit-rate control is then able to adjust the quantizer of the alpha plane for the bit-rate control purpose. The extension for video object (VO) coding is also studied. In this bit-rate control approach, a buffer feedback controller is employed to determine the target bit rates for different sources, i.e., binary shape, gray-level shape, and texture. Accordingly, the distortion scales of different sources are adjusted based on their rate-distortion relationships to maintain a stable encoder buffer. The advantages of the proposed method are demonstrated through various tests for different scenarios. Overall, the algorithm can efficiently encode the shape information and shows a good performance for VO coding.

In this paper, we propose a statistical learning-based approach to analyze the rate-distortion characteristics of MPEG-4 multiscale binary shape coding. We employ the polynomial kernel function and epsiv-insensitive loss function for our support vector regression. To improve the accuracy of the estimation, rate and distortion related features are incorporated in the statistical learning framework. Our experimental results show that the proposed approach can achieve good performance, e.g., modelling the rate-distortion curves accurately.

In this paper, we address to the key problem in real-time video coding, the rate-distortion tradeoff. We found that the distribution of the integer transform coefficients in H.264 color video coding can be approximated by Laplacian distribution but the rate-distortion characteristics should be modelled for the luminance and chrominance components separately. So we propose separable rate-distortion models for color video coding. The feedback from the encoder buffer is analyzed by a control-theoretic adaptation approach. To achieve smooth video quality and satisfy the delay constraints in real-time applications, a novel rate distortion tradeoff controller is employed. Simulations results demonstrate that the proposed approach can achieve smooth video quality without sacrificing overall quality while efficiently utilize the bandwidth without buffer overflow or underflow.

In MPEG-4, a visual scene may be treated as a composition of video objects and coded at object level. Such a flexible video coding framework makes it possible to code different video objects with different priority according to human perceptual characteristics. In this paper, we introduce a novel dynamic bit allocation framework to improve the subjective quality in such an object-based video coding system. We incorporate the rate distortion models with the dynamic priorities of the video objects and jointly encode video objects to minimize the weighted distortion within the bit budget constraint. We guarantee the human-interested video objects a better reconstructed quality by using the weighted bit allocation strategy in favour of the video objects with higher priority. To obtain the priority automatically, we apply a visual attention model. Comparing with traditional bit allocation algorithms, the objective quality of the object with higher priority is significantly improved under this framework. These results demonstrate the usefulness of this dynamic bit allocation framework.

In this paper, we present a uniform framework for optimal multiple video object bit allocation in MPEG-4. We combine the rate-distortion (R-D) models for the texture and shape information of arbitrarily shaped video objects to develop the joint texture-shape R-D models. The dynamic programming technique is applied to optimize the bit allocation for the multiple video objects. The simulation results demonstrate that the proposed joint texture-shape optimization algorithm outperforms the MPEG-4 verification model on the decoded picture quality.

In this paper, we present an optimal bit allocation algorithm for multiple video objects (MVO's) in MPEG-4 video coding. We combine the rate-distortion models for texture and shape information to achieve the accurate rate control. To optimize the bit allocation for multiple video objects, we use the dynamic programming (DP). The simulation results demonstrate that the proposed algorithm outperforms the MPEG-4 verification model not only in terms of buffer stability but the objective quality.

In this letter, we explore the rate-distortion (R-D) characteristics of binary shape information in the MPEG-4 standard. The shape coding scheme is a block-based context-based arithmetic encoding approach and distortion is introduced by down and up sampling. Generally, the shape bit rate and nonnormalized distortion increase in proportion with the number of border blocks. At any given resolution scale, the more complex an object, the more distortion that is introduced. Consequently, we propose an R-D model based on the parameters derived from the border blocks and a block-based shape complexity for the video object. The computational complexity is much lower than other existing methods. Experimental results show that it can accurately predict the bit rate and distortion of the binary shape for rate control purposes.

Rate control plays a central role in constant bit-rate (CBR) video coding applications using MPEG-4. The paper considers single video object (SVO) rate control for MPEG-4 and presents a new rate control algorithm based on the quadratic rate-distortion model. Based on a new measure for the encoding complexity, the constraints of the model parameter estimation and a novel quantizer control strategy are proposed. Simulation results show that the MPEG-4 coder, using the proposed algorithm, can achieve a higher PSNR than a coder using the conventional rate control algorithm.

Rate control plays a central role in constant bit-rate video coding applications using MPEG-4. This paper considers single-video-object rate control for MPEG-4 and presents a new rate-control algorithm based on the quadratic rate-distortion (R-D) model. The major innovations are a novel constraint for the least-mean-square estimation of the model parameters of the R-D function, a new measure for the encoding complexity, novel quantizer control and an efficient frame skipping strategy. Simulation results show that the MPEG-4 coder, using the proposed algorithm, can achieve a higher PSNR than a coder using the conventional rate-control algorithm.