Omnidirectional Video Quality Assessment (OVQA) is a challenging task due to the limited availability of adequate numbers of training samples for learning representations of distortions on omnidirectional videos. The recent masked autoencoder (MAE) has shown promising performance in learning local and global representations in a self-supervised way, and can be used to attempt to mitigate the difficulty of having insufficient annotated samples to adequately train omnidirectional video quality prediction models. But the reconstruction tasks that MAE models are designed for do not pertain to predicting diverse perceptual distortions, especially those relevant to the task of OVQA. We have attempted to overcome these limitations to harness and apply the power of the MAE concept to the OVQA problem. Towards this purpose, we create a Distortion-Sensitive Masked AutoEncoder (DS-MAE) that is able to represent perceptual distortions on omnidirectional videos. DS-MAE extracts viewports from omnidirectional videos and employs a masked autoencoding module (MAM) and a knowledge replay module (KRM) to learn representations on each viewport. In the MAM, distorted patches from omnidirectional videos are masked, by replacing them with undistorted counterparts. The autoencoder is trained to reconstruct the masked distortions, imbuing them with the ability to represent diverse video degradations. The KRM extracts and stores content representations, which are then “replayed” to mitigate potential catastrophic forgetting of content during training of the DS-MAE. Finally, a simple OVQA model is constructed using the pre-trained DS-MAE across all viewports. The new model, called OmniVQA, was tested on three public OVQA datasets. The experimental results show that OmniVQA delivers competitive performance against all compared models.

In recent years, deep learning-based methods have made significant progress on the image quality assessment problem; however, challenges remain arising from the lack of annotated, real-world training data and consequent poor generalization ability. Towards addressing these challenges, we propose a no-reference image quality assessment (NR-IQA) method based on generative AI (GenAI) images. Specifically, we use GenAI images as reference images, employing a cold diffusion model to generate distorted images of four different distortion types, and we label these distorted images using a full-reference model, thereby making it possible to construct a large-scale pre-training dataset. We use this resource generation method to facilitate NR-IQA model building. We deploy a Multi-scale Cross Attention Block (MCAB) and a Scale Simple Attention Module (SSAM) to enhance feature representation by extracting multi-scale feature information from both the channel and spatial dimensions that are predictive of image quality. Extensive experiments on eight public databases demonstrate that the proposed method achieves state-of-the-art (SOTA) performance. A public release of all the codes associated with this work will be made available on GitHub.

Most currently developed video quality assessment (VQA) algorithms have achieved excellent performance by using deep neural network (DNN). However, DNN is vulnerable to adversarial attacks , as an efficient surrogate for validating the model robustness, and there lack adversarial attack methods against VQA models. To this end, we propose a spatiotemporal attack network to generate adversarial examples for evaluating the robustness of VQA models that contains a spatial subnetwork and a temporal subnetwork . The proposed network, dubbed the Space-Time Quality Attack Network (STQA-Net 1 ), first computes the just noticeable difference (JND) maps of a video sequence as the input of the spatial subnetwork. The spatial subnetwork encodes the computed maps as spatial features and feeds the spatial features to the temporal subnetwork. Then, the spatial features are fused with the output of the temporal subnetwork and the fused features are decoded as attack weight maps. A visual constraint is used to control the visibility of perturbations and guide the generation of perturbation maps by multiplying JND maps with attack weight maps. Finally, the generated perturbation maps are added to the original video to form an adversarial example. Further, we also try to design a two-branch network to generate two opposite examples in a targeted attack scenario. The proposed attack methods against six state-of-the-art VQA algorithms are thoroughly tested on three VQA databases. The experimental results show that the proposed attack methods are very effective for testing the robustness of VQA models.

Omnidirectional images can be viewed through various projection formats. Different projection formats could offer different views, which may capture complementary information to boost feature representation effect. However, previous omnidirectional image quality assessment (OIQA) methods mostly focus on single projection format, the relationship between different projection contents is rarely explored. In this letter, we propose a knowledge distillation based OIQA (KD-OIQA) framework that improves quality feature representation capability of student network under the guidance of the quality feature representation of teacher network through different projection formats. Specially, we firstly train a teacher network with viewport images. Then, we distill the knowledge from teacher network into student network trained on the equirectangular projection (ERP) images for boosting the feature representation of student network. Based on recent advance regarding knowledge distillation by applying masks, we also design a masked distillation module to screen out effective information from teacher's features to achieve more efficient knowledge distillation effect. Finally, the student network extracts more comprehensive features from ERP images for quality prediction. Extensive experiments conducted on three OIQA databases demonstrate the effectiveness of the proposed framework.

Abstract With the development of mobile phone camera technology, mobile phones can take a large number of macro photography images that previously could only be taken by professional cameras. Therefore, it is of great significance to study the quality of macro photography images. For this reason, a macro photography image visual quality evaluation database is established and it is named as MP2020. The database contains 100 reference images and 800 distorted images of four distortion types, including 200 distorted images of JPEG 2000, 200 distorted images of JPEG, 200 distorted images of white noise, and 200 distorted images of Gaussian blur. The DMOS values in the database were calculated from 48000 data which are provided by 60 subjects. Ten classical image quality assessment algorithms were tested on the MP2020 database. The experimental results show that the existing image quality assessment algorithms, which are widely used, are not applicable to the macro photography images. Therefore, MP2020 would contribute to the improvement of existing algorithms and the development of new algorithms. MP2020 has been uploaded to GitHub for download.

The excessive emulation of the human visual system and the lack of connection between chromatic data and distortion have been the major bottlenecks in developing image quality assessment. To address this issue, we develop a new no-reference (NR) image quality assessment (IQA) metric that accounts for the impact of pre-attention and spatial dependency on the perceived quality of distorted images. The resulting model, dubbed the Pre-attention and Spatial-dependency driven Quality Assessment (PSQA) predictor, introduces the pre-attention theory to emulate early phase visual perception by refining luminance-channel data. Chromatic data are also processed concurrently by transforming images from RGB to the perceptually optimized SCIELAB color space. Considering that the gray-tone spatial dependency matrix conveys important texture properties that are closely related to visual quality, this matrix, as a mathematical solution for subsequent visual process emulation, is calculated along with its statistical features on both gray and color channels. To clarify the influence of different regression procedures on model output, support vector regression and AdaBoosting Back Propagation (BP) neural networks are adopted separately to train the prediction models. We thoroughly evaluated PSQA on four public image quality databases: LIVE, TID2013, CSIQ, and VCL. The experimental results show that PSQA delivers highly competitive performance compared with top-rank NR and full-reference IQA metrics.

In order to get better semantic annotation performance, block-global features are extracted as low-level visual features for image semantic annotation. Specifically, wellknown global feature extraction method, namely two-dimensional principal component analysis (2DPCA) is applied to extract the image block-global features. Unlike typical image annotation methods which use local features or global features separately, we propose to extract global features from image local regions (block) with the expectation of: a) combining the advantages of local and global features; b) discovering multiple semantic meanings in one image. In the experiment, comparative studies have been done for the performance of block-global feature extraction methods with widely used local feature extraction method such as scale invariant feature transform. The results show that 2DPCA has a significantly better performance than the performance of other methods.

Self-Organizing Maps (SOM) have presented excellent effect in color image segmentation; the scale of SOM will directly affect the accuracy of segmentation results. In this paper, we proposed a novel scale estimated of self-organizing map (SE-SOM) for color image segmentation based on SOM clustering. Different from conventional SOM model, it determines the number of nodes of competition layer by 3-D spatial distribution of pixels in HSV (Hue-Saturation-value) color space. Then sample pixels to train the map topology of the image and segment pixels by computing similarity between their feature vectors with weights of each node. Finally, design a connectivity filter to update labels of image to decrease noise. Statistical information are used to design map scale, which adapted the final SOM scale to the distribution feature of pixels, clustering results more accurate and stable, Experiments results show that the algorithm can produce ideal results with manual segmentation and suitable PNSR values.

Gradient vector flow (GVF) is an effective external force for active contours, but its isotropic nature handicaps its performance. The recently proposed NGVF model is anisotropic since it only keeps the diffusion along the normal direction of the isophotes; however, it is sensitive to noise and could erase weak boundaries. In this letter, the normally biased GVF (NBGVF) external force is proposed for snake models, which keeps the diffusion along the tangential direction of the isophotes and biases that along the normal direction. The biasing weight approaches zero at boundaries and is 1 in homogeneous regions. Consequently, the NBGVF snake can preserve weak edges and smooth out noise while maintaining other desirable properties of GVF and NGVF snakes such as enlarged capture range, insensitivity to initialization and convergence to u-shape concavity. These properties are evaluated on synthetic and real images.

A novel image fusion algorithm based on orthogonal matching pursuit (OMP) is proposed, which is named IFOMP algorithm and can be used for noise free or noise images. The matching pursuit algorithm provides an efficient image representation using an overcomplete dictionary. Firstly, the source images are decomposed into sub-images at different scales through Laplacian pyramid transform. The OMP method is exploited for capturing and fusing edges and texture features in the high frequency domain. Experimental results show that the proposed algorithm improves performance compared to traditional wavelet fusion with slightly increased computational complexity, and the fusion image has good robustness.

Multi-focus fusion is an important technique to integrate focal information from a set of input registered images. In this effort, a novel image fusion algorithm based on multi-scheme is presented. Given an over-complete Gabor dictionary, through matching pursuit signal decomposition algorithm, each source image can be described by sparse combinations of these atoms. Then the coefficients of the fused images are constructed according to different fuse rules. The last experiment results show that our algorithm can achieve better fusion effect than traditional wavelet image fusion method or spatial frequency method, whatever from subjective visual effect or objective metric.