Light field (LF) contains abundant spatial geometric information of the real-world scenes, and it can enhance the performance of the computer vision tasks. However, it is challenging to acquire LF images with high spatial resolution. So far, super-resolution (SR) techniques based on deep learning make insufficient use of frequency information, which limits the performance of LFSR. To address this issue, we propose a frequency-guided feature fusion network (i.e., LF-F \({}^{3}\) Net) for LFSR. To be specific, the proposed LF-F \({}^{3}\) Net is a dual-branch network. One branch employs the multi-dimensional frequency feature extraction (MFFE) module to capture frequency information from individual views, while the other branch further integrates the extracted frequency information with spatial features through the multi-dimensional spatial–frequency fusion (MSFF) module. Furthermore, we introduce a frequency loss to prevent the loss of critical frequency content during training, thereby maximizing the potential performance of network. The experimental results show that the LF-F \({}^{3}\) Net can significantly improve the SR performance and outperforms the state-of-the-art methods.

It is still challenging for perceptual quality assessment for the novel view synthesis methods, especially the Neural Radiance Fields (NeRF). Existing subjective quality assessment datasets lack coverage of large-scale, outward-facing scenes. In addition, they are limited to systematically assessing the quality of the scenes that involve multiple types of distortions and diverse viewing trajectories. To address these limitations, we introduce OF-NeRF, a subjective benchmark of perceptual quality assessment for outward-facing NeRF scenes. It contains 11 real-world scenes and systematically introduces three types of distortions: reconstruction artifacts, exposure variations, and gamma distortions. It also includes two viewing trajectories designed to simulate different navigation behaviors. We perform a subjective quality assessment experiment. Subsequently, we intensively analyze the impact of the distortions and view trajectories on subjective scores. Analytic results show that blurriness and artifacts dominate user perception in the scenario with ideal lighting. Local viewing trajectories significantly improve comfort. However, the Mean Opinion Score (MOS) is not ideal with the injection of serious rendering artifacts. It suggests that the overall MOS is jointly determined by view trajectories and the rendering artifacts. We also estimate the performance of the state-of-the-art no-reference quality metrics on the OF-NeRF. Experimental results demonstrate that these metrics are unable to reliably predict perceptual quality. The OF-NeRF will be a complementary benchmark for the development of NeRF-specific objective metrics.

Abstract To address issues such as object‐background confusion and difficulties in multi‐scale object feature extraction in underwater scenarios, this article proposes an underwater organisms detection algorithm based on multi‐scale perception and representation enhancement. The key innovation of the proposed algorithm is the perception improvement of the deep learning model for underwater multi‐scale objects. First, for underwater large‐scale objects, omni‐dimensional dynamic convolution is embedded as an attention mechanism (AM) into the deep network to improve the network's sensitivity to large‐scale underwater objects. For underwater small‐scale objects, an information retention downsampling module is designed to reduce the effects of serious information loss. Then, a contextual transformer as an AM is introduced into shallow networks to strengthen the network's ability to extract features from small objects. The second innovation of the proposed algorithm is an underwater spatial pooling pyramid module which enhances the representation ability of the model. Furthermore, a lightweight decoupled head is designed to eliminate the conflict between classification and localization. The ablation experiment on the URPC dataset shows that the proposed models are effective for underwater object detection. The comparative experiments on the URPC and DUT‐USEG datasets demonstrate that the proposed algorithm achieves an advantage in detection performance compared with the mainstream detection algorithms and underwater detection algorithms.

Abstract 360‐degree videos have drawn great attention from both the academia and the industry. For transmission and storage of 360‐degree video, the joint video exploration team proposed the Versatile Video Coding (VVC) standard. VVC can significantly reduce the bitrate while maintaining the same subjective visual quality compared to the preceding high efficiency video coding. However, the computational complexity of VVC is extremely high which hinders the interactive applications. This paper proposes a fast intra partition method and mode prediction algorithm for equirectangular projection 360‐degree video coding. First, a latitude‐based preprocessing is introduced to early terminate the Coding Unit (CU) partition in the polar region. Second, the support vector machine is used to predict the CU partition type. Third, the fast intra mode search method accelerates the intra mode prediction. Experimental results show that the proposed algorithm can significantly obtain an average time reduction rate of 60.40% and a Bjontegaard delta rate increase of 1.96%.

High Efficiency Video Coding (HEVC) can significantly improve the compression efficiency in comparison with the preceding H.264/Advanced Video Coding (AVC) but at the cost of extremely high computational complexity. Hence, it is challenging to realize live video applications on low-delay and power-constrained devices, such as the smart mobile devices. In this article, we propose an online learning-based multi-stage complexity control method for live video coding. The proposed method consists of three stages: multi-accuracy Coding Unit (CU) decision, multi-stage complexity allocation, and Coding Tree Unit (CTU) level complexity control. Consequently, the encoding complexity can be accurately controlled to correspond with the computing capability of the video-capable device by replacing the traditional brute-force search with the proposed algorithm, which properly determines the optimal CU size. Specifically, the multi-accuracy CU decision model is obtained by an online learning approach to accommodate the different characteristics of input videos. In addition, multi-stage complexity allocation is implemented to reasonably allocate the complexity budgets to each coding level. In order to achieve a good trade-off between complexity control and rate distortion (RD) performance, the CTU-level complexity control is proposed to select the optimal accuracy of the CU decision model. The experimental results show that the proposed algorithm can accurately control the coding complexity from 100% to 40%. Furthermore, the proposed algorithm outperforms the state-of-the-art algorithms in terms of both accuracy of complexity control and RD performance.

image can be represented with different formats, such as the equirectangular projection (ERP) image, viewport images or spherical image, for its different processing procedures and applications. Accordingly, the 360-degree image quality assessment (360-IQA) can be performed on these different formats. However, the performance of 360-IQA with the ERP image is not equivalent with those with the viewport images or spherical image due to the over-sampling and the resulted obvious geometric distortion of ERP image. This imbalance problem brings challenge to ERP image based applications, such as 360-degree image/video compression and assessment. In this paper, we propose a new blind 360-IQA framework to handle this imbalance problem. In the proposed framework, cubemap projection (CMP) with six inter-related faces is used to realize the omnidirectional viewing of 360-degree image. A multi-distortions visual attention quality dataset for 360-degree images is firstly established as the benchmark to analyze the performance of objective 360-IQA methods. Then, the perception-driven blind 360-IQA framework is proposed based on six cubemap faces of CMP for 360-degree image, in which human attention behavior is taken into account to improve the effectiveness of the proposed framework. The cubemap quality feature subset of CMP image is first obtained, and additionally, attention feature matrices and subsets are also calculated to describe the human visual behavior. Experimental results show that the proposed framework achieves superior performances compared with state-of-the-art IQA methods, and the cross dataset validation also verifies the effectiveness of the proposed framework. In addition, the proposed framework can also be combined with new quality feature extraction method to further improve the performance of 360-IQA. All of these demonstrate that the proposed framework is effective in 360-IQA and has a good potential for future applications.

During the process of watching 3D synthesised video (3D‐SV) and switching viewpoints, there is a case of asymmetric distortion, the left(right) viewpoint is a synthesised video generated by rendering technique, and the right(left) viewpoint is a real video taken by the camera. How to accurately estimate the quality of 3D‐SV with binocular asymmetric distortions is a new and challenging problem. Aiming at this problem, a blind quality assessment method for 3D‐SV with binocular asymmetric distortions is proposed. Firstly, the local edge deformations of synthesised videos at different scales are measured by calculating their standard deviations. Secondly, the global naturalness of synthesised videos is computed by analysing their natural statistical characteristics. Thirdly, a strategy for fusing left and right quality scores is proposed, which considers their texture information in different directions. Finally, the random forest is used to obtain an objective quality score. The experimental results show the superiority of the proposed method on asymmetry 3D‐SV database.

Stereo high dynamic range (HDR) image/video can be generated by using a pair of stereo cameras with different exposure parameters. This paper proposes a new stereo HDR imaging method using generative adversarial networks (GAN) with a low dynamic range (LDR) stereo imaging system. It is assumed here that the left-view (LV) image is under-exposed and the right-view (RV) image is overexposed. First, a view exposure transfer GAN (VET-GAN) is constructed to transfer exposure information of the RV image to the LV image to generate the multi-exposure LV images, and then an HDR fusion GAN is constructed to fuse the generated multi-exposure LV images into an LV HDR image. Similarly, an RV HDR image can be generated using the same way to form a stereo HDR image pair. The experimental results show that the proposed method can obtain stereo HDR images with high visual quality and effectively avoid the ghost artifacts caused by parallax.

In this paper, we propose an interpretable machine learning-based complexity control method for efficiently im-plementing HEVC on live video applications with different computing capacities and limited powers. Specifically, a complexity allocation method is designed to reasonably assign the complexity resources. Then, a multi-accuracy Coding Unit (CU) decision model is obtained by interpret-ably adjusting the parameters to efficiently and flexibly achieve a tradeoff between encoding complexity and rate distortion performance. Finally, a coding tree unit-level complexity control method is proposed to select appropri-ate accuracy of the CU decision model for making the en-coding complexity approach the target. The experimental results show that the proposed method outperforms state-of-the-art methods in terms of accuracy and encoding efficiency.

As a representation of three dimensional scenes, light field has received increasing attention. In light field image processing, reconstruction method plays an important role, which can not only produce dense views to improve the spatial and angular resolution of the light field, but also effectively reduce the transmission data. However, the reconstruction methods inevitably reduce the quality of light field images, so the corresponding light field image quality assessment is necessary. In this paper, a reconstruction distortion oriented light field image dataset is firstly established, with several different reconstruction methods and the corresponding subjective evaluation scores. Secondly, the subjective scoring results of source sequences and their types of distorted versions are compared and analyzed. Finally, the dataset is evaluated with the existing objective quality assessment metrics. Experimental results show that different reconstruction methods have different preferences on the input light field resolution, and the performance of the state-of-the-art objective quality metrics can be improved.

The full-reference image quality assessment (IQA) method are limited in practical applications. Here we propose a no-reference quality assessment method for high dynamic range (HDR) images based on tensor space. First, the tensor decomposition is used to generate three feature maps of an HDR image, considering color and structure information of the HDR image. Second, for a given HDR image, the corresponding multi -scale manifold structure features are extracted from the first feature map. For the second and third feature maps of the HDR image, multi-scale contrast features are extracted. Finally, the extracted features are aggregated by support vector regression to obtain the objective quality score of the HDR image. Experimental results show that the proposed method is superior to some representative full and no-reference methods, and even superior to the full-reference HDR IQA method, HDR-VDP-2.2, on the Nantes database. The proposed method has a higher consistency with human visual perception.

Perceptual quality assessment of stereoscopic images is a challenge in three‐dimensional video systems. Existing studies suggest that simply averaging the quality of left and right views can effectively predict the quality of symmetrically distorted stereoscopic images, but prediction deviation occurs in the case of asymmetrically distorted stereoscopic images. Most previous stereoscopic image quality assessment (SIQA) methods have been based only on the luminance component of the images; in addition, the basis of human visual perception is critical to image quality assessment and lies on the low‐dimensional manifold. Inspired by this, a new perceptual SIQA method is proposed, which includes two stages: training stage and quality prediction stage. In the training stage, the authors apply Tucker decomposition to RGB images to reduce dimensions along colour channels to produce training sets, and the projection matrix is obtained through manifold learning. In the quality prediction stage, considering the binocular visual characteristics of visual perception, the overall stereoscopic estimate depends on the monocular image quality via a local energy ratio based pooling strategy and cyclopean based binocular quality. Extensive experiments on three available benchmark databases demonstrate that the proposed metric has better performance and achieves highly consistent alignment with subjective assessment compared with state‐of‐the‐art SIQA metrics.

Tone-mapping operators are designed to apply high dynamic range (HDR) images on widely-used low dynamic range (LDR) devices. Developing well-performed tone-mapped image quality assessment (IQA) method is highly desired because traditional IQA method cannot be adopted in cross dynamic range quality measuring. To this end, we proposed a quality assessment method based on image exposure property. Specifically, an image exposure property determination model is utilized to segment HDR image into different exposure region. Then, quality features are extracted according to the distortion characteristics of each exposure region. Finally, the quality of tone-mapped image can be acquired by a trained regression model. Validation experiments on public database show that the proposed method can accurately predict the quality of tone-mapped image.

Existing perceptual researches on stereoscopic images mainly focus on the threshold of whole image distortion, rather than the effect of texture feature on the so-called threshold of just-noticeable distortion. Obviously, it is unreasonable to use a single unified perception threshold for natural stereoscopic images as the texture complexity typically varies in different blocks of natural images. To solve this problem, we generated an asymmetrically distorted stereoscopic image database with different texture densities and conducted a large number of subjective experiments. A strong correlation between the asymmetrical visibility threshold and texture complexity was revealed from the subjective experiments. Finally, a nonlinear fitting model was designed to uncover this relationship, which can be applied to asymmetrical coding to control the perceived quality of stereoscopic images.

In this paper, we propose a supervised dictionary learning framework for blind image quality assessment (BIQA) by using quality-constraint sparse coding. Different with the traditional dictionary learning framework which only ensures the learnt dictionary accounting for image features, we add a quality-related regularization term in the framework to learn a feature-related dictionary and a quality-related dictionary jointly. Specifically, the feature-related and quality-related dictionaries share the same sparse coefficients, so that the reconstruction errors form the image feature vectors and quality score vectors are both minimized. Once the feature-related and quality-related dictionaries are learned, given a testing sample, we first abstract its feature vector and then compute the corresponding sparse coefficients w.r.t. the learnt feature-related dictionary, its quality score can be directly reconstructed based on the learnt quality-related dictionary and the estimated sparse coefficients. Experiment results on three publicly available IQA databases show the promising performance of the proposed model.

As the popularity of three dimensional video, security of stereo image has become an evident issue to be solved. This paper presents a disparity based stereo image reversible data hiding by using histogram shifting, which can recover the original stereo image from marked stereo image without any distortion. Inter-correlations between left and right views of stereo image are utilized to predict pixels accurately. Then prediction error bins are constructed, and many points are around zero-valued bin for embedding data with low distortion of stereo images. The zero-valued bin is used twice to embed data, so that embedding capacity can reach more than 1 bit per pixel. Experimental results demonstrate that the proposed method outperforms the extended stereo image data hiding methods.

Stereoscopic image quality assessment (SIQA) is an important and challenging issue in three dimensional applications. In this letter, a perceptual modulated feature similarity (PMFS) metric for SIQA is proposed by considering the monocular and binocular perception properties. Specifically, stereoscopic image is first classified into monocular occlusion and binocular rivalry regions. Then, feature similarities between the original and distorted stereoscopic images are defined and measured for the monocular occlusion and binocular rivalry regions as the local monocular and binocular quality maps, respectively. Monocular and binocular just noticeable difference visual saliency models are presented to construct a modulation function to derive monocular and binocular quality scores. Finally, those scores are integrated into an overall quality score by support vector regression. Extensive experiments performed on LIVE phase II and MICT asymmetric databases demonstrate that the proposed PMFS metric can achieve much higher consistency with the subjective quality scores than some state-of-the-art SIQA metrics.

Three-dimensional (3D) video technologies are becoming increasingly popular, as they can provide high quality and immersive experience to end users, where depth maps are employed to generate the virtual views by depth-image-based rendering technique. However, how to reduce the compression and rendering complexities for depth maps while maintaining high rendering quality is still unresolved. In this study, a novel depth map compression and depth-aided view rendering method is proposed. In the proposed method, depth maps are represented with different layers and compressed with different macroblock-mode decision procedure, and several optimisation techniques, including spatio-temporal consistent warping, colour correction and temporal consistent hole filling are embedded into the view rendering framework. Experimental results show that compared with the traditional method, the proposed method can reduce more than 79% compression computational complexity and more than 45% rendering computational complexity, while maintaining high rendering quality.

Subjective quality evaluation is the basis of quality evaluation of stereoscopic images. As the lack of a public and diverse testing database currently, in this paper, a symmetric stereoscopic images database is built. And then the subjective quality of stereoscopic images is analyzed from two aspects, one is the effects of JPEG, JPEG2000, H.264. The other is the comparisons between symmetric and asymmetric stereoscopic images from Gaussian blurring, white Gaussian noise, JPEG and JPEG2000, respectively. The results show three compressions are quite different in the subjective quality of symmetric stereoscopic images at different bitrates, and the comparisons between symmetric and asymmetric stereoscopic images investigate the properties of binocular fusion, binocular suppression, and binocular summation.

A fast macroblock mode selection algorithm based on dynamic multi-threshold is proposed to improve the encoding speed of multiview video, but with insignificant degradation in rate distortion (RD) performance. The macroblock modes are divided into four classes after statistically analyzing the macroblock mode selection results. Three thresholds are adopted based on the great RD cost gaps between the macroblock mode classes. An approximate computing method and a dynamic updating method of the three thresholds are proposed for implementing the fast algorithm. Simulation results demonstrate that the proposed fast algorithm promotes the encoding speed by 1.92~7.07 times in comparison with JMVM, while the algorithm hardly influences the RD performance.