Existing stereo image quality assessment (SIQA) methods generally have limitations in binocular fusion and fine-grained perception modeling. To address these issues, we propose a Perception-Inspired Network for SIQA that simulates binocular difference-guided fusion, high-frequency sensitivity, and hierarchical perception mechanisms of the human visual system (HVS). First, a difference-guided binocular fusion (DGBF) module is designed to mimic the binocular difference sensitivity mechanism, which exploits difference information at both the feature-level and image-level to optimize binocular fusion. Furthermore, the image distortion primarily affects the high-frequency components, which are critical for perceptual quality. To reflect this, we propose a high-frequency enhancement module (HFEM) to simulate the human eye's sensitivity to edge and texture distortions. Finally, to better achieve fine-grained perception modeling, we propose a hierarchical quality regression strategy that simulates the human perceptual process, from perceiving local details to forming a global quality judgment, thereby achieving a quality prediction more aligned with human subjective evaluation. Experimental results demonstrate that the proposed method outperforms mainstream approaches, achieving a PLCC of 0.9734 on the LIVE I database, and a PLCC of 0.9632 on the LIVE II database.

In recent years, stereo image super-resolution based on convolutional neural network has been extensively researched and achieved impressive performance by introducing complementary information from another view. However, most existing methods still cannot fully capture both intra- and cross-view information due to the neglect of multi-scale information perception, multi-scale binocular alignment and the excitation of large scale to small scale in human vision system. And they generated blurry results due to the consideration of irrelevant information in search for cross-view information. To address these issues, we propose a multi-scale visual perception based progressive feature interaction network (MS-PFINet) for stereo image super-resolution. Specifically, to exploit comprehensive intra- and cross-view information for image reconstruction, we design a two-stream network with multi-branch structure to extract multi-scale features and progressively use cross-view interaction at larger scales to guide that at smaller scales. Moreover, to explore more proper and accurate cross-view information, we propose a feature transformer module (FTM) to search and transfer the most relevant features from another view by hard attention maps and soft attention maps, which are calculated by patch-wise similarity rather than pixel-wise. In addition, in order to encourage a more effective way to transfer texture features for the target view, we propose a perceptual texture matching loss to supervise the accuracy of feature transformer modules. Experimental results show that our proposed method is superior to the state-of-the-art methods in most cases.

Recently, parallax attention based stereo image super-resolution (SR) methods, which can better explore cross-view information, have been widely studied. Despite the impressive performance of these methods, almost all of them calculate parallax attention maps at a single low resolution, which will lead to ambiguous stereo correspondence. Besides, the widely used parallax attention module (PAM) cannot handle the illuminance variations in stereo image pairs, and cannot distinguish the contribution of the captured cross-view features to the reconstruction of the target view. To this end, in this paper, we propose a coarse-to-fine cross-view interaction based network (C2FNet) to achieve more accurate cross-view information capturing. Firstly, in C2FNet, a coarse-to-fine cascaded parallax attention structure (C2F-CPAS), which conforms with the human visual mechanism, is constructed to gradually perform parallax attention from the low-resolution to high-resolution level. Thus, richer textures can be used to learn more reliable stereo correspondence. Meanwhile, a multi-level attention transfer loss is designed to further calibrate the accuracy of stereo correspondence at each level. Secondly, we propose a modified PAM (MPAM) to alleviate the limitations of common PAM so that illuminance-robust stereo correspondence can be learned and more important cross-view information can be selected. Extensive experimental results show that our proposed C2FNet outperforms the state-of-the-art methods on various datasets.

Considering that the human brain always follows a coarse-to-fine (low-to-high spatial frequency) visual processing and fusion mechanism, we propose a coarse-to-fine feedback guidance based stereo image quality assessment (SIQA) network which considers a coarse-to-fine feedback guidance and adaptive dominant eye mechanism. The proposed network consists of two main sub-network streams, each of which has three branches to extract low, middle and high spatial frequency information in parallel. To better realize the guidance of the high-level features in the low spatial frequency branch to the low-level features in the high spatial frequency branch, an information feedback guidance module (IFGM) is proposed, which realizes a top-down guidance mechanism in each sub-network stream. Simultaneously, according to the theory of ocular dominance in human visual system (HVS), we design an adaptive bi-directional parallax-based binocular fusion module (BPBFM), which synthesizes two types of fusion feature by taking the left and right view features as dominant eye input. Furthermore, in order to obtain the better perceptual quality of stereo images, we design a weighted fusion strategy to weigh the quality scores from the two types of fusion features obtained by using an ensemble model with two multi-layer perceptrons (MLPs). The experimental results on four public stereo image datasets show that the proposed method is superior to the mainstream metrics and achieves an excellent performance.

With the huge development of stereoscopic video technology, the research of stereoscopic video quality assessment (SVQA) has become very important for promoting the development of stereoscopic video system. These years, many SVQA methods based on convolutional neural network (CNN) have emerged. In this paper, we proposed a multi-scale feature-guided 3D convolutional neural network for SVQA which not only use 3D convolution to capture spatio-temporal features but also aggregate multi-scale information by a new multi-scale unit. Besides, we employ a multi-stage growing attention mechanism in this network to learn more critical deep semantic information. The proposed method is tested on two public stereoscopic video quality datasets, and the result shows that this method correlates highly with human visual perception and outperforms state-of-the-art methods by a large margin.

In recent years, single image super-resolution based on convolution neural network (CNN) has been extensively researched. However, most CNN-based methods only focus on mining features at a single resolution, which will cause the loss of some useful information. Besides, most of them still have difficulty in training and obtaining high-quality images for large scale factors. To address these issues, we propose a multi-resolution attention network (MRAN), which progressively reconstructs images at large scale factors by aggregating features from multiple resolutions. Specially, a multi-resolution residual block (MRRB) is designed as basic block to specialize features at different resolutions and share information across different resolutions, improving the representation ability of features. Simultaneously, we design a resolution-wise attention block (RAB) to evaluate the importance of features from different resolutions, making the use of features more effective and enhancing feature fusion. Experimental results show that our proposed method is superior to the state-of-the-art methods.

Stereoscopic image quality assessment (SIQA) is to predict the human perception quality of stereoscopic image pairs, which is more challenging than previous 2D image quality assessment due to the complicated binocular vision mechanism in the human visual system (HVS). Recently witnessed the significant progress of biotechnology and motivated by the deeper research on the HVS, we take a step to bridge the gap between HVS and SIQA by generalizing the optic chiasm algorithm and introducing biological vision fusion mechanism in our work. Firstly, a network structure is proposed in our work, which consists of an optic chiasm module for binocular information exchange and a multi-scale feature extraction module for binocular information fusion. Secondly, a mutual-perception attention fusion module is designed for simulating binocular fusion. In addition, we come up with an innovative data enhancement method. Experimental results show that the image quality assessment score obtained by the network is more consistent with human perception.

To simulate the characteristics of perceiving things from binocular vision, a dual-pathway convolutional neural network (CNN) for quality assessment of screen content images (SCIs) is proposed. Considering the different sensitivity of retinal photoreceptor cells to RGB colors and the human visual attention mechanism, we employ a convolutional block attention module (CBAM) to weight the RGB channels and their spatial position on each channel. And 3D convolution considering inter-frame information is used to extract the correlation features between RGB channels. Moreover, because of the important role of optic chiasm in binocular vision, we design its simulation strategy in the proposed network. Furthermore, since the characteristics of multi-scale and multi-level are indispensable to perception of any objects in human visual system (HVS), a new multi-scale and multi-level feature fusion (MSMLFF) module is built to obtain perceptual features of different scales and levels. Experimental results show that the proposed method is superior to several mainstream SCIs metrics on publicly accessible databases.

In recent years, with the popularization of 3D technology, stereoscopic image quality assessment (SIQA) has attracted extensive attention. In this paper, we propose a two-stage binocular fusion network for SIQA, which takes binocular fusion, binocular rivalry and binocular suppression into account to imitate the complex binocular visual mechanism in the human brain. Besides, to extract spatial saliency features of the left view, the right view, and the fusion view, saliency generating layers (SGLs) are applied in the network. The SGL apply multi-scale dilated convolution to emphasize essential spatial information of the input features. Experimental results on four public stereoscopic image databases demonstrate that the proposed method outperforms the state-of-the-art SIQA methods on both symmetrical and asymmetrical distortion stereoscopic images.

In this paper, we propose an optimized dual stream convolutional neural network (CNN) considering binocular disparity and fusion compensation for no-reference stereoscopic image quality assessment (SIQA). Different from previous methods, we extract both disparity and fusion features from multiple levels to simulate hierarchical processing of the stereoscopic images in human brain. Given that the ocular dominance plays an important role in quality evaluation, the fusion weights assignment module (FWAM) is proposed to assign weight to guide the fusion of the left and the right features respectively. Experimental results on four public stereoscopic image databases show that the proposed method is superior to the state-of-the-art SIQA methods on both symmetrical and asymmetrical distortion stereoscopic images.

Recently, scene text detection based on deep learning has progressed substantially. Nevertheless, most previous models with FPN are limited by the drawback of sample interpolation algorithms, which fail to generate high-quality up-sampled features. Accordingly, we propose an end-to-end trainable text detector to alleviate the above dilemma. Specifically, a Back Projection Enhanced Up-sampling (BPEU) block is proposed to alleviate the drawback of sample interpolation algorithms. It significantly enhances the quality of up-sampled features by employing back projection and detail compensation. Further-more, a Multi-Dimensional Attention (MDA) block is devised to learn different knowledge from spatial and channel dimensions, which intelligently selects features to generate more discriminative representations. Experimental results on three benchmarks, ICDAR2015, ICDAR2017- MLT and MSRA-TD500, demonstrate the effectiveness of our method.

The design of stereo image quality assessment (SIQA) methods cannot be well based on the biological theory of human vision, so the performance of many SIQA methods cannot achieve good consistency with the subjective perception. The research on the visual system tends to the dorsal and ventral pathways, which ignores the information asymmetry in the early visual pathways. It is worth noting that the ON and OFF receptive fields in retinal ganglion cells (RGCs) respond asymmetrically to the statistical features of images. Inspired by this, we propose a SIQA method based on monocular and binocular visual features, which takes into account the asymmetry of local contrast bright and dark features in early visual pathways. First, this paper extracts the response maps of ON and OFF cell in RGCs to left and right views respectively. And then the different information fusion modes of visual cortex are used to fuse the response maps information of left and right views. Final, monocular and binocular features were extracted and sent to support vector regression (SVR) for quality regression. Experimental results show that the proposed method is superior to several mainstream SIQA metrics on two publicly available databases.

In this paper, a three-channel convolutional neural network (CNN) constrained by multiple loss functions is designed for stereoscopic image quality assessment (SIQA). Given that both monocular and binocular information are crucial for SIQA, we take the patches of left images, right images and difference images as the inputs of the three channels respectively. Since using the ground truth as the labels of image patches cannot accurately characterize their quality, we propose to individually label each image patch to preserve the quality difference among different regions and views. Moreover, the multi-loss structure is adopted in the proposed method to consider both local features and global features simultaneously, which can constrain the feature learning from multiple perspectives. And the additional adaptive loss weights make the multi-loss network more flexible and universal. The experimental results show that the proposed method is superior to other existing SIQA methods with state-of-the-art performance.

In this paper, we proposed an optimized model based on the visual attention mechanism(VAM) for no-reference stereoscopic image quality assessment (SIQA). A CNN model is designed based on dual attention mechanism (DAM), which includes channel attention mechanism and spatial attention mechanism. The channel attention mechanism can give high weight to the features with large contribution to final quality, and small weight to features with low contribution. The spatial attention mechanism considers the inner region of a feature, and different areas are assigned different weights according to the importance of the region within the feature. In addition, data selection strategy is designed for CNN model. According to VAM, visual saliency is applied to guide data selection, and a certain proportion of saliency patches are employed to fine tune the network. The same operation is performed on the test set, which can remove data redundancy and improve algorithm performance. Experimental results on two public databases show that the proposed model is superior to the state-of-the-art SIQA methods. Cross-database validation shows high generalization ability and high effectiveness of our model.

In this paper, we propose a new two-column dense Convolutional Neural Network (CNN) for stereoscopic image quality assessment. The input of one column is the cyclopean image which conforms to the binocular combination and rival mechanism in our brain. The input of other column is the disparity map which provides some compensation information for the cyclopean image. More importantly, we employ the features of disparity map to guide and weight the feature maps obtained from the cyclopean image, which is implemented by modifying the structure of Squeeze and Excitation block. This weighting strategy recalibrates the importance of feature maps extracted from cyclopean image. At the end of CNN, we combine the outputs from the two-column through 'Concat', and then process them to get the final quality score of the stereoscopic image. Experimental results demonstrate that the proposed method can achieve high consistent alignment with subjective assessment.

Over the years, with the popularization of 3D technology, the demands of accurate and efficient 3D image quality evaluation (SIQA) methods are increasing constantly. Due to the wide application of CNN, CNN-based SIQA methods emerge one after another. However, current methods only consider a single scale or resolution, and some CNN-based methods directly take left and right views as an input of the network ignoring the visual fusion mechanism. In this work, a multi-scale no-reference SIQA method is proposed based on dilation convolution neural network (DCNN). Different from other CNN-based SIQA methods, the proposed one uses dilation convolution to imitate different scale of information processing fields in the human brain. Instead of left or right image, the cyclopean image generated by a new method is used as the input of the network. Moreover, the proposed multi-scale unit significantly can reduce computational parameters and computational complexity. Experimental results on two public databases show that the proposed model is superior to the state-of-the-art no-reference SIQA methods.

In this paper, we proposed an effective 3-D image-quality assessment method based on an adaptive cyclopean image by using ensemble learning. Our cyclopean image is not only suitable for a symmetrical distortion image, but also especially suitable for an asymmetrical distortion image. This adaptivity of our cyclopean image can be attributed to the consideration of gain control and gain enhancement in a binocular rivalry visual mechanism. In addition, we use a salient map to modify our cyclopean image to let the salient area of our cyclopean become more attractive. As a result, we can get better results. To remove redundant information out from our cyclopean, the sparse representation is applied to extract essential features. Finally, to get better regression accuracy on extracted feature, we use ensemble learning to get the final quality score of a stereoscopic image. The ensemble learner can improve the regression accuracy by 2% than a single learner. Experimental results show that the proposed algorithm outperforms the state-of-the-art methods on two publicly available stereoscopic image-quality assessment databases LIVE I and LIVE II.

3D image quality assessment confronts more difficulties than 2D image quality assessment. In this paper a 3D image quality assessment metric based on sparse representation was proposed. The contributions of the proposed method mainly include the following points: a color cyclopean image is used to better simulate the process of image processing in human brain, which is also very suitable for evaluating the quality of asymmetric distortion image. Meanwhile, for during sparse reconstruction some important information will be lost, we use the corresponding color cyclopean image to do compensation before feature extracting. And the paper creatively extracts spatial and spectral entropy feature of the distortion color cyclopean image and the corresponding reconstruction cyclopean image, respectively. Finally, we uses SVR to evaluate the quality of stereoscopic image. Experimental results show that the proposed method is very much in line with human visual perception.