Point cloud is one of the most widely used digital representation formats for three-dimensional (3D) contents, the visual quality of which may suffer from noise and geometric shift distortions during the production procedure as well as compression and downsampling distortions during the transmission process. To tackle the challenge of point cloud quality assessment (PCQA), many PCQA methods have been proposed to evaluate the visual quality levels of point clouds by assessing the rendered static 2D projections. Although such projectionbased PCQA methods achieve competitive performance with the assistance of mature image quality assessment (IQA) methods, they neglect that the 3D model is also perceived in a dynamic viewing manner, where the viewpoint is continually changed according to the feedback of the rendering device. Therefore, in this paper, we evaluate the point clouds from moving camera videos and explore the way of dealing with PCQA tasks via using video quality assessment (VQA) methods. First, we generate the captured videos by rotating the camera around the point clouds through several circular pathways. Then we extract both spatial and temporal quality-aware features from the selected key frames and the video clips through using trainable 2D-CNN and pretrained 3D-CNN models respectively. Finally, the visual quality of point clouds is represented by the video quality values. The experimental results reveal that the proposed method is effective for predicting the visual quality levels of the point clouds and even competitive with full-reference (FR) PCQA methods. The ablation studies further verify the rationality of the proposed framework and confirm the contributions made by the qualityaware features extracted via the dynamic viewing manner. The code is available athttps://github.com/zzc-1998/VQA_PC.

User-generated content (UGC) live videos are often bothered by various distortions during capture procedures and thus exhibit diverse visual qualities. Such source videos are further compressed and transcoded by media server providers before being distributed to end-users. Because of the flourishing of UGC live videos, effective video quality assessment (VQA) tools are needed to monitor and perceptually optimize live streaming videos in the distributing process. In this paper, we address UGC Live VQA problems by constructing a first-of-a-kind subjective UGC Live VQA database and developing an effective evaluation tool. Concretely, 418 source UGC videos are collected in real live streaming scenarios and 3,762 compressed ones at different bit rates are generated for the subsequent subjective VQA experiments. Based on the built database, we develop a Multi-12imensional VQA (MD-VQA) evaluator to measure the visual quality of UGC live videos from semantic, distortion, and motion aspects respectively. Extensive experimental results show that MD-VQA achieves state-of-the-art performance on both our UGC Live VQA database and existing compressed UGC VQA databases.

Recently, increasing interest has been drawn in Transformer-based models for No-reference Image Quality Assessment (NR-IQA), especially for the hybrid approach. The hybrid approach tend to apply Transformer to aggregate quality information from feature maps extracted by Convolutional Neural Networks (CNN). However, existing methods cannot fully utilize the information of hierarchical features extracted by the deep neural network, resulting in the limited performance of image quality evaluation. In this work, we propose a novel Hierarchical Feature Fusion Transformer for NR-IQA (HiFFTiq), which is able to effectively exploit complementary strengths of features extracted by different layers. Further, we propose a new Uniform Partition Pooling (UPP) which can reduce the resolution of input features via uniform partitions and can well retain the quality-related information compared to the traditional pooling method Sliding Window Pooling (SWP). The results of experiment demonstrate that HiFFTiq leads to improvements of performance over the state-of-the-art methods on three large scale NR-IQA datasets.

Mobile games have played an increasingly significant role in people's leisure lives in recent years, thanks to the fast expansion of the gaming industry and the widespread use of mobile devices. The aesthetic quality of game pictures is a very important factor that attracts users' interest. However, evaluating the aesthetic quality of mobile game pictures is difficult since the painting styles of games vary greatly and the evaluation criteria are also diversified. In this article, we propose a multitask deep learning-based method, which is able to predict the aesthetic quality of mobile game images in multiple dimensions. The proposed model consists of two modules, a feature extraction module and a quality regression module. We extract quality-aware features from intermediate layers of the deep convolution neural network and then incorporate them into the final feature representation in the feature extraction module, allowing the model to fully use visual information from low to high levels. The quality regression module uses fully connected layers to map quality-aware features into quality scores across multiple dimensions. The multidimensional aesthetic quality scores are trained using a multitask learning approach, in which quality-aware features are shared across multiple dimensional quality prediction tasks. Finally, several key factors which help the proposed model perform better are analyzed. The experimental results indicate that our proposed method not only achieves the greatest performance on mobile game images, but also is applicable to natural scene images.

Summary Human behavior would lead to a significant impact on the environment. By monitoring the environment, we can indirectly monitor human behavior. Remote sensing (RS) technology provides a large number of multispectral (MS) images. When combining the Internet of things (IoT) technology, those images can be used for human behavioral monitoring. However, due to the limitation of the optical sensors embedded in satellites, the spatial resolution of MS image is relatively low, which poses a huge problem for further understanding these images. Pansharpening, also known as multisensor image fusion, aims to sharp an MS image to a high‐resolution multisensor image (HMS) by integrating a corresponding high‐resolution panchromatic (PAN) image. By doing so, the redundancy among big data can be effectively reduced. Traditional Intensity‐Hue‐Saturation (IHS)–based methods often suffer from spectral distortion. To address this problem, a novel pansharpening method is proposed in this paper. Different from those traditional IHS methods, the proposed method first decomposes MS and PAN into high‐frequency‐component (HFC) and low‐frequency‐component (LFC), respectively. Then, the guided filter (GF) is utilized to enhance the spectral information on the detail map. Furthermore, the detail map is refined according to the adaptive coefficients for each band of MS. By performing experiments, we demonstrate the proposed method can obtain satisfying results in both visual quality and object assessment among existing methods.

Summary Doctors always desire high‐resolution medical images to have accurate diagnosis. Super‐resolution (SR) is a technology that can improve the resolution of medical images. Convolutional neural network (CNN)–based SR methods have achieved desired performance in natural images. In this paper, we apply a deep dense SR (DDSR) convolutional neural networks model to two types of medical images, including Computerized Tomography (CT) images and Magnetic Resonance imaging (MRI) images. This network densely connects every hidden layer to learn high‐level features, which was first proposed for object recognition. A set of medical images is used for experiments. We compare the performance of DDSR with three state‐of‐the‐art SR network models, including SR Convolutional Neural Network (SRCNN), Fast SR Convolutional Neural Network (FSRCNN), and Very Deep SR Convolutional Neural Network (VDSR). Both the objective indices and subjective evaluations are used for comparison. The results show that the proposed network has better performances both on CT and MRI images.

Recent advances in image super-resolution (SR) explored the power of deep learning to achieve a better reconstruction performance. However, the feedback mechanism, which commonly exists in human visual system, has not been fully exploited in existing deep learning based image SR methods. In this paper, we propose an image super-resolution feedback network (SRFBN) to refine low-level representations with high-level information. Specifically, we use hidden states in a recurrent neural network (RNN) with constraints to achieve such feedback manner. A feedback block is designed to handle the feedback connections and to generate powerful high-level representations. The proposed SRFBN comes with a strong early reconstruction ability and can create the final high-resolution image step by step. In addition, we introduce a curriculum learning strategy to make the network well suitable for more complicated tasks, where the low-resolution images are corrupted by multiple types of degradation. Extensive experimental results demonstrate the superiority of the proposed SRFBN in comparison with the state-of-the-art methods. Code is avaliable at https://github.com/Paper99/SRFBN_CVPR19.

Multisensor image fusion or pansharpening aims to sharpen a multispectral (MS) image by integrating the detail map derived from a panchromatic (Pan) image. The intensity-hue-saturation (IHS)-based methods are well adopted in pansharpening applications. However, the pansharpened MS images by IHS-based methods usually suffer from serious spectral distortions and local artifacts due to the mismatch between the estimated detail map and its ground truth. To overcome these defects, we propose a weighted least squares (WLS)-filter-based method in this letter. Different from existing IHS-based methods, the proposed method eliminates the influence of the low-frequency components of the Pan and MS images with the WLS filter. Moreover, the derived detail map is further refined based on the spectral signatures for different bands of the MS image. We test the proposed method on various satellites data; the experimental results demonstrate that the proposed method performs well in both spectral and spatial qualities.

Demosaicking is a digital image process to reconstruct full color digital images from incomplete color samples from an image sensor. It is an unavoidable process for many devices incorporating camera sensor (e.g., mobile phones, tablet, and so on). In this paper, we introduce a new demosaicking algorithm based on polynomial interpolation-based demosaicking. Our method makes three contributions: calculation of error predictors, edge classification based on color differences, and a refinement stage using a weighted sum strategy. Our new predictors are generated on the basis of on the polynomial interpolation, and can be used as a sound alternative to other predictors obtained by bilinear or Laplacian interpolation. In this paper, we show how our predictors can be combined according to the proposed edge classifier. After populating three color channels, a refinement stage is applied to enhance the image quality and reduce demosaicking artifacts. Our experimental results show that the proposed method substantially improves over the existing demosaicking methods in terms of objective performance (CPSNR, S-CIELAB ΔE*, and FSIM), and visual performance.

Traditional methods for automatic recognition of container code in visual images are based on segmentation and recognition of isolated characters. However, when the segment fails to separate each character from the others, those methods will not function properly. Sometimes the container code characters are printed or arranged very closely, which makes it a challenge to isolate each character. To address this issue, a new framework for automatic container code recognition (ACCR) in visual images is proposed in this paper. In this framework, code-character regions are first located by applying a horizontal high-pass filter and scan line analysis. Then, character blocks are extracted from the code-character regions and further classified into two categories, i.e. single-character block and multi-character block. Finally, a segmentation-based approach is implemented for recognition of the characters in single-character blocks, and a hidden Markov model (HMM)-based method is proposed for the multi-character blocks. The experimental results demonstrate the effectiveness of the proposed method, which can successfully recognize the container code with closely arranged characters.

In this paper, a sensing image super-resolution (SR) reconstruction method is proposed. Sparse dictionary dealing with remote sensing image SR problem is introduced in this work. The sparse dictionary is based on a sparsity model where the dictionary atoms have sparse representation over a basic dictionary. The sparse dictionary consists of two parts: basic dictionary and atom representation matrix. The sparse dictionary leads to compact representation and it is both adaptive and efficient. Furthermore, compared with conventional SR methods, two dictionary pairs, i.e. primitive sparse dictionary pair and residual sparse dictionary pair, are proposed. The primitive sparse dictionary pair is learned to reconstruct initial high-resolution (HR) remote sensing image from a single low-resolution (LR) input. However, the initial HR remote sensing image loses some details compare with the corresponding original HR image completely. Therefore, residual sparse dictionary pair is learned to reconstruct residual information. The proposed method is tested on remote sensing images, and the experimental results indicate that the proposed algorithm can provide substantial improvement in resolution of remote sensing images, and the results are superior in quality to the results produced by other methods.

Comparison of image processing techniques is critically important in deciding which algorithm, method, or metric to use for enhanced image assessment. Image fusion is a popular choice for various image enhancement applications such as overlay of two image products, refinement of image resolutions for alignment, and image combination for feature extraction and target recognition. Since image fusion is used in many geospatial and night vision applications, it is important to understand these techniques and provide a comparative study of the methods. In this paper, we conduct a comparative study on 12 selected image fusion metrics over six multiresolution image fusion algorithms for two different fusion schemes and input images with distortion. The analysis can be applied to different image combination algorithms, image processing methods, and over a different choice of metrics that are of use to an image processing expert. The paper relates the results to an image quality measurement based on power spectrum and correlation analysis and serves as a summary of many contemporary techniques for objective assessment of image fusion algorithms.

Automatic container code recognition (ACCR) has become an indispensable aspect of current intelligent container management systems. In real applications, an ACCR module sometimes faces the problem of missing characters, i.e., not all the 11 container code characters (CCCs) appear in the input image. However, a few of the present methods can process container code images with missing characters. Therefore, a method is proposed to extract the CCCs for both the situation wherein all the 11 CCCs appear in an image and the situation wherein some CCCs are missing. In this method, hidden Markov model (HMM)-based segmentation confidence is proposed to describe the probability of the segmented characters belonging to the container code. Based on the segmentation confidence, the segmented characters are determined whether they belong to the container code or not, and if there are some characters missing, the positions of these characters can be estimated. Various container code images have been used to test the proposed method. The results of the tests show that the method is effective.