As understanding and monitoring global warming and heatwaves have become increasingly important, the demand for higher spatial and temporal resolution in satellite-observed Land Surface Temperature (LST) data has risen. LST derived from geostationary satellite observations plays a crucial role in temperature monitoring, offering high temporal resolution across wide areas. However, a primary limitation of geostationary satellite-derived LST products is their low spatial resolution. In this study, we aim to overcome this limitation by employing a deep learning-based super-resolution (SR) approach and propose a model calledReferences and Residual in Residual Blocks to Perform Super Resolution Network (3R-Net). This model incorporates terrain-guided reference images and residual blocks to enable more accurate super-resolution. By using LST products from GEO-KOMPSAT-2A (GK2A) as low-resolution input, visible channel imagery from GEO-KOMPSAT-2B (GK2B) as terrain-reflective high-resolution reference, and Landsat-8 LST products as high-resolution target images, our model effectively enhances the 2 km resolution of GK2A LST products to the 500 m resolution of Landsat-8 with improved spatial detail and accuracy. Unlike many previous studies relying on single-image super-resolution with synthetically downsampled inputs, our approach uses real-world LR inputs, HR targets, and reference images, making the learning process more realistic and practical. Experimental results show that, when clear guidance is provided through reference images, 3R-Net surpasses existing SR methods, achieving higher PSNR, SSIM, and lower RMSE while capturing critical spatial and temporal features, including surface characteristics and daily heating patterns. By integrating residual-in-residual blocks our model achieves a simple yet powerful enhancement in capturing fine-grained spatial and temporal patterns. These advancements suggest that 3R-Net can provide enhanced LST data crucial for climate research, environmental monitoring, and early warning systems.

We propose an approach to reconstruct a precise and dense 3d point cloud from a single image. Previous works employed reconstruction to complexity 3D shape or directly regression location from image. However, while the former requires overhead construction of 3D shape or is inefficient because of high computing cost, the latter does not scale well as the number of trainable parameters depends on the number of output points. In this paper, we explore a method to infer a point cloud representation given an input image. We extract shape information from an input image, and then we embed the two kinds of shape information into the point cloud: point-specific and global shape features. After that, we deform a randomly generated point cloud to the final representation based on the embedded point cloud feature. Our method does not require overhead construction, and is efficient and scalable because the number of trainable parameters is independent of the point cloud size, which is the first work to be able to do so according to our knowledge. Thorough experimental results suggest that our proposed method outperforms with other state-of-the-art methods in dense and precise point cloud generation.

In previous quality assessment studies on stereoscopic 3D (S3D) images, researchers have concentrated on deriving manually extracted features which represent the quality of images. These features are based on the human visual system or natural scene statistics, but they have not been revealed as a deterministic function, preventing to guarantee the robustness of features. To solve this problem, we introduce a deep learning method for predicting the quality of S3D images without a reference. A convolutional neural network (CNN) model is trained through two-step learning. First, to overcome the lack of training data, patch-based CNNs are introduced. And then, automatically extracted patch features are pooled into image features. Finally, the trained CNN model parameters are updated iteratively using holistic image labeling, i.e., mean opinion score (MOS). The proposed method represents a significant improvement compared to other no-reference (NR) S3D image quality assessment (IQA) algorithms.

Even though zero padding is usually a staple in convolutional neural networks to maintain the output size, it is highly suspicious because it significantly alters the input distribution around border region. To mitigate this problem, in this paper, we propose a new padding technique termed as distribution padding. The goal of the method is to approximately maintain the statistics of the input border regions. We introduce two different ways to achieve our goal. In both approaches, the padded values are derived from the means of the border patches, but those values are handled in a different way in each variant. Through extensive experiments on image classification and style transfer using different architectures, we demonstrate that the proposed padding technique consistently outperforms the default zero padding, and hence can be a potential candidate for its replacement.

Due to the evolution of ultra-high-definition (UHD) technologies, viewers can enjoy more realistic contents. Furthermore, in order to maximize visual attraction, post-processing is conducted in commercial devices. In this paper, we propose a new terminology called visual preference to quantify viewer's preferences for sharpness- and contrast- enhanced UHD images in a particular viewing geometry. Visual preferences depend on the spatial characteristics and are affected by the viewing geometry of display like resolution, display size, and viewing distance. Therefore, we propose a method called visual preference assessment model that accounts for content enhancement features and diverse viewing geometry. By rigorous experiments, our proposed model outperforms other state-of-the-art models.

The high performance video quality assessment (VQA) algorithm is a necessary skill to provide high quality video to viewers. However, since the nonlinear perception function between the distortion level of the video and the subjective quality score is not precisely defined, there are many limitations in accurately predicting the quality of the video. In this paper, we propose a deep learning scheme named Deep Blind Video Quality Assessment (DeepBVQA) to achieve a more accurate and reliable video quality predictor by considering various spatial and temporal cues which have not been considered before. We used CNN to extract the spatial cues of each video in VQA and proposed new hand-crafted features for temporal cues. Performance experiments show that performance is better than other state-of-the-art no-reference (NR) VQA models and the introduction of hand-crafted temporal features is very efficient in VQA.

The direct use of a deep convolutional neural network (CNN) in no-reference image quality assessment (NR-IQA) usually struggles for a good performance due to a lack of training data, which can be alleviated by transfer learning. However, depending on the similarity between the source and target tasks, the final performance differs vastly. In particular, various kinds of distortion types exist in IQA, which requires different kinds of features to predict visual quality. In this paper, to make the transferred model robust to various distortion types, we propose a Multiple-level Feature-based Image Quality Assessor (MFIQA) which considers multiple levels of features simultaneously. Through rigorous experiments, we prove that MFIQA consistently yields state-of-the-art performance regardless of the distortion types including synthetic and authentic corruption.

Predicting the degree of sickness is an imperative goal to guarantee viewing safety when watching virtual reality (VR) contents. Ideally, such predictive models should be explained in terms of the human visual system (HVS). When viewing VR contents using a head mounted display (HMD), there is a conflict between user's actual motion and visually perceived motion. This results in an unnatural visual-vestibular sensory mismatch that causes side effects such as onset of nausea, oculomotor, disorientation, asthenopia (eyestrain). In this paper, we propose a framework called VR sickness predictor (VRSP) using the interaction model between user's motion and the vestibular system. VRSP extracts two types of features: a) perceptual motion feature through a visual-vestibular interaction model, and b) statistical content feature that affects user motion perception. Furthermore, we build a VR sickness database including 36 virtual scenes to evaluate the performance of VRSP. Through rigorous experiments, we demonstrate that the correlation between the proposed model and the subjective sickness score yields ~72 %.

Most prior approaches to the problem of stereoscopic 3D (S3D) visual discomfort prediction (VDP) have focused on the extraction of perceptually meaningful handcrafted features based on models of visual perception and of natural depth statistics. Towards advancing performance on this problem, we have developed a deep learning based VDP model named Deep Visual Discomfort Predictor (DeepVDP). DeepVDP uses a convolutional neural network (CNN) to learn features that are highly predictive of experienced visual discomfort. Since a large amount of reference data is needed to train a CNN, we develop a systematic way of dividing S3D image into local regions defined as patches, and model a patch-based CNN using two sequential training steps. Since it is very difficult to obtain human opinions on each patch, instead a proxy ground-truth label that is generated by an existing S3D visual discomfort prediction algorithm called 3D-VDP is assigned to each patch. These proxy ground-truth labels are used to conduct the first stage of training the CNN. In the second stage, the automatically learned local abstractions are aggregated into global features via a feature aggregation layer. The learned features are iteratively updated via supervised learning on subjective 3D discomfort scores, which serve as ground-truth labels on each S3D image. The patchbased CNN model that has been pretrained on proxy groundtruth labels is subsequently retrained on true global subjective scores. The global S3D visual discomfort scores predicted by the trained DeepVDP model achieve state-of-the-art performance as compared to previous VDP algorithms.

In recent years, how to quantify visualizations of an object and surface displayed in 3D space is now more prominent with a rapid increase in the demand for three-dimensional (3D) content. In order to measure the content information in terms of human visual perception, it is necessary to quantify the visual information in accordance with the human visual system. In this paper, we propose a framework for expressing visual information in bits termed visual entropy based on information theory. The visual entropy of 2D content (2DVE) is composed of texture entropy on the 2D surface and depth entropy based on the monocular cue. In addition to 2DVE, the visual entropy of 3D content (3DVE) includes the depth entropy based on the binocular cue. A series of simulations are conducted to demonstrate the effectiveness of visual entropy, including a performance trade-off between 2D and 3D visualizations measured according to the bitrate.

Previously, no-reference (NR) stereoscopic 3D (S3D) image quality assessment (IQA) algorithms have been limited to the extraction of reliable hand-crafted features based on an understanding of the insufficiently revealed human visual system or natural scene statistics. Furthermore, compared with full-reference (FR) S3D IQA metrics, it is difficult to achieve competitive quality score predictions using the extracted features, which are not optimized with respect to human opinion. To cope with this limitation of the conventional approach, we introduce a novel deep learning scheme for NR S3D IQA in terms of local to global feature aggregation. A deep convolutional neural network (CNN) model is trained in a supervised manner through two-step regression. First, to overcome the lack of training data, local patch-based CNNs are modeled, and the FR S3D IQA metric is used to approximate a reference ground-truth for training the CNNs. The automatically extracted local abstractions are aggregated into global features by inserting an aggregation layer in the deep structure. The locally trained model parameters are then updated iteratively using supervised global labeling, i.e., subjective mean opinion score (MOS). In particular, the proposed deep NR S3D image quality evaluator does not estimate the depth from a pair of S3D images. The S3D image quality scores predicted by the proposed method represent a significant improvement over those of previous NR S3D IQA algorithms. Indeed, the accuracy of the proposed method is competitive with FR S3D IQA metrics, having ~ 91% correlation in terms of MOS.

By analyzing the statistical behaviors on human visual attention, we discover a clue that the fixation behaviors are highly correlated with how much the viewers feel visual discomfort on stereoscopic images differently from conventional subjective assessments. In order to quantify the correlation between visual attention and discomfort, we explore a novel methodology termed transition of visual attention (ToVA) according to various disparities, which accounts depth attributes of 3D images by eye-tracker experiments. Moreover, the saliency entropy is defined to quantify the distribution of fixations for 3D images. Then, we measure ToVA in terms of the relative saliency entropy using Kullback-Leibler divergence. In order to evaluate the effectiveness of ToVA, a successful example application is also provided, whereby ToVA is applied to obtaining subjective results of measuring discomfort experienced when viewing 3D displays rather than relying on the conventional subjective test by using scoring system.