Multispectral (MS) images contain richer spectral information than RGB images due to their increased number of channels and are widely used for various applications. However, achieving accurate estimation in MS images remains challenging, as previous studies have struggled with spectral diversity and the inherent entanglement between the illuminant and surface reflectance spectra. To tackle these challenges, in this paper, we propose a novel Illumination spectrum estimation technique for MS images via Surface reflectance modeling and Spatial-spectral feature generation (ISS). The proposed technique employs a learnable spectral unmixing (SU) block to enhance surface reflectance modeling, which was unattempted in the illumination spectrum estimation, and a feature mixing block to fuse spectral and spatial features of MS images with cross-attention. The features are refined iteratively and processed through a decoder to produce an illumination spectrum estimator. Experimental results demonstrate that the proposed technique achieves state-of-the-art performance in illumination spectrum estimation in various MS image datasets. The code is available at https://github.com/heyjinnii/ISS-MSI.git.

Multispectral (MS) imaging systems have a wide range of applications for computer vision and computational photography tasks, but do not yet enjoy widespread adoption due to their prohibitive costs. Recently, advances in the design and fabrication of photonic metamaterials have enabled the development of MS sensors suitable for integration into consumer grade mobile devices. Augmenting existing RGB cameras and their processing algorithms with richer spectral information has the potential to be utilized in many steps of the image processing pipeline, but diverse real world datasets suitable for conducting such research are not freely available. We introduce Beyond RGB1, a real-world dataset comprising thousands of multispectral and RGB images in diverse real world and lab conditions that is suitable for the development and evaluation of algorithms utilizing multispectral and RGB data. All the scenes in our dataset include a colorimetric reference and a measurement of the spectrum of the scene illuminant. Additionally, we demonstrate the practical use of our dataset through the introduction of a novel illuminant spectral estimation (ISE) algorithm. Our algorithm surpasses the current state-of-the-art (SoTA) by up to 58.6% on established benchmarks and sets a baseline performance on our own dataset.

Video coding in the YCbCr color space has been widely used, since it is efficient for compression, but it can result in color distortion due to conversion error. Meanwhile, coding in the RGB color space maintains high color fidelity, having the drawback of a substantial bitrate increase with respect to YCbCr coding. Cross-component prediction (CCP) efficiently compresses video content by decorrelating color components while keeping high color fidelity. In this scheme, the chroma residual signal is predicted from the luma residual signal inside the coding loop. This paper gives a description of the CCP scheme from several points of view, from theoretical background to practical implementation. The proposed CCP scheme has been evaluated in standardization communities and adopted into H.265/High Efficiency Video Coding (HEVC) Range Extensions. The experimental results show significant coding performance improvements for both natural and screen content video, while the quality of all color components is maintained. The average coding gains for natural video are 17% and 5% bitrate reduction in the case of intra coding and 11% and 4% in the case of inter coding for RGB and YCbCr coding, respectively, while the average increment of encoding and decoding times in the HEVC reference software implementation are 10% and 4%, respectively.

Referring expression comprehension, which locates the object instance described by a natural language expression, gains increasing interests in recent years. This paper aims at improving the task from two aspects: visual feature extraction and language features extraction. For visual feature extraction, we observe that most of the previous methods utilize only relative spatial information to model the visual relationship between object pairs while discarding rich semantic relationship between objects. This makes the visual-language matching difficult when the language expression contains semantic relationship to discriminate the referred object from other objects in the image. In this work, we propose a Semantic Visual Relationship Module (SVRM) to exploit this important information. For language feature extraction, a major problem comes from the long-tail distribution of words in the expressions. Since more than half of the words appear less than 20 times in the public datasets, deep models such as LSTM tend to fail to learn accurate representations for these words. To solve this problem, we propose a word2vec based word mapping method that maps these low frequency words to high frequency words with similar meaning. Experiments show that the proposed method outperforms existing state-of-the-art methods on three referring expression comprehension datasets.

In order to improve 3D video coding efficiency, we propose methods to estimate rendered view distortion in synthesized views as a function of the depth map quantization error. Our approach starts by calculating the geometric error caused by the depth map error based on the camera parameters. Then, we estimate the rendered view distortion based on the local video characteristics. The estimated rendered view distortion is used in the rate-distortion optimized mode selection for depth map coding. A Lagrange multiplier is derived using the proposed distortion metric, which is estimated based on an autoregressive model. Experimental results show the efficiency of the proposed methods, with average savings of 43% in depth map bitrate as compared with encoding the depth maps using the same coding tools but with the rate-distortion optimization based on the conventional distortion metric.

This paper presents a new coding tool named cross component decorrelation in the emerging High Efficiency Video Coding Range Extension (HEVC RExt) standard. Color video is generally composed of three color components, e.g., RGB or YCbCr. It has been known for over a decade that the three color components have correlation among each other. Although global out-of-loop color space conversion, e.g., RGB-to-YCbCr, can reduce the cross component correlation, local correlation still exists in YCbCr signal. Many methods have been developed in the literature to exploit such redundancy to improve coding efficiency. However, existing methods introduce high computational or implementation complexity, which makes them never be included in mainstream video coding standard such as H.264/AVC and HEVC version 1. In this research, a new hardware friendly cross component decorrelation method is presented which reduces implementation cost while achieving significant BD-rate reduction. For example, based on JCT-VC common test condition for HEVC RExt standardization, the proposed method results in (17.3%, 18.1%, 16.6%) BD-rate reduction for the three color components of the RGB test sequences, in the case of All Intra configuration.

Sample adaptive offset (SAO) is the new in-loop filter in High Efficiency Video Coding (HEVC) standard. In this paper, the problems in the early version of SAO technique is discussed, and it is shown how the proposed methods improve its performance. It is proposed to restrict edge offset sign according to edge shape to reduce visual artifact caused by edge offset. It is also proposed to reduce number of band offset to facilitate implementation. Experimental results shows the proposed methods effectively reduce the artifact caused by edge offset, and reduce number of band offset by half without compromising coding efficiency.

In this paper a graph-based transform is proposed as an alternative to the discrete cosine transform. An image or video signal is represented as a graph signal, where the graph is generated so as not to cross an image edge in a local region, i.e., square block. Then, spectral representation of graph signal is used to form transform kernels by finding eigenvectors of Laplacian matrix of the graph. This method requires to include additional information, i.e., edge map or adjacency matrix, into a bitstream so that a decoder can regenerate the exactly same graph used at an encoder. The novelty of this paper includes finding the optimal adjacency matrix and compressing it using context-based adaptive binary arithmetic coding. Coding efficiency improvement can be achieved when an image block contains arbitrarily shaped edges by applying the transform not across the edges. The proposed transform is applied to coding depth maps used for view synthesis in a multi-view video coding system, and provides 14% bit rate savings on average.

To improve rendered view quality in a 3-D video system, we propose to encode and transmit depth transition data, which represents, for each pixel in a frame, the location in between two existing views where the depth corresponding to that pixel changes. Given the highly localized and non-linear characteristics of rendered view distortion, it is possible to achieve better coding performance by providing this depth transition data only for subjectively important regions. In this paper, a method to apply the depth transition data to the view rendering procedure is proposed. Experimental results verify that improvements in subjective quality can be achieved by the proposed method.

Depth map compression is important for compact “texture-plus-depth” representation of a 3D scene, where texture and depth maps captured from multiple camera viewpoints are coded into the same format. Having received such format, the decoder can synthesize any novel intermediate view using texture and depth maps of two neighboring captured views via depth-image-based rendering (DIBR). In this paper, we combine two previously proposed depth map compression techniques that promote sparsity in the transform domain for coding gain-graph-based transform (GBT) and transform domain sparsification (TDS) - together under one unified optimization framework. The key to combining GBT and TDS is to adaptively select the simplest transform per block that leads to a sparse representation. For blocks without detected prominent edges, the synthesized view's distortion sensitivity to depth map errors is low, and TDS can effectively identify a sparse depth signal in fixed DCT domain within a large search space of good signals with small synthesized view distortion. For blocks with detected prominent edges, the synthesized view's distortion sensitivity to depth map errors is high, and the search space of good depth signals for TDS to find sparse representations in DCT domain is small. In this case, GBT is first performed on a graph defining all detected edges, so that filtering across edges is avoided, resulting in a sparsity count ρ in GBT. We then incrementally add the most important edge to an initial no-edge graph, each time performing TDS in the resulting GBT domain, until the same sparsity count ρ is achieved. Experimentation on two sets of multiview images showed gain of up to 0.7dB in PSNR in synthesized view quality compared to previous techniques that employ either GBT or TDS alone.

This work proposes a new intra prediction coding scheme for depth map images used in view interpolation. The main goal is to design a prediction scheme which can reduce the prediction error energy in blocks with arbitrary edge shapes. This will reduce the rate needed to encode such blocks while also eliminating some of the annoying artifacts caused by quantization. Since depth maps typically consist of smooth regions separated by edges, we find it sufficient to design prediction schemes which can make effective use of edge information. Working from the intra prediction framework in H.264, we provide a graph representation of pixels in a block and pixels from previously coded blocks and construct an edge-aware prediction scheme based on this. We also employ existing rate-distortion (RD) optimization methods to further improve the coding performance. Our proposed methods reduce the bit rate for depth maps by up to 29% for a fixed interpolated PSNR for some sequences.

The objective of this work is to develop a new 3-D video coding system which can provide better coding efficiency with improved subjective quality as compared to existing 3-D video systems. We have analyzed the distortions that occur in rendered views generated using depth image based rendering (DIBR) and classified them in order to evaluate their impact on subjective quality. As a result, we found that depth map coding distortion leads to “erosion artifacts” at object boundaries, which lead to significant degradation in perceptual quality. To solve this problem, we propose a solution in which depth transition data is encoded and transmitted to the decoder. Depth transition data for a given pixel indicates the camera position for which this pixel's depth will change. A main reason to consider transmitting explicitly this information is that it can be used to improve view interpolation at many different intermediate camera positions. Simulation results show that the subjective quality can be significantly improved by reducing the effect of erosion artifacts, using our proposed depth transition data. Maximum PSNR gains of about 0.5 dB can also be observed.

In this work a new set of edge-adaptive transforms (EATs) is presented as an alternative to the standard DCTs used in image and video coding applications. These transforms avoid filtering across edges in each image block, thus, they avoid creating large high frequency coefficients. These transforms are then combined with the DCT in H.264/AVC and a transform mode selection algorithm is used to choose between DCT and EAT in an RD-optimized manner. These transforms are applied to coding depth maps used for view synthesis in a multi-view video coding system, and provides up to 29% bit rate reduction for a fixed quality in the synthesized views.

Video representations that support view synthesis based on depth maps, such as multiview plus depth (MVD), have been recently proposed, raising interest in efficient tools for depth map coding. In this paper, we derive a new distortion metric that takes into consideration camera parameters and global video characteristics in order to quantify the effect of lossy coding of depth maps on synthesized view quality. In addition, a new skip mode selection method is proposed based on local video characteristics. Experimental results with the proposed mode selection scheme show coding gains of up to 2 dB for the synthesized views, as well as better subjective quality.

This paper presents an efficient video coding method in RGB space. Generally RGB space is regarded as a bad space from a compression point of view. Each RGB component contains the color information along with luminance information. This redundancy degrades the coding efficiency. To exploit the intercolor redundancy, the RGB space is usually converted into another color space. But it inevitably causes color distortion due to the rounding operation involved. For professional applications it is essential to maintain the original color fidelity. To cope with these problems we propose an inter-plane prediction (IPP) that increases the coding efficiency in RGB space while avoiding color distortion. The proposed IPP exploits the redundancy of the residual images, which exists even after intra/inter prediction in MC-DCT based codec. The simulation results show that by applying the IPP to the MPEG-4 AVC/H.264 4:4:4 codec we can achieve up to 40% higher coding efficiency than that of the coding without IPP option. We also show there exists an achievable PSNR limit if color space conversion is involved. In other words, coding in a given space is important to keep the fidelity of the original data for professional applications.

This paper presents a new color transform (YSbSr) that increases the coding efficiency for RGB space by reducing the conversion error and coding error propagation that happen during the color space conversion. Traditionally the use of color transforms has been limited to maintain the compatibility with black-and-white transmission. But for high quality video for professional applications, the focus should change accordingly. Recently during the course of development for Professional Extensions of the MPEG-4 AVC H.264 standard, new color transforms have been proposed. These transforms are focusing on the decorrelation and reversibility using a lifting scheme for integer implementation. But in a real coding scenario, lossless coding is rare and coding error is inevitable. So we have to deal with the propagation of coding error due to the backward conversion to RGB space. Even though integer mapping is a charming factor, it is more desirable not to employ integer mapping if we lose the original goal, high decorrelation gain with small conversion error. Since the conversion equation takes up a very small portion of coder complexity, the difference between integer mappings and floating point operation is diluted. Based on this observation, we propose a new color transform that gives: high decorrelation gain with small rounding and conversion error. The simulation results show that we can achieve almost up to 2 dB gain compared with the traditional YCbCr color space using the proposed transform. It makes the proposed color transform a good candidate for applications that need to preserve the original color fidelity.