Since camera modules become more and more affordable, multi-spectral camera arrays have found their way from special applications to the mass market, e.g., in automotive systems, smartphones, or drones. Due to multiple modalities, the registration of different viewpoints and the required cross-spectral disparity estimation is up to the present extremely challenging. To overcome this problem, we introduce a novel spectral image synthesis in combination with a color agnostic transform. Thus, any recently published stereo matching network can be turned to a cross-spectral disparity estimator. Our novel algorithm requires only RGB stereo data to train a cross-spectral disparity estimator and a generalization from artificial training data to camera-captured images is obtained. The theoretical examination of the novel color agnostic method is completed by an extensive evaluation compared to state of the art including self-recorded multispectral data and a reference implementation. The novel color agnostic disparity estimation improves cross-spectral as well as conventional color stereo matching by reducing the average end-point error by 41 % for cross-spectral and by 22 % for mono-modal content, respectively.

This paper presents a novel method to determine rate-distortion optimized transform coefficients for efficient compression of videos generated from point clouds. The method exploits a generalized frequency selective extrapolation approach that iteratively determines rate-distortion-optimized coefficients for all basis functions of two-dimensional discrete cosine and sine transforms. The method is applied to blocks containing both occupied and unoccupied pixels in video based point cloud compression for HEVC encoding. In the proposed algorithm, only the values of the transform coefficients are changed such that resulting bit streams are compliant to the V-PCC standard. For all-intra coded point clouds, bitrate savings of more than 4% for geometry and more than 6% for texture error metrics with respect to standard encoding can be observed. These savings are more than twice as high as savings obtained using competing methods from literature. In the randomaccess case, our proposed method outperforms competing V-PCC methods by more than 0.5%.

Numerous applications deal with distorted images, e.g., during transmission over lossy channels in image coding, or to reconstruct occlusions in multi-color multi-view imaging scenarios. In many cases, not all spectral channels are distorted, or the losses distribute differently between the channels. Recently, efforts were made to develop reference guided approaches to reconstruct distorted spectral content. However, these methods are only able to exploit information from a single reference, even if there are multiple spectral components available that could be used for reconstruction. To overcome this limitation, a novel method is proposed in this paper, which introduces a content-adaptive dictionary learning approach that is applicable to an arbitrary number of references. With the novel approach, an average PSNR gain of approx. 1.5 dB is achieved in comparison to the best recently published state-of-the-art methods for a single reference component. Moreover, the presence of multiple guidance channels can enhance the reconstruction by another 6 dB on average. At the same time, the complexity of the novel algorithm is significantly lower than for previously published methods.

Recently, cross-spectral stereo-camera setups found their way from special applications to mass market, especially in smartphones, automotive systems, or drones. In the following, a novel concept is introduced to bring stereo cameras and cross-spectral disparity estimation together. So far, either monomodal stereo algorithms exist that are not suitable for cross-spectral image registration, or structural template matching is applied that achieves a low quality. To overcome these limitations, a technique is proposed to synthesize arbitrary spectral components from widely available color stereo databases, and to retrain mono-modal deep learning methods. In this contribution, the estimation of spectral bands based on random processes is shown together with noise models, which also allow for a robust registration of narrowband components. The theoretical examination is completed by an extensive evaluation, including a self-manufactured cross-spectral camera setup. In comparison to state-of-the-art techniques, the end-point error is on average reduced by a factor of seven.

Many applications in image processing require re-sampling of arbitrarily located samples onto regular grid positions. This is important in frame-rate up-conversion, super-resolution, and image warping among others. A state-of-the-art high quality model-based resampling technique is frequency-selective mesh-to-grid resampling which requires pre-estimation of key points. In this paper, we propose a new key point agnostic frequency-selective mesh-to-grid resampling that does not depend on pre-estimated key points. Hence, the number of data points that are included is reduced drastically and the run time decreases significantly. To compensate for the key points, a spectral weighting function is introduced that models the optical transfer function in order to favor low frequencies more than high ones. Thereby, resampling artefacts like ringing are supressed reliably and the resampling quality increases. On average, the new AFSMR is conceptually simpler and gains up to 1.2 dB in terms of PSNR compared to the original mesh-to-grid resampling while being approximately 14.5 times faster.

Recent developments in optical sensors enable a wide range of applications for multispectral imaging, e.g., in surveillance, optical sorting, and life-science instrumentation. Increasing spatial and spectral resolution allows creating higher quality products, however, it poses challenges in handling such large amounts of data. Consequently, specialized compression techniques for multispectral images are required. High Efficiency Video Coding (HEVC) is known to be the state of the art in efficiency for both video coding and still image coding. In this paper, we propose a cross-spectral compression scheme for efficiently coding multispectral data based on HEVC. Extending intra picture prediction by a novel inter-band predictor, spectral as well as spatial redundancies can be effectively exploited. Dependencies among the current band and further spectral references are considered jointly by adaptive linear regression modeling. The proposed backward prediction scheme does not require additional side information for decoding. We show that our novel approach is able to outperform state-of-the-art lossy compression techniques in terms of rate-distortion performance. On different data sets, average Bjøntegaard delta rate savings of 82 % and 55 % compared to HEVC and a reference method from literature are achieved, respectively.

Recently, many new applications arose for multispectral and hyper-spectral imaging. Besides modern biometric systems for identity verification, also agricultural and medical applications came up, which measure the health condition of plants and humans. Despite the growing demand, the acquisition of multi-spectral data is up to the present complicated. Often, expensive, inflexible, or low resolution acquisition setups are only obtainable for specific professional applications. To overcome these limitations, a novel camera array for multi-spectral imaging is presented in this article for generating consistent multispectral videos. As differing spectral images are acquired at various viewpoints, a geometrically constrained multi-camera sensor layout is introduced, which enables the formulation of novel registration and reconstruction algorithms to globally set up robust models. On average, the novel acquisition approach achieves a gain of 2.5 dB PSNR compared to recently published multi-spectral filter array imaging systems. At the same time, the proposed acquisition system ensures not only a superior spatial, but also a high spectral, and temporal resolution, while filters are flexibly exchangeable by the user depending on the application. Moreover, depth information is generated, so that 3D imaging applications, e.g., for augmented or virtual reality, become possible. The proposed camera array for multi-spectral imaging can be set up using off-the-shelf hardware, which allows for a compact design and employment in, e.g., mobile devices or drones, while being cost-effective.

High quality algorithms are demanded to reconstruct distorted color images in a variety of applications. For example, distortions can result during transmission over lossy channels in image coding or in multi-view imaging scenarios. In general, not all color channels are equally affected and the losses distribute differently in-between channels. However, state-ofthe-art methods process color channels independently and do not take the cross color information into account. Thus, a novel and powerful reconstruction algorithm is formulated in this contribution that exploits color as well as spatial information. Therefore, an initial model is estimated for the distorted area using a reference channel. Then, its quality is estimated and a spatial weighting model is set-up. Afterwards, the initial inter channel prediction is refined by generating a sparse model that takes the spatial correlations into account, as well. Consequently, the proposed method achieves an outstanding quality compared to state-of-the-art methods.

In many cases, image and video signal processing demands for high quality extrapolation algorithms, e.g., to solve inpainting problems or to increase image resolution. Indeed, a high computational load goes hand in hand with a good reconstruction quality as expensive models are calculated to estimate the missing data. To overcome this, the high-speed sparse Hartley modeling is introduced in this paper. This algorithm is based on Frequency Selective Extrapolation. In contrast to that, the model generation is carried out in the Hartley domain to exploit its real-valued transform properties. Due to this, it is possible to reduce the computational complexity significantly as no complex-valued arithmetic operations have to be conducted. In other words, a slightly higher reconstruction quality is obtained, while the proposed method is more than three times faster than the competing Frequency Selective Extrapolation.

The concealment of errors is an important task in image and video signal processing. Often, complex models are calculated to reconstruct the missing samples, which results in a long computation time. One method that achieves a very high reconstruction quality, but demands a moderate computational complexity only, is the block based Frequency Selective Extrapolation. Nevertheless, the reconstruction of a Full HD image can still take several minutes depending on the error pattern. To accelerate the computation, a novel algorithm is introduced in this paper that analyzes the adjacent, undistorted samples and optimizes the reconstruction parameters accordingly. Moreover, the analyzation is further used to adapt the partitioning of the blocks and the processing order. Similar to modern video codecs, e.g., High Efficiency Video Coding, a content based partitioning and processing is proposed as it takes the signal characteristics into account. Thus, the novel algorithm is on average four times faster than the state-of-the-art method and up to 25× quicker at best, while achieving a slightly higher reconstruction quality as well.

We propose to adapt the local statistics estimation, which has been designed for rapid error concealment originally. By merging the approaches, the reconstruction of images is fastened significantly. We will show in this demonstration that several frames per second (fps) can be processed using conventional computers and HD sized image data.

Image and video signal processing demands in many areas for error concealment algorithms, whereby the execution time plays an important role. Within this paper, we introduce the scaled fixed-point Frequency Selective Extrapolation for fast image error concealment. This algorithm is based on the complex-valued Frequency Selective Extrapolation, but reduces computational complexity. It is shown that a fixed-point realization of the state-of-the-art algorithm requires an impracticable word-length, which makes it necessary to design a novel, scaled Frequency Selective Extrapolation that reduces the required word-length. Regarding the new approach, execution time can be reduced on average by 22.84 % and at best by up to 40.45 % compared to the state-of-the-art floating point Frequency Selective Extrapolation at the same reconstruction quality. Moreover, platforms can be exploited, which support fixed-point arithmetic only.