We propose the structure and color based learned image codec (SLIC) in which the task of compression is split into that of luminance and chrominance. The deep learning model is built with a novel multi-scale architecture for Y and UV channels in the encoder, where the features from various stages are combined to obtain the latent representation. An autoregressive context model is employed for backward adaptation and a hyperprior block for forward adaptation. Various experiments are carried out to study and analyze the performance of the proposed model, and to compare it with other image codecs. We also illustrate the advantages of our method through the visualization of channel impulse responses, latent channels and various ablation studies. The model achieves Bjøntegaard delta bitrate gains of 7.5% and 4.66% in terms of MS-SSIM and CIEDE2000 metrics with respect to other state-of-the-art reference codecs.

In this work, we present a comparison between color spaces namely YUV, LAB, RGB and their effect on learned image compression. For this we use the structure and color based learned image codec (SLIC) from our prior work, which consists of two branches - one for the luminance component (Y or L) and another for chrominance components (UV or AB). However, for the RGB variant we input all 3 channels in a single branch, similar to most learned image codecs operating in RGB. The models are trained for multiple bitrate configurations in each color space. We report the findings from our experiments by evaluating them on various datasets and compare the results to state-of-the-art image codecs. The YUV model performs better than the LAB variant in terms of MSSSIM with a Bjøntegaard delta bitrate (BD-BR) gain of 7.5% using VTM intra-coding mode as the baseline. Whereas the LAB variant has a better performance than YUV model in terms of CIEDE2000 having a BD-BR gain of 8%. Overall, the RGB variant of SLIC achieves the best performance with a BD-BR gain of 13.14% in terms of MS-SSIM and a gain of 17.96% in CIEDE2000 at the cost of a higher model complexity.

The objective of this study is to evaluate the effectiveness of state-of-the-art codecs in compressing selected medical imaging modalities while maintaining visual quality and reducing file sizes. To achieve this, a detailed comparative analysis is conducted comparing the performance of JPEG XS, JPEG 2000, HEVC, and AV1. The analysis takes into consideration compression efficiency, codec complexity, and visual fidelity in the context of medical imaging. Advanced evaluation methods, including the AIC-2 Flicker test, are utilized to determine the visually lossless threshold, which is crucial for preserving diagnostically important details. Additionally, the study explores the potential of crowd-sourcing as a means of assessing the visual quality of compressed medical images. Subjective lab and crowd-sourcing tests reveal varying proportions of correctly identifying the reference images among participants. Furthermore, the study proposes outlier detection methods to improve the reliability of the subjective evaluation and employs kappa analysis to measure the inter-rater agreements. The study analyzes the encoding time taken on a consumer-level CPU, and the results reveal that JPEG XS maintains a fast and consistent speed across different compression levels. The results also indicate that JPEG XS achieves visually lossless performance for diagnostic purposes at 2 BPP, JPEG 2000 at 1.5 BPP, HEVC, and AV1 at 1 BPP.

Deep learning based image compression has gained a lot of momentum in recent times. To enable a method that is suitable for image compression and subsequently extended to video compression, we propose a novel deep learning model architecture, where the task of image compression is divided into two sub-tasks, learning structural information from luminance channel and color from chrominance channels. The model has two separate branches to process the luminance and chrominance components. The color difference metric CIEDE2000 is employed in the loss function to optimize the model for color fidelity. We demonstrate the benefits of our approach and compare the performance to other codecs. Additionally, the visualization and analysis of latent channel impulse response is performed.

Joint Photographic Experts Group (JPEG) XS is a new International Standard from the JPEG Committee (formally known as ISO/International Electrotechnical Commission (IEC) JTC1/SC29/WG1). It defines an interoperable, visually lossless low-latency lightweight image coding that can be used for mezzanine compression within any AV market. Among the targeted use cases, one can cite video transport over professional video links (serial digital interface (SDI), internet protocol (IP), and Ethernet), real-time video storage, memory buffers, omnidirectional video capture and rendering, and sensor compression (for example, in cameras and the automotive industry). The core coding system is composed of an optional color transform, a wavelet transform, and a novel entropy encoder, processing groups of coefficients by coding their magnitude level and packing the magnitude refinement. Such a design allows for visually transparent quality at moderate compression ratios, scalable end-to-end latency that ranges from less than one line to a maximum of 32 lines of the image, and a low-complexity real-time implementation in application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), central processing unit (CPU), and graphics processing unit (GPU). This article details the key features of this new standard and the profiles and formats that have been defined so far for the various applications. It also gives a technical description of the core coding system. Finally, the latest performance evaluation results of recent implementations of the standard are presented, followed by the current status of the ongoing standardization process and future milestones.

While traditional image compression algorithms take a full three-component color representation of an image as input, capturing of such images is done in many applications with Bayer CFA pattern sensors that provide only a single color information per sensor element and position. In order to avoid additional complexity at the encoder side, such CFA pattern images can be compressed directly without prior conversion to a full color image. In this paper, we describe a recent activity of the JPEG committee (ISO SC 29 WG 1) to develop such a compression algorithm in the framework of JPEG XS. It turns out that it is important to understand the "development process" from CFA patterns to full color images in order to optimize the image quality of such a compression algorithm, which we will also describe shortly. We introduce (1) a novel decorrelation step upfront processing (the so-called Star-Tetrix transform), along with (2) a pre-emphasis function to improve the compression efficiency of the subsequent compression algorithm (here, JPEG XS). Our experiments clearly indicate a gain over a RGB compression workflow in terms of complexity and quality (between 1.5dB and more than 4dB depending on the target bitrate). A comparison is also made with other state-of-the-art CFA compression techniques.

Image sensors in digital cameras use a technology called "Bayer Patterns" allowing color photography with a planar arrangements of photosensitive elements. Alternating arrangements of red, green and blue filter masks on top of a rectangular grid of such elements allow capturing of color information, but also require a de-mosaicing algorithm to reconstruct a full-resolution color image from the sensor data. In high-speed applications, or applications where the system design requires low-latency, low-complexity compression, the JPEG XS standard of the JPEG committee offers an elegant solution to compress Bayer pattern images close to the sensor, and to transmit the compressed data over a lower bandwidth connection while maintaining visually lossless quality. This paper presents contributions to JPEG XS that are currently under discussion in the JPEG committee (SC29WG1) as parts of an amendment for Bayer pattern compression.

JPEG XS is an upcoming lightweight image compression standard that is especially developed to meet the requirements of compressed video-over-IP use cases. It is designed with not only CPU, FPGA or ASIC platforms in mind, but explicitly also targets GPUs. Though not yet finished, the codec is now sufficiently mature to present a first NVIDIA CUDA-based GPU decoder architecture and preliminary performance results. On a 2014 mid-range GPU with 640 cores a 12 bit UHD 4:2:2 (4:4:4) can be decoded with 54 (42) fps. The algorithm scales very well: on a 2017 high-end GPU with 2560 cores the throughput increases to 190 (150) fps. In contrast, an optimized GPU-accelerated JPEG 2000 decoder takes 2x as long for high compression ratios that yield a PSNR of 40 dB and 3x as long for lower compression ratios with a PSNR of over 50 dB.

Plenoptic cameras based on micro lens arrays as well as multi aperture cameras are able to capture a multitude of images with slightly shifted viewpoints. Although the amount of parallax between adjacent views is limited, precautions have to be taken in order to avoid alias when performing direct lightfield rendering. Against this background, we present an approach for the dense reconstruction of a lightfield based on a sparse lightfield acquired from a multi aperture camera with subsequent disparity estimation and depth image based view interpolation. Results show that the approach is suitable for all-in-focus-rendering.

This paper gives an introduction on the current challenges when storing and accessing movies in a very high quality, e.g. in the field of digital cinema or digital archiving. In contrast to the existing RAID configurations for performance improvements of disk drives, adaptive RAID approaches to be used with scalable media such as JPEG 2000 are introduced. These algorithms allow for assured real-time playback of several storage technologies including standard disk drives as well as cartridges when storing scalable data.

We present a new method for capturing high dynamic range video (HDRV). Our method is based on spatially varying exposures, where individual pixels are covered with filters for different optical attenuation. For preventing the loss in resolution we use a new non-regular arrangement of the attenuation pattern. Subsequent image reconstruction based on the sparsity assumption allows the reconstruction of natural images with high detail.

Professional movie distribution and mastering formats use high quality, intra-frame compression formats like JPEG 2000, a codec allowing for scalability in resolution and quality. Latest processor types, either CPUs or high-end GPUs, can decode digital cinema-compliant JPEG 2000 image sequences in real-time. This brings out a new bottleneck during a movie playback since current consumer hard-drives are not fast enough for data rates being used in digital cinema or digital archiving. This paper shows a method for using file-inherent scalability on file system level as well as a caching strategy especially adapted for scalable media files.

In high quality imaging even tiny distortions as small as a single pixel are visible and can not be accepted. Although the production quality of CMOS image sensors is very high, for reasonable yields we still need to accept some defect pixels and clusters of defects in large image sensors. In this paper we will compare compensation algorithms for raw image sensor data. We propose a new approach based on the sparsity assumption that outperforms existing defect compensation algorithms. Furthermore, our proposed interpolation algorithm is universal and not at all adapted to Bayer pattern images. It can directly be applied to any regular color filter pattern or gray scale image. Our examples show, that image sensors with large clusters of defects can still be used for the generation of high quality images.

Up to now, an increase in camera resolution required image sensors with more and more pixels. However, acquisition systems are limited in their pixels per second throughput given as power and complexity constraints. Simply capturing more pixels in a given system is often not possible. We propose a new non-regular imaging architecture that samples only few pixels and reconstructs a high resolution image afterwards. Our sampling is optimized to provide non-regular spatial sampling from a sensor with regular readout circuits. An existing slow image acquisition system can then be used to capture the data. The image reconstruction is performed with a local sparsity-based approach. The result is a high resolution image that requires a much smaller effort during acquisition.

In typical image sensors the spectral sensitivity of color channels is fixed. The illumination spectrum in natural scenes can vary to a great extent. This leads to an unbalanced response in color channels and hence a reduction in dynamic range. We propose a new method to adjust the relative sensitivity of color channels based on multi-exposure frame combination. Instead of a single long exposure we capture a different number of gapless exposures in each color channel and combine them. This offers a digital option for reducing sensitivity for some color channels and aligns the color channels. The method preserves motion blur and can be used in any long exposure or moving picture photography. We can now get a higher dynamic range from a camera system under any unfavorable illumination conditions with very little effort.

In CMOS image sensors tiny semiconductor variations cause a distortion of the image known as fixed pattern noise (FPN). With a good FPN compensation CMOS sensors can deliver very good image quality. Compensation algorithms differ in complexity, maximum frame rate, and compensation quality with thermal drift. This paper analyzes existing approaches to offset FPN compensation and their drawbacks. We also show a new compensation algorithm that offers fine grained per-pixel compensation and is able to compensate temperature variations while still operating at the maximum sensor frame rate. This enables the construction of motion picture cameras without active cooling or even temperature measurements while still delivering a high image quality at high frame rates.

A digital camera samples the continuous real world. As with any sampling process, questions of aliasing for certain sampling frequencies and the prevention thereof arise. In this paper we will discuss the spatial domain sampling and prevention of aliasing in digital cameras. We focus on the widely used birefringent anti alias filters that are often called optical low pass filters (OLPF). We show 2D models for all contributions to spatial domain sampling and derive optimum filter parameters for minimum aliasing and best possible image sharpness. Compared to previously used selection rules, we can show that the optimum selection of filter parameters can easily deliver more sharpness and reduce aliasing by a factor of 2. The simulated results are finally confirmed in real world experiments.

For motion imaging the perceived smoothness of a sequence highly depends on motion blur. The exposure for each frame is started and ended with a shutter mechanism. There are different implementations and some of them have non-ideal behavior which can introduce artifacts. In this paper an overview of real-world shutter implementations both for analog and digital camera systems is shown. We develop a general description that models all types of shutters and their imperfections. Specific models for common shutter types are presented. Measurements are used to estimate unknown parameters. The modeled shutters are finally used for a virtual camera simulation. Typical artifacts can be simulated and directly compared for different shutter types and parameters. This powerful tool is useful for the construction of camera systems and allows design decisions to be directly compared before building the camera system.