Deep learning-based demosaicking for the Bayer color filter array (CFA) has great advances. However, most existing demosaicking methods only work in the spatial domain and rarely explore the spectral characteristic of Bayer CFA. In this letter, we first attempt to address Bayer demosaicking in both spatial and frequency domains and propose a spatial-frequency fusion network. It consists of three key modules: spatial-domain branch, frequency-domain branch, and dual-domain fusion. Spatial-domain branch employs the standard convolution to extract local features in the spatial domain, while frequency-domain branch adopts frequency selection to maintain CFA's periodicity and achieve the image-wide receptive field for obtaining global features. Dual-domain fusion integrates the complementary representation of the two types of features. Extensive experiments validate the effectiveness of the proposed network.

Interpolation-friendly RGBW color filter arrays (CFAs) and the popular sequential demosaicking contain the idea of computational photography, where the CFA and the demosaicking method are co-designed. Due to the advantages, interpolation-friendly RGBW CFAs have been extensively used in commercial color cameras. However, most associated demosaicking methods rely on strict assumptions or are limited to a few specific CFAs with a given camera. In this paper, we propose a universal demosaicking method for interpolation-friendly RGBW CFAs, which enables the comparison of different CFAs. Our new method belongs to sequential demosaicking, i.e., W channel is interpolated first and then RGB channels are reconstructed with guidance from the interpolated W channel. Specifically, it first interpolates the W channel using only available W pixels followed by an aliasing reduction technique to remove aliasing artifacts. Then it employs an image decomposition model to built relations between W channel and each of RGB channels with known RGB values, which can be easily generalized to the full-size demosaicked image. We apply the linearized alternating direction method (LADM) to solve it with convergence guarantee. Our demosaicking method can be applied to all interpolation-friendly RGBW CFAs with varying color cameras and lighting conditions. Extensive experiments confirm the universal property and advantage of our proposed method with both simulated and real raw images.

In most of existing digital cameras, color images have to be reconstructed from raw images which only have one color sensed at each pixel, as their imaging sensors are covered by color filter arrays (CFAs). At each pixel a CFA usually allows only a portion of the light spectrum to pass through and thereby reduces the light sensitivity of pixels. To address this issue, previous works have explored adding panchromatic pixels into CFAs. However, almost all existing methods assign panchromatic pixels empirically, making the designed CFAs prone to aliasing artifacts. In this paper, based on a mathematical model we propose a fully automatic approach to designing high-sensitivity CFAs using panchromatic pixels. By the frequency structure representation of CFAs, we formulate high-sensitivity CFA design as a continuous multi-objective optimization problem, where robustness to aliasing artifacts and percentage of panchromatic pixels are simultaneously maximized. We analyze the characteristics of our new formulation. According to the analysis, we develop a new method to propose frequency structure candidates, which can produce CFAs that reach a desired percentage of panchromatic pixels. Then for each candidate, we optimize parameters to obtain the final CFA, which is an appropriately balanced solution to the multi-objective optimization problem. We formulate the two design procedures as constrained optimization problems and solve them using the alternating direction method (ADM). Extensive experiments confirm the advantage of the proposed method in both low-light and normal-light conditions.

Demosaicking is the problem of reconstructing a color image from the raw image captured by a digital color camera that covers its only imaging sensor with a color filter array (CFA). Sparse representation-based demosaicking has been shown to produce superior reconstruction quality. However, almost all existing algorithms in this category use the CFAs, which are not specifically optimized for the algorithms. In this paper, we consider optimally designing CFAs for sparse representation-based demosaicking, where the dictionary is well-chosen. The fact that CFAs correspond to the projection matrices used in compressed sensing inspires us to optimize CFAs via minimizing the mutual coherence. This is more challenging than that for traditional projection matrices because CFAs have physical realizability constraints. However, most of the existing methods for minimizing the mutual coherence require that the projection matrices should be unconstrained, making them inapplicable for designing CFAs. We consider directly minimizing the mutual coherence with the CFA's physical realizability constraints as a generalized fractional programming problem, which needs to find sufficiently accurate solutions to a sequence of nonconvex nonsmooth minimization problems. We adapt the redistributed proximal bundle method to address this issue. Experiments on benchmark images testify to the superiority of the proposed method. In particular, we show that a simple sparse representation-based demosaicking algorithm with our specifically optimized CFA can outperform LSSC [1]. To the best of our knowledge, it is the first sparse representation-based demosaicking algorithm that beats LSSC in terms of CPSNR.

In digital color imaging, the raw image is typically obtained through a single sensor covered by a color filter array (CFA), which allows only one color component to be measured at each pixel. The procedure to reconstruct a full color image from the raw image is known as demosaicking. Since the CFA may cause irreversible visual artifacts, the CFA and the demosaicking algorithm are crucial to the quality of demosaicked images. Fortunately, the design of CFAs in the frequency domain provides a theoretical approach to handling this issue. However, almost all the existing design methods in the frequency domain involve considerable human effort. In this paper, we present a new method to automatically design CFAs in the frequency domain. Our method is based on the frequency structure representation of mosaicked images. We utilize a multi-objective optimization approach to propose frequency structure candidates, in which the overlap among the frequency components of images mosaicked with the CFA is minimized. Then, we optimize parameters for each candidate, which is formulated as a constrained optimization problem. We use the alternating direction method to solve it. Our parameter optimization method is applicable to arbitrary frequency structures, including those with conjugate replicas of chrominance components. Experiments on benchmark images confirm the advantage of the proposed method.

The Penrose pixel layout, an aperiodic pixel layout in rhombus Penrose tiling, has been shown to substantially outperform the existing square pixel layout in super-resolution. However, it was tested only on grayscale images. To study its performance on color images, we have to reconstruct regular color images from Penrose raw images, i.e., images with only one color component at each Penrose pixel, resulting in the problem of demosaicking from Penrose pixels. Penrose demosaicking is more difficult than regular demosaicking, because none of the color components of the reconstructed regular color images are available. Therefore, most of the traditional demosaicking methods do not apply. We develop a sparse representation-based method for Penrose demosaicking. Extensive experiments show that Penrose pixel layout outperforms regular pixel layouts in terms of both perceptual evaluation and S-CIELAB. The Penrose pixel layout is unique among all irregular layouts because it is uniformly three-colorable and it has only two pixel shapes, thick and thin rhombi, making its manufacturing relatively easy.