Sparse regularization is being applied to solve indeterminate inverse problems. However, current regularization is unable to manage sparsity and small perturbations at the same time, and does not perform well enough for some applications. In this study, we propose reversed ordered weighted L1-norm regularization (ROWL) that can tolerate small perturbations while well-handling sparsity. Since ROWL can make proximity mapping easy to compute, it is possible to construct an algorithm to find a suboptimal solution to the inverse problem using the proximity splitting method. Using ROWL for image edge-preserving smoothing, allows us to control both edge sharpness and gradation smoothness.

Deep learning has been widely studied for processing and understanding multimedia data, and it does help improve performance. Recent research has shown that deep models are vulnerable to images containing adverse weather corruptions, leading to a safety risk for numerous safety-critical systems (e.g., autonomous driving systems). There are two problems with the current situation. First, collecting data under different weather scenarios is highly difficult in practice. Second, the performance degrades significantly when the training and test data are from different distributions, as exemplified by the weather corrupted test data. As a result, it is challenging to train a model without access to the images containing variations of various weather conditions, and it is difficult to make trained model generalized to unknown data under different weather conditions. In this paper, we introduce aCounterfactual Representation Learning(CRL) method to address these problems. Without access to training data including weather condition variations, our CRL makes the model resistant to unseen test data that has been corrupted by weather condition variations. Our basic idea is inspired by the perspective of counterfactual regularization. We build a causal model that introduces a counterfactual variable to eliminate the unobserved characteristics brought about by weather conditions. In particular, such a counterfactual variable is approximated by randomly shuffled features, echoing the previous empirical observation that the shuffling technique can perturb the shape details while preserving the local textures. We use information theoretic representation learning to encourage the neural networks to learn more powerful and robust features, which consist of two components. We conduct experiments on five benchmark datasets, namely, CIFAR-100-C, ImageNet-C, KITTI-C, BDD100 k, and CityScapes-C, all of which contain weather corruption. The results of our experiments show that our proposed method can not only be a plug-and-play technique but also work nicely for both object recognition and detection.

Denoising is one of the most fundamental and important problems in signal processing, and graph signal denoising methods have been actively studied. Several graph signal denoising methods based on mathematical programming require solving linear equations involving Laplacian matrix, which creates problem with computational accuracy and running time. This study proposes a fast and accurate solution of linear equations for denoising based on the fast graph Fourier transform method. Moreover, the proposed method can perform denoising not only on graphs for which the fast graph Fourier transform can be performed, but also on a wide class of graphs with more relaxed conditions, without loss of accuracy. Experiments demonstrate the efficiency of the proposed method and confirm that denoising can be performed up to 167.3 times faster without loss of accuracy.

We propose a simultaneous method of multimodal graph signal denoising and graph learning. Since sensor networks distributed in space can capture multiple modalities of data, referred to as modalities, they are assumed to have an underlying structure or correlations both in space and modality. Such multimodal data are regarded as graph signals on a twofold graph. Like regular signals, multimodal graph signals can be corrupted by noise during their sensing process. Furthermore, their spatial/modality relationships are not given a priori: We need to estimate twofold graphs during denoising. In this paper, we propose a signal denoising method on twofold graphs where graphs are learned simultaneously. Specifically, we formulate an optimization problem for that, and an iterative algorithm for solving it is unrolled with deep algorithm unrolling (DAU). In the proposed method, the parameters in iterations are learned from training data that results in faster convergence and denoising quality improvements. Experimental results demonstrate that the proposed method outperforms existing graph signal denoising methods.

In image classification, a deep neural network (DNN) that is trained on undistorted images constitutes an effective decision boundary. Unfortunately, this boundary does not support distorted images, such as noisy or blurry ones, leading to accuracy drop-off. As a simple approach for classifying distorted images as well as undistorted ones, previous methods have optimized the trained DNN again on both kinds of images. However, in these methods, the decision boundary may become overly complicated during optimization because there is no regularization of the decision boundary. Consequently, this decision boundary limits efficient optimization. In this paper, we study a simple yet effective decision boundary for distorted image classification through the use of a novel loss, called a "neural activation pattern matching (NAPM) loss". The NAPM loss is based on recent findings that the decision boundary is a piecewise linear function, where each linear segment is constructed from a neural activation pattern in the DNN when an image is fed to it. The NAPM loss extracts the neural activation patterns when the distorted image and its undistorted version are fed to the DNN and then matches them with each other via the sigmoid cross-entropy. Therefore, it constrains the DNN to classify the distorted image and its undistorted version by the same linear segment. As a result, our loss accelerates efficient optimization by preventing the decision boundary from becoming overly complicated. Our experiments demonstrate that our loss increases the accuracy of the previous methods in all conditions evaluated.

Graph convolution network (GCN) has been extensively applied to the area of hyperspectral image (HSI) classification. However, the graph can not effectively describe the complex relationships between HSI pixels and the GCN still faces the challenge of insufficient labeled pixels. In order to alleviate the above two issues faced by the GCN in HSI classification, we propose a novel framework that integrates the active learning and the hypergraph neural network. First, we construct a hypergraph that can reveal the complex non-pairwise relationships embedded in the hyperspectral images. Next, we train a semi-supervised hypergraph neural network (GNN) with the fewer labeled training set. Then, exploiting the local structural properties of the hypergraph, the most useful HSI pixels are actively selected for labeling. Finally, we fine-tune the GNN with original training set along with the newly labeled pixels. And the last three steps are iteratively carried on. Compared with the other traditional and active learning approaches of HSI classification, the proposed active hypergraph neural network (ACGNN) can achieve better performance on the three HSI datasets.

Big-data analysis by edge/cloud systems is becoming more important. However, when information may lead to personal identification, such information tends to be encrypted and restricted to its owners to ensure privacy protection. The resulting data is often insufficiently detailed to permit useful analysis. As a result, the desired analysis accuracy may not be achieved. To deal with this issue, several studies have examined encryptions based on the random unitary transform. This is because the random unitary transform has lower computational complexity than other encryption schemes, and its encryption domain supports several signal processing algorithms. However, analysis models on distributed encrypted data, have not been studied deeply enough. In this paper, we construct an analysis model for data encrypted with the random unitary transform by deriving a LASSO solution for encrypted data. The analytical model can derive the same LASSO solution as that yielded by processing the original data (i.e. without encryption). The analytical model supports distributed encryption, where a data set consists of different components that are encrypted at different sites independently. The collaboration enables us to improve the accuracy of analysis for distributed privacy-sensitive information.

Designing an optimum quantizer can be treated as the optimization problem of finding the quantization indices that minimize the quantization error. One solution to the optimization problem, DP quantization, is based on dynamic programming. Some applications, such as bit-depth scalable codec and tone mapping, require the construction of multiple quantizers with different quantization levels, for example, from 12bit/channel to 10bit/channel and 8bit/channel. Unfortunately, conventional DP quantization optimizes the quantizer for just one quantization level. That is, it is unable to simultaneously optimize multiple quantizers. Therefore, when DP quantization is used to design multiple quantizers, there are many redundant computations in the optimization process. This paper proposes an extended DP quantization with a complexity reduction algorithm for the optimal design of multiple quantizers. Experiments show that the proposed algorithm reduces complexity by 20.3%, on average, compared to conventional DP quantization.

We studied a coding scheme where light field (LF) images (dense multi-view images) are regarded as a sequence of temporal video frames and encoded with video codecs such as High Efficiency Video Coding (HEVC). An important issue with this scheme is how to determine the frame order of the LF images. We propose a method to find the optimum frame order through a formulation of the traveling salesman problem (TSP). Under the assumption that video codecs are more effective with temporally smooth videos, our method, named LF-TSP, defines frame-to-frame distances for each image pairs in an LF, and attempted to find the shortest route that visits all frames. Experiments showed that our method achieved an overall better rate-distortion performance than several previous methods.

The design of entropy-constrained quantization is formulated as the minimization of quantization error with constraint which gives a maximum amount of information of quantized values. It is known that the optimization of unconstrained quantizer (e.g. a quantizer that minimizes summation of quantization error) is achieved by using approaches based on dynamic programming, which is called DP quantization. However, conventional DP quantization is an approach to optimize a quantizer whose quantization level is fixed. In this paper, we propose a complexity reduction algorithm for an optimal design for entropy-constrained quantizer by extending DP quantization.

The design of an optimum quantizer can be formulated as an optimization problem that finds the quantization indices that minimize the quantization error. One solution of the optimization problem is DP quantization, an approach based on dynamic programming. It is known that a quantized signal does not always contain signal values that can be represented with a given bit-depth. This property is called amplitude sparseness. Because quantization is the amplitude discretization of signal value, amplitude sparseness is closely related to the design of the quantizer. Since signal values with zero frequency do not affect quantization error, there is the potential to reduce complexity when designing the optimum quantizer by skipping the processing of signal values that have zero frequency. However, conventional methods on DP quantization do not design for amplitude sparseness and so are unduly complex. In this paper, we propose an algorithm that yields an optimum quantizer that minimizes quantization error with reduced complexity given the existence of amplitude sparseness.

The acquisition rate of video equipment is advancing rapidly, for example, a video with Full-HD resolution can now be acquired at 1000 Hz. However, the acquisition process of imaging systems is independent from the video encoding process. The imaging system fails to utilize signals acquired at high temporal resolution in an attempt to improve video coding efficiency. This paper proposes a video generation algorithm that uses temporally over-sampled frames as input to produce a temporally down-sampled video signal optimized in terms of video encoding efficiency. The proposed method introduces an adaptive temporal filter whose filter coefficients are selected from adaptively generated set (called a dictionary) of candidates. As an extension of our previous work that studies filter coefficient selection from a fixed dictionary, the proposal of this paper designs the dictionary through Bayesian optimization. The proposed method makes it possible to jointly optimize the filter coefficient selection and the dictionary construction. Experiments show that the proposed method can reduce the encoding rate on average by 4.79 [%] compared to the constant mean-filter.

The acquisition rate of video equipment makes positive advances, for example, a video with Full-HD resolution can be acquired at 1000 Hz. However, the acquisition process in imaging system is independent from video encoding process. This means that the imaging system does not consider to reduce video encoding rate. This paper proposes a video generation algorithm optimized in terms of video encoding. The proposed method features to use a video signal sampled at high temporal resolution. The video signal sampled at high temporal resolution contains much information of captured scene. Utilizing such information for generating video signal, we can pioneer a way to improve coding efficiency outside the conventional research area of video coding. The proposed method applies a temporal filter that features to select filter coefficients from the candidate set. The selection of filter coefficients for each frame is designed based on dynamic-programming in order to minimize total encoding rate of all generated frames. Experimental results show that the proposed method can reduce encoding rate on average by 2.14 [%] compared to constant mean filter.

It is important to reduce intra prediction error for efficient image coding. However, as the existing methods are based on static structure model, there is no guarantee of their prediction efficiency for images whose structure model are unknown. To remedy the problem of existing methods, we formulate a linear prediction design problem with the goal of minimizing prediction error by placing sparsity constraints on the prediction coefficients. To solve the predictor design problem, we propose a novel method that extends Least Angle Regression(LARS). Coding gain of 1.01% to 2.24% is achieved over HM16.7.

Recently, visual representations using high dynamic range (HDR) images become increasingly popular, with advancement of technologies for increasing the dynamic range of image. HDR image is expected to be used in wide-ranging applications such as digital cinema, digital photography and next generation broadcast, because of its high quality and its powerful expression ability. HDR imaging technologies will spread its sphere of influence in imaging industry. In this paper, we review the state-of-the-art studies and the trends of the HDR imaging, in terms of the following three points: (1) HDR imaging sensor and HDR image generation techniques as image acquisition technologies, (2) encode method of HDR images for efficient transmission and storage, (3) human visual system issues associated with reproduction of HDR image.

In recent years, high bit-depth (HBD) images with 10 [bits/channel] or more are being used more often for their improved image quality. However, the size of HBD images become large and so more effective encoding techniques are needed. This paper considers the bit-depth transform process from the view point of minimizing bit-depth transform error. The minimization method is designed according to an optimum quantization algorithm. As a result, our method can improve the coding efficiency by an average of 11 [%] compared to the conventional method with AVC/H.264.

Conventionally, generation methods for the input sequences of a video encoder have been independent from its following encoding process. In other words, such generation method has not been designed in terms of improving coding efficiency, although encoder optimization and pre-filter design have been studied in order to improve coding efficiency. However, it is prospective that we can improve coding efficiency furthermore by designing the generation method based on encoder design. In this study, we propose an algorithm for generating sequences using frames sampled in the temporal high density. Over the past decade, video acquisition rates, which had been 24 Hz (cinema), 30-60 Hz (webcam) or 50-60 Hz (SD/HDcam), has broken through to reach 1000 Hz. The proposed algorithm designs an adaptive temporal down-sampling filter in terms of the reduction of inter-frame prediction error, subject to prevent from the jerki-ness caused by temporal down-sampling. Experimental results show that our approach can reduce the inter-frame prediction error on average by 0.23 [dB] compared to constant mean filter without degrading its subjective image quality.

Motion compensation with quarter-pel accuracy was added to H.264/AVC to improve the coding efficiency of images exhibiting fractional-pel movement. To enlarge the reference pictures, a fixed 6-tap filter is used. However, the values of the filter coefficients are constant regardless of the characteristic of the input video. An improved interpolation filter, called the Adaptive Interpolation Filter (AIF), that optimizes the filter coefficients on a frame-by-frame basis was proposed to solve the problem. However, when the image is divided into multiple regions, each of which has different characteristics, the coding efficiency could be further improved by performing optimization on a region-by-region basis. Therefore, we propose a Region-Based AIF (RBAIF) that takes account of image locality. Simulations show that RBAIF offers about 0.43 point higher coding gain than the conventional AIF.

Transform coding is a essential tool for picture coding applications, and coding schemes that can achieve high energy compaction are essential. In terms of energy compaction, Karhunen-Loeve transform (KLT) is known to be optimal. The energy compaction provided by KLT depends on the statistical property of the input signal and the dimensionality of KLT. However, the quantitative effect of KLT dimensionality on energy compaction has not been clarified. This paper establishes a mathematical model of the relationship among the dimensionality of KLT and the energy compaction of the transform coefficients, using mathematical tools in quantum information theory.

Over the past decade, video acquisition rates, which had been 24 Hz (cinema), 30-60 Hz (webcam) or 50-60 Hz (SD/HDcam), has broken through to reach 1000 Hz. In order to display these high frame-rate video signals on current display devices, they must be down-sampled first. This study proposes a down-sampling method suitable for high frame-rate video signals. It is designed with the goal of reducing the inter-frame prediction error and suppressing jerkiness between sub-sampled frames. Our method can improve the PSNR of prediction signal by 0.10 [dB] to 0.14 [dB] compared to simple sub-sampling with constant interval.

Over the past decade, video acquisition rates, which had been 24 Hz (cinema), 30-60 Hz (webcam) or 50-60 Hz (SD/HDcam), has broken through to reach 1000 Hz. In order to display these high frame-rate video signals on current display devices in real time, they must be down-sampled first. This study proposes a down-sampling method suitable for high frame-rate video signals. It is designed with the goal of reducing the inter-frame prediction error. Our method can improve the PSNR of prediction signal by 0.13 [dB] to 0.23 [dB] compared to simple sub-sampling with constant interval.

It is really important to use the proper coding mode in the H.264 encoder, since it offers many more modes than the conventional methods such as MPEG-2. Typical H.264 encoders like JM and JSVM use squared error as the criterion of distortion for mode decision. However, squared error does not always provide a correct measure of the distortion from the viewpoint of subjective quality. In this paper, we investigate a mode decision algorithm based on the spatio-temporal contrast sensitivity model in order to improve the H.264 encoder. Our algorithm is designed considering the direction dependency of spatio-temporal frequency. Experiments show that our method can achieve average bit-rate savings of the order of 4.0 to 5.5% compared to the original JSVM. We confirm that both methods yield reconstructed images that basically have the same subjective image quality.

Higher frame-rates are being considered to achieve more realistic representations. Since increasing the frame-rate increases the total amount of information, efficient coding methods are required. However, the statistical properties of such data has not been clarified. This paper establishes, for high frame-rate video, a mathematical model of the relationship between frame-rate and bit-rate. The model incorporates the effect of the low-pass filtering induced by shutter open. By incorporating the open interval of shutter, our model can be extended to describes the various cases of downsampling frame-rates. A coding experiment confirms the validity of the mathematical model.

Higher frame-rates are being considered to achieve more realistic representations. Since increasing the frame-rate increases the total amount of information, efficient coding methods are required. However, the statistical properties of such data has not been clarified. This paper establishes, for high frame-rate video, a mathematical model of the relationship between frame-rate and bit-rate. The model incorporates the effect of the integral phenomenon which occurs when the frame-rate is downsampled. A coding experiment confirms the validity of the mathematical model.

Higher frame-rates are being considered to achieve more realistic representations. Since increasing the frame-rate increases the total amount of information, efficient coding methods are required. However, its statistical properties are not clarified. This paper establishes for high frame-rate video a mathematical model of the relationship between frame-rate and bit-rate. A coding experiment confirms the validity of the mathematical model.

The Hilbert curve is a one-to-one mapping between N-dimensional (N-D) space and 1-D space. The Hilbert curve has been applied to image processing as a scanning technique (Hilbert scan). Applications to multi-dimensional image processing are also studied. In this application. We use the N-D Hilbert scan which maps N-D data to 1-D data along the N-D Hilbert curve. However, the N-D Hilbert scan is the application limited to data in a hyper-cube region. In this paper, we present a novel algorithm for generating N-D pseudo-Hilbert curves in a hyper-rectangular parallelepiped region. Our algorithm is suitable for real-time processing and is easy to implement in hardware, since it is a simple and non-recursive computation using look-up tables.

Hilbert curve is a one-to-one mapping between N-dimensional (N-D) space and 1-D space. The Hilbert curve has been applied to image processing as a scanning technique (Hilbert scan). Recently the application to moving-image processing is also studied actively. In this application, we use 3-D Hilbert scan which maps 3-D data to l-D data along 3-D Hilbert curve. However, 3-D Hilbert scan is the application limited to data in a cube region. In this paper, we present a novel algorithm for generating 3-D pseudo-Hilbert curves in a cuboid region. Our algorithm is suitable for real-time processing and easy to implement in hardware, since it is a simple and non-recursive computation using look-up tables.

There have been many applications of the Hilbert curve, such as image processing, image compression, computer hologram, etc. The Hilbert curve is a one-to-one mapping between N-dimensional space and one-dimensional (l-D) space which preserves point neighborhoods as much as possible. There are several algorithms for N-dimensional Hilbert scanning, such as the Butz algorithm and the Quinqueton algorithm. The Butz algorithm is a mapping function using several bit operations such as shifting, exclusive OR, etc. On the other hand, the Quinqueton algorithm computes all addresses of this curve using recursive functions, but takes time to compute a one to-one mapping correspondence. Both algorithms are complex to compute and both are difficult to implement in hardware. In this paper, we propose a new, simple, nonrecursive algorithm for N-dimensional Hilbert scanning using look-up tables. The merit of our algorithm is that the computation is fast and the implementation is much easier than previous ones.

The Hilbert curve is one of the space-filling curves published by G. Peano. There are several applications using this curve such as image processing, computer graphics, etc. We concentrate on a compression technique for color images using the Hilbert curve. The merit of this curve is to pass through all points in a quadrant, and always to move to the neighboring quadrant. Our method is based on this neighborhood property by a simple segmentation of the scanned one dimensional data using a zero order interpolation. From our experiments, we have confirmed that inspite of the simple computation in comparison to JPEG, acceptable quality images can be obtained at the same bit-rates.

The Hilbert curve is one of space filing curves presented by G. Peano in 1890. We apply this curve to scanning an arbitrary sized image for image compression, image processing, etc. In this paper, we propose a new, simple, non-recursive algorithm for pseudo-Hilbert scanning using lookup tables. The merit of our algorithm is that the computation is fast and the hardware implementation is much easier than previous ones. From our experimental results, we have confirmed that our method is about 50 times faster than other methods.