With the advancement of quantum computing, quantum image representations (QIRs) have been widely studied as platforms for quantum image processing. While previous studies have focused on theoretical qubit counts and circuit complexities, there has been limited quantitative evaluation of image reconstruction performance from quantum states. In this study, we clarify the theoretical upper bounds of restoration performance for three representative QIRs: qubit lattice (QL), flexible representation of quantum images (FRQI), and novel enhanced quantum representation (NEQR). Also, we validate them through large-scale GPU-based simulations using cuQuantum. Specifically, we (1) derive a prediction formula for PSNR in probabilistic schemes like QL and FRQI, independent of image content and size; (2) show that deterministic schemes like NEQR require a number of samples following a harmonic number for complete restoration; and (3) perform large-scale simulations to verify these findings. These results provide fundamental insights for designing QIRs and their applications on NISQ devices.

Lookup tables (LUTs) are commonly used to speed up image processing by handling complex mathematical functions like sine and exponential calculations. They are used in various applications such as camera image processing, high-dynamic range imaging, and edge-preserving filtering. However, due to the increasing gap between computing and input/output performance, LUTs are becoming less effective. Even though specific circuits like SIMD can improve LUT efficiency, they still need to bridge the performance gap fully. The gap makes it difficult to choose between direct numerical and LUT calculations. For this problem, a register-LUTs method with the nearest neighbor was proposed; however, it is limited for functions with narrow-range values approaching zero. In this paper, we propose a method for using register LUTs to process images efficiently over a wide range of values. Our contributions include proposing register-LUT with linear interpolation for efficient computation, using a smaller data type for further efficiency, and suggesting an efficient data retrieving method.

Abstract The resolution of cameras is increasing, and speedup of various image processing is required to accompany this increase. A simple way of acceleration is processing the image at low resolution and then upsampling the result. Moreover, when we can use an additional high-resolution image as guidance formation for upsampling, we can upsample the image processing results more accurately. We propose an approach to accelerate various image processing by downsampling and joint upsampling. This paper utilizes per-pixel look-up tables (LUTs), named local LUT, which are given a low-resolution input image and output pair. Subsequently, we upsample the local LUT. We can then generate a high-resolution image only by referring to its local LUT. In our experimental results, we evaluated the proposed method on several image processing filters and applications: iterative bilateral filtering, $$\ell _0$$ ℓ 0 smoothing, local Laplacian filtering, inpainting, and haze removing. The proposed method accelerates image processing with sufficient approximation accuracy, and the proposed outperforms the conventional approaches in the trade-off between accuracy and efficiency. Our code is available at https://fukushimalab.github.io/LLF/ .

Image compression is essential in image processing, and image quality assessment (IQA) is important in determining the image compression level. This study aims to construct a dataset for evaluating image quality at low compression in this coding degradation, i.e., for high-quality images. Typical IQA databases are selected for general-purpose image degradation, not high-quality images. If one tries to evaluate high-quality images, multi-level evaluations are difficult to construct successfully. In addition, the evaluated encoder is not a de facto standard encoding algorithm. Therefore, this study constructs a dataset for subjective evaluation of visually near-lossless level image compression quality based on binary level evaluation of the just noticeable difference (JND). Experimental results showed that the new dataset was validated for correlation by various IQAs. It was also shown that more than the compression quality covered by the conventional dataset is needed for the binary evaluation of high-quality images. The dataset is available at: https://norishigefukushima.github.io/iqanearlossless/.

Multi-scale processing is essential in image processing and computer graphics. Halos are a central issue in multi-scale processing. Several edge-preserving decompositions resolve halos, e.g., local Laplacian filtering (LLF), by extending the Laplacian pyramid to have an edge-preserving property. Its processing is costly; thus, an approximated acceleration of fast LLF was proposed to linearly interpolate multiple Laplacian pyramids. This paper further improves the accuracy by Fourier series expansion, named Fourier LLF. Our results showed that Fourier LLF has a higher accuracy for the same number of pyramids. Moreover, Fourier LLF exhibits parameter-adaptive property for content-adaptive filtering.

Bilateral filtering is a typical edge-preserving smoothing and it is used in various applications. The main issue of bilateral filtering is the processing time. In order to solve this problem, constant-time bilateral filtering has been proposed. The constant-time bilateral filter is an effective method for grayscale images, but it takes high cost for color images because of the curse of dimensionality. Some algorithms specialize in constant-time color bilateral filtering for color images by using clustering. However, the clustering has randomness, and computational cost itself is also high. In this paper, we propose an acceleration method of clustering by using K-means ++, tiling, and subsampling, and also achieve improvement of the stability.

Smoothing filters have been used for pre/post-processing in various fields, such as computer vision and computer graphics. Bilateral filtering (BF) has a typical edge-preserving filter for such applications. The main issue of BF is its computational cost. Constant-time BF (O(1) BF) is one of the solutions to this problem, and compressive BF is a kind of O(1) BF. Compressive BF has, however, a restriction that we can only use Gaussian kernel as a range kernel until now. In this paper, we propose the method to extend compressive BF to handle arbitrary range kernels. Experimental results show that our extension handles arbitrary range kernels, and becomes the number of convolutions into half.

Gaussian filtering is a smoothing filter used in various tasks. The main disadvantage is the dependence of the processing time on its kernel radius. One solution is using a sliding-discreet cosine transform (DCT), a constant-time algorithm for the kernel radius, and it provides the best performance in terms of both speed and accuracy. However, the speed and accuracy differ according to the type of DCT used. We can also accelerate the sliding-DCT based Gaussian filter by hardware accelerators, but the acceleration requires modification of the algorithms. In this paper, we focus on the fused multiply-add (FMA) instruction of hardware accelerators in modern computer architectures. The FMA instruction simultaneously performs multiplication and addition, i.e.,ax+b. We proposed an acceleration method of the sliding-DCT based Gaussian filtering for the FMA instruction. Moreover, we evaluate the performance of it in terms of computational time and approximation accuracy.

In this paper, we propose an acceleration method of redundant DCT filtering for deblurring and denoising. Current CCD cameras have high-resolution images, and the resolution has been increasing. Even if pixels are in focus, the pixels have slight blurring due to diffraction and Bayer interpolation. Therefore, we focus deblurring for slight blurring on real-time performance. Traditional approaches have a fast computational performance for this purpose, but these methods do not contain denoising architecture. In this paper, we simultaneously perform deblurring and denoising on the redundant DCT domain for accelerating the process. Also, we show that a post-scaling DCT can accelerate the proposed filtering. Experimental results show that the proposed method is the fastest method and the accuracy is also high among the fast approaches.

Image filtering is fundamental in image processing. The acceleration is essential since image resolution has highly increased. For the acceleration, image subsampling is a general approach for any filtering. However, this approach has a drawback in accuracy due to image aliasing. Several papers moderate this problem by sub-sampling image or filtering kernel non-sequential sampling. In this paper, we improve the sampling combined with image and kernel subsampling. We also accelerate the work of non-sequential sampling by vector addressing of hardware acceleration. Experimental results show that the proposed method accelerate bilateral filtering and adaptive Gaussian filtering. Also, the proposed vector addressing accelerate both filters.

This paper presents a constant-time bilateral filter that supports arbitrary range kernel designed via singular value decomposition (SVD). Bilateral filter (BF) suffers from high computational complexity in real-time processing due to the time-variant kernel. Although various accelerations for BF have been proposed, most of them have not achieved both arbitrary range kernel and tight computational complexity simultaneously. The proposed method supports arbitrary range kernel but requires half computational complexity of most state-of-the-art methods. Moreover, we present two implementation techniques well matched to the SVD approach: range fitting and tiling strategy. Experiments show that, in the cases of major range kernels, the proposed method not only runs faster (200 FPS) but also achieves higher accuracy than the state-of-the-art methods.

Halide is a domain-specific language for image processing on CPUs and GPUs. The language is powerful for image processing with deep pipeline, e.g., guided image filtering. The guided image filtering is utilized for various applications. Some papers implement the filter on hardware; however, the implementing hardware and the purposes are different. Hardware implementation is hard; therefore, compiler supports are necessary. We utilize Halide with extended FPGA backend, called GENESIS. In our experiment, Halide code with CPU/FPGA backend is faster than the optimized C++. Also, the code length of the C++ and Halide for CPU/FPGA is 575, 141, 139 lines, respectively.

In this paper, we propose an extension of guided image filtering to support arbitrary window functions. The guided image filtering is a fast edge-preserving filter based on a local linearity assumption. The filter supports not only image smoothing but also edge enhancement and image interpolation. The guided image filter assumes that an input image is a local linear transformation of a guidance image, and the assumption is supported in a local finite region. For realizing the supposition, the guided image filtering consists of a stack of box filtering. The limitation of the guided image filtering is flexibilities of kernel shape setting. Therefore, we generalize the formulation of the guide image filter by using the idea of window functions in image signal processing to represent arbitrary kernel shapes. Also, we reveal the relationship between the guided image filtering and the variants of this filter.

In this paper, we propose an infinite impulse response (IIR) filtering with complex coefficients for Euclid distance based filtering, e.g. bilateral filtering. Recursive filtering of edge-preserving filtering is the most efficient filtering. Recursive bilateral filtering and domain transform filtering belong to this type. These filters measure the difference between pixel intensities by geodesic distance. Also, these filters do not have separability. The aspects make the filter sensitive to noises. Bilateral filtering does not have these issues, but it is time-consuming. In this paper, edge-preserving filtering with the complex exponential function is proposed. The resulting stack of these IIR filtering is merged to approximated edge-preserving in FIR filtering, which includes bilateral filtering. For bilateral filtering, a raised-cosine function is used for efficient approximation. The experimental results show that the proposed filter, named IIR bilateral filter, approximates well and the computational cost is low.

In this paper, we carry out QoE (Quality of Experience) assessment of operability of haptic interface device for each player and fairness between players for soft objects in a networked real-time game. We handle a balloon bursting game in which two players burst balloons (i.e., soft objects) in a 3D virtual space by using haptic interface devices. Assessment results show that the operability depends on the network delay, and the fairness is mainly dependent on the difference in network delay between the players' terminals.

In this paper, we propose an accurate approximation framework for separable edge-preserving filtering. Naïve implementation of edge-preserving filtering, such as bilateral filtering and non-local means filtering, consumes enormous computational costs. Separable implementation of such filters is an efficient approximation method for real-time filtering. The accuracy of the conventional separable representation, however, is inadequate when the kernel size is immense. To improve the accuracy, we prepare dual kernels that have different kernel weights for horizontal and vertical filtering of separable filtering. In the experiment, we validate the proposed implementation by using three kinds of filters; bilateral filtering, dual bilateral filtering, and non-local means filtering. Experimental results show that the proposed implementation has higher accuracy while the computational time is almost the same. Moreover, the proposed implementation is practical for denoising and disparity map refinement applications.

We propose a fast post filter set for removing coding distortions of depth maps. Lossy coding generates various distortions, such as mosquito and block noises, edge blurs, and over quantization. These distortions seriously deteriorate image quality of synthesized views in free viewpoint image rendering. Thus, we propose the post filter set which includes median filter, Gaussian filter, min-max blur remove filer, and binary weighted range filter to remove these noises. In experiments, we use various codecs, which are JPEG, JPEG-LS, JPEG2000, and H.264/AVC, for depth map coding, and synthesize views with the coded depth maps. Experimental results show that our post filter set can improves every codecs performance. Improvement of PSNR is larger than the conventional post filter, and especially JPEG is large. The computational time of the filter set, which is implemented by C++ with SIMD optimization, is within 5.2 ms at high bit rate cases, and within 15.3 ms at low bit rate cases.

In this paper, we propose a free viewpoint image rendering method combined with filter based alpha matting for improving the image quality of image boundaries. When we synthesize a free viewpoint image, blur around object boundaries in an input image spills foreground/background color in the synthesized image. To generate smooth boundaries, alpha matting is a solution. In our method based on filtering, we make a boundary map from input images and depth maps, and then feather the map by using guided filter. In addition, we extend view synthesis method to deal the alpha channel. Experiment results show that the proposed method synthesizes 0.4 dB higher quality images than the conventional method without the matting. Also the proposed method synthesizes 0.2 dB higher quality images than the conventional method of robust matting. In addition, the computational cost of the proposed method is 100x faster than the conventional matting.

In this paper, we propose dynamic control of output timing of fragrance in a networked virtual environment to improve Quality of Experience (QoE). The control changes the output timing of fragrance dynamically according to the movement speed and direction of an object which is a fragrance source. We implement the control in a remote ikebana system using olfactory and haptic media and investigate the effect of the control by QoE assessment. As a result, we demonstrate that Mean Opinion Score (MOS) in the case where the control is used is high and does not largely depend on the average movement speed and movement direction of a flower, which is a fragrance source.

In this paper, we handle an air hockey game which two users play with haptic interface devices as an example of networked real-time games. In order to improve the interactivity of the air hockey game, we propose the adaptive Δ (delta)-causality control scheme with prediction, which keeps the consistency and causality high and predicts positional information after a constant time. In QoE (Quality of Experience) assessment, we assess the output quality and interactivity of local mallet, output quality of opponent's mallet, output quality and interactivity of puck, and comprehensive quality. As a result, we illustrate that the proposed scheme is effective and there exists the optimal value of the prediction time.

Free viewpoint video attracts many researchers. With the video, users can change viewpoint of the video freely at the user side. Transmitting such media over IP network, the network delay deteriorates qualities of the media. In this paper, we investigate the influence of network delay in free-viewpoint video transmission by QoE (Quality of Experience) assessment. We address two transmission methods for free-viewpoint normal and stereoscopic videos. One is the synthesized image transmission method, and the other is the depth and image transmission method. Depth and image are data for synthesizing an image. We assess the image quality, the interactivity of viewpoint change, and the comprehensive quality. Assessment results demonstrate that the image quality of the synthesized image transmission method is higher than that of the depth and image transmission method, which is advantageous in terms of interactivity. The results also illustrate that the free-viewpoint stereoscopic video has higher image quality than the normal one.

In this paper, we investigate the influence of network delay on the quality of experience (QoE) for a tele-operation system with 3D (i.e., stereoscopic) video and haptic media. We assess the video output quality, operability of the haptic interface device, inter-stream synchronization quality (or inter activity), and comprehensive quality as QoE. We also evaluate the application-level quality of service (QoS). By multiple regression analysis, we demonstrate that it is possible to estimate QoE parameters from QoS parameters with a high degree of accuracy.

The availability of multi-view images of a scene makes new and exciting applications possible, including Free-Viewpoint TV (FTV). FTV allows us to change viewpoint freely in a 3D world, where the virtual viewpoint images are synthesized by Image-Based Rendering (IBR). In this paper, we introduce a FTV depth estimation method for forward virtual viewpoints. Moreover, we introduce a view generation method by using a zoom camera in our camera setup to improve virtual viewpoint-ts' image quality. Simulation results confirm reduced error during depth estimation using our proposed method in comparison with conventional stereo matching scheme. We have demonstrated the improvement in image resolution of virtually moved forward camera using a zoom camera setup.

In this paper, we propose a method of free viewpoint image generation with super resolution. In the conventional approaches, such as nearest neighbor and linear interpolation, the synthetic image on zoomed virtual view tends to have low resolution, because the reference images do not have enough textures. To overcome this problem, we reconstruct the image with super resolution. Super resolution can generate higher image resolution than the input image one, and then we combine super resolution with free viewpoint image generation. In the experiment, we use a camera array which contains 11 × 11 aligned cameras and use 4 × 4 cameras subset per pixel to reconstruct image by means of super resolution. The experimental results show that synthesized image in the effective range has about 4.5 dB higher PSNR than ones created by the nearest neighbor and 2.5 dB higher than ones created by the linear interpolation.

This paper analyzes the influences of frame delay and packet loss on stereoscopic vision when stereoscopic video is transferred over a IP network. We employ live action videos which are transferred to a head-mount-display (HMD) and do the assessment on stereoscopic perception. As a result, we found that speed and movement direction of the attention object play a great role on the deterioration when frame delay and packet loss exist.

In this paper, we propose a parallel processing method to generate free viewpoint image in realtime. It is impossible to arrange the cameras in a high density realistically though it is necessary to capture images of the scene from innumerable cameras to express the free viewpoint image. Therefore, it is necessary to interpolate the image of arbitrary viewpoint from limited captured images. However, this process has the relation of the trade-off between the image quality and the computing time. In proposed method, it aimed to generate the high-quality free viewpoint image in realtime by applying the parallel processing to time-consuming interpolation part.

In this paper, we propose a semi-automatic depth estimation algorithm whereby the user defines object depth boundaries and disparity initialization. Automatic depth estimation methods generally have difficulty to obtain good depth results around object edges and in areas with low texture. The goal of our method is to improve the depth in these areas and reduce view synthesis artifacts in Depth Image Based Rendering. Good view synthesis quality is very important in applications such as 3DTV and Free-viewpoint Television (FTV). In our proposed method, initial disparity values for smooth areas can be input through a so-called manual disparity map, and depth boundaries are defined by a manually created edge map which can be supplied for one or multiple frames. For evaluation we used MPEG multi-view videos and we demonstrate our algorithm can significantly improve the depth maps and reduce view synthesis artifacts.

In this paper, we propose two novel methods for arbitrary listening-point generation for 3D audio-video (3DAV) integration in a large-scale multipoint cameras and microphones system with abilities to process, and display information of any recorded 3D scene in realtime. With this system, users are able to control their own viewpoint/listening-point position, freely. Arbitrary listening-point can be generated by either (i) ray-space representation of sound wave field (i.e. source sound independent) for multi frequency layers, or (ii) acoustic transfer function estimation (i.e. source sound dependent) and blind separation of sources of sounds. Arbitrary viewpoint generation is based on ray-space method, which is enhanced by using multipass dynamic programming for geometry compensation. Integration is done by either (i) ray-space representation of sound wave and image together, or (ii) integrating each camera video signal and acoustic transfer function of the same location as integrated 3DAV data. The prototype system of integrated audio-visual viewer achieves both good image and sound qualities with 15 frames/second.