Quanta image sensors are an emerging type of image sensor offering single-photon sensitivity. In this paper, we propose a deep unfolding-based image reconstruction method that integrates an alternating direction method of multipliers optimization with a total variation prior. The proposed method effectively combines model-based approaches with deep learning, providing enhanced interpretability and performance. Experimental results demonstrate that our approach outperforms conventional methods based on the observation model for quanta image sensors.

Recent dense multi-view 3D systems may provide consistently augmented reality, where we enjoy a simple solution of occlusion caused by visual obstructions in the real world, as if they were transparent for many users at the same time. In this paper, we propose a novel real-time method to reconstruct dense 4D light fields from RGB-D data for such virtual transparency of visual obstructions. First, we discuss appearances of our proposed multi-view system, while examining how densely 4D light fields should be reproduced. Then, in order to develop an inexpensive system, we propose real-time reconstruction of the entire 4D light field from point cloud data acquired by a single RGB-D camera instead of massive camera arrays. Actually, by applying appropriate geometrical transformation to the acquired RGBD data, we obtain a simple 3D array while effectively reducing their redundancy. Moreover, it allows us to achieve structured analysis of the scene for reconstructing a 4D light field composed of equally-spaced multi-view images, where its computational cost is significantly suppressed.

Heart rate (HR) is a key parameter in evaluating the physiological and emotional states of a person. In this paper, we propose a novel video-based heart rate (HR) estimation method based on physiological modeling with multispectral imaging. To capture blood volume pulse (BVP) associated with a person’s heartbeat, we utilize a camera that records multispectral video consisting of red, green, blue, and near-infrared information. The novelty of the proposed method is the incorporation of a physiological BVP model into a multispectral HR estimation framework. The integration of a physiological model-based BVP signal extraction scheme into an adaptive multispectral framework enables the suppression of noise derived from ambient light and the accurate extraction of the BVP signal, thereby enhancing HR estimation performance. The experiments using RGB/NIR video datasets demonstrate the effectiveness of the proposed method.

Quanta image sensors are a novel paradigm in image sensor technology. Their direct application to quanta image sensors-based imaging systems is challenging because a bit-plane image is a set of binary images. In this paper, we introduce spatio-temporal priors based on the intensity invariance and smoothness characteristics of the motion vector. Specifically, we model when the image sequences align with the correct motion vector, the spatiotemporal structure becomes more consistent. Moreover, the spatial smoothness prior is incorporated through the smoothing filtering of the evaluation metrics of motion vector candidates. The experimental results show that the proposed method is more effective than conventional methods.

We previously implemented an inexpensive imaging system that combines a single real camera with a mirror array located along a paraboloid. It allows us to robustly acquire dynamic light fields composed of multi-view videos by providing a virtual camera array, where its viewpoints exist in the mirrors. Actually, as moving the real camera to the focus of the paraboloid, virtual viewpoints in the mirrors get equally-spaced to achieve multi-view imaging with structured disparity. In this paper, we discuss an efficient method for adjusting the pose of a single camera to acquire high quality dynamic light fields as multi-view videos. Specifically, we introduce some indicator values determined by detected corners of the mirror array on acquired images while adjusting the camera. By using these values for camera adjustment, we easily know how to move its position and virtually correct its angle through homography transform. Experimental results of simulations demonstrate that our proposed method sufficiently achieves structured light field video acquisition with equally-spaced virtual viewpoints, where we do not need camera rotation requiring complex devices and only the camera position is controlled by a simple 3D system like XYZ stages.

A light field (LF) image is composed of multiple sub-aperture images (SAIs) from slightly offset viewpoints. To denoise a noise-corrupted LF image, leveraging recent development in deep algorithm unfolding, we pursue a hybrid graph-model-based / data-driven approach. Specifically, we first connect each pixel in a target patch of an SAI to neighboring pixels within the patch, and to pixels in co-located "similar" patches in adjacent SAIs. Given graph connectivity, we formulate a maximum a posteriori (MAP) problem using graph total variation (GTV) as signal prior. We then unroll the iterations of a corresponding optimization algorithm into a sequence of neural layers. In each unrolled layer, we learn relevant features per pixel from data using a convolutional neural net (CNN) in a supervised manner, so that edge weights can be computed as functions of feature distances. Each neural layer can be interpreted as a graph low-pass filter for a 4D LF image patch. Experiments show that our proposal outperformed two model-based and two deep-learning-based implementations in numerical and visual comparisons.

We propose a novel blood volume pulse (BVP) signal extraction method for heart rate estimation that incorporates the self-similarity properties of BVP in the spatial and temporal domains. The main novelty of the proposed method is the incorporation of the temporal self-similarity of BVP via low-rank approximation in the time-delay coordinate system for BVP signal extraction. To make a low-rank approximation of BVP in the time domain, we introduce knowledge of linear time-invariant systems, i.e., the autoregressive (AR) model lies in the low-rank subspace in the time-delay coordinate system. In the medical field, it is widely known that BVP has quasi-periodic temporal characteristics owing to the cardiac pulse and exhibits self-similarity properties in the temporal domain. Hence, we model the temporal behavior of BVP as an AR process, allowing for a low-rank approximation of BVP in the time-delay coordinate system. Low-rank approximation of BVP in the time and spatial domains enables reliable BVP signal extraction, resulting in accurate heart rate estimation. The experiments demonstrate the effectiveness of the proposed method.

We propose a novel method of light field compression using multi-focus images and reference views. Light fields enable us to observe scenes from various viewpoints. However, it generally consists of 4D enormous data, that are not suitable for storing or transmitting without effective compression at relatively low bit-rates. On the other hand, 4D light fields are essentially redundant because it includes just 3D scene information. While robust 3D scene estimation such as depth recovery from light fields is not so easy, a method of reconstructing light fields directly from 3D information composed of multi-focus images without any scene estimation is successfully derived. Based on the method, we previously proposed light field compression via multi-focus images as effective representation of 3D scenes. Actually, its high performance can be seen only at very low bit-rates, because there exists some degradation of low frequency components and occluded regions on light fields predicted from multi-focus images. In this paper, we study higher quality light field compression by using reference views to improve quality of the prediction from multi-focus images. Our contribution is twofold: first, our improved method can keep good performance of 4D light field compression at a wider range of low bit-rates than the previous one working effectively only for very low bit-rates; second, we clarify how the proposed method can improve its performance continuously by introducing recent video codec such as HEVC and VVC into our compression framework, that does not depend on 3D-SPIHT previously adopted for the corresponding component. We show experimental results by using synthetic and real images, where quality of reconstructed light fields is evaluated by PSNR and SSIM for analyzing characteristics of our novel method well. We notice that it is much superior to light field compression using HEVC directly at low bit-rates regardless of its light field scan order.

We propose a non-contact heart rate (HR) estimation method that is robust to various situations, such as bright, low-light, and varying illumination scenes. We utilize a camera that records red, green, and blue (RGB) and near-infrared (NIR) information to capture the subtle skin color changes induced by the cardiac pulse of a person. The key novelty of our method is the adaptive fusion of RGB and NIR signals for HR estimation based on the analysis of background illumination variations. RGB signals are suitable indicators for HR estimation in bright scenes. Conversely, NIR signals are more reliable than RGB signals in scenes with more complex illumination, as they can be captured independently of the changes in background illumination. By measuring the correlations between the lights reflected from the background and facial regions, we adaptively utilize RGB and NIR observations for HR estimation. The experiments demonstrate the effectiveness of the proposed method.

Group-level emotion recognition is a technique for estimating the emotion of a group of people. In this paper, we propose a novel method for group-level emotion recognition. Our method lies in the two-fold contributions: (1) recognition of group-level emotion using a hierarchical classification approach; (2) incorporation of novel features to contribute to the description of the group-level emotion. We consider that the use of facial expressions of people will only be effective in differentiating images labeled as “Positive” because those labeled as “Neutral” or “Negative” are likely to include similar facial expressions. Therefore, we first perform binary classification based on facial expression recognition to distinguish “Positive” labels that include discriminative facial expressions (e.g., smile) from the others. We evaluate outcomes that are not classified as “Positive” during the first classification by exploiting scene features that describe what type of events (e.g., demonstration or funeral) are shown in the image. The other novelty of our method lies in two-fold. The first is the exploitation of visual attention for the first classification. It allows us to estimate which faces are the main subjects in the target image, thereby suppressing the influences of faces in the background that contribute less to group-level emotion. The second is the exploitation of object-wise semantic information (labels) for the second classification. This allows a more detailed description of the scene context in the image and enables performance enhancement in the second classification. We demonstrate the effectiveness of our method through experiments using public datasets.

We propose a novel method of light field denoising for 3D consistent visualization. Light field data are often significantly degraded by awful noise due to lack of exposure under insufficient lighting, especially, if acquired by a lens array. Then, recent studies notice that a 4D light field corresponds to structured multi-view images that are correlated to each other well according to 3D scenes. For example, a CNN-based denoising method utilizes the correlation by analyzing various subspaces of 4D light fields. In this paper, 4D light field denoising is achieved by applying simple restoration based on Total Variation minimization to small images composed of corresponding pixels extracted from multi-view images. We show experimental results using synthetic and real images to clarify how our proposed method effectively works for 3D consistent visualization in comparison with the other methods. In addition, we discuss possible future improvement of the proposed method when integrating our novel approach with the conventional 2D image denoising for more effective light field denoising.

Image colorization based on numerical modeling gives a highly accurate restoration result when colors are given to enough regions. A lot of numerical models focus on the relation between adjacent pixels; therefore, it is required to give the same color to various regions, and high spatial frequency regions are not colored properly. This letter proposes a colorization algorithm using graph signal processing. The key novelty of our algorithm is a new modeling method for a chrominance image using two different graphs. The first graph is a global graph, which connects the important pixels on an image. The second graph is a local graph, which connects the global graph and each pixel. Based on the hierarchical combination of these two graphs, color image is recovered. Numerical experiments show the effectiveness of the proposed algorithm by comparing with four existing methods.

Three-dimensional (3D) object detection in point clouds is an important technique for various high-level computer vision tasks. In this study, we propose a method for point-wise detection of regions of objects in a scene. We regard the 3D object detection problem as a series of optimal matching problems between object and scene images, which are obtained by projecting point clouds into multiple viewpoints. The main novelty of this study is treating the 3D object detection problem as the determination of optimal correspondence among image sets. Unlike the existing methods that directly employ individual correspondences between projected image pairs, the simultaneous matching of projected image sets allows the evaluation of the appearance consistency of the target object in multi-viewpoint scene images. The other novelty of the proposed method is using principal component analysis to estimate effective image-projection directions for object point clouds. By projecting object point clouds in directions orthogonal to the first principal component basis, the projected images can include plenty of point clouds information, thus providing highly discriminative features for image matching. We back-project reliable matching results retrieved from the image-set correspondence into 3D space to achieve point-wise object detection. Experiments using public datasets demonstrate the effectiveness and performance of the proposed method.

We propose a method for group-level emotion recognition in the wild. The main novelty of our method lies with the recognition of group-level emotions using a hierarchical classification approach. We consider that using the facial expressions of people will only be effective in differentiating images labeled as "Positive" because those labeled as "Neutral" or "Negative" are likely to include similar facial expressions (i.e., less discriminative). Therefore, we first perform binary classification based on facial expression recognition to distinguish "Positive" labels that include discriminative facial expressions (e.g., smile) from the others. We evaluate outcomes that are not classified as "Positive" at the first classification by exploiting scene features that describe what type of events (e.g., demonstration or funeral) are taking place in the image. Classification using scene features will not only be effective in differentiating "Negative" and "Neutral" labels but also in recognizing "Positive" labels, where facial expression features show less discriminative characteristics. The other novelty of the proposed method is to the exploitation of visual attention. Using visual attention allows us to estimate which faces are the main subjects in the target image, thereby suppressing the influences of faces in the background that contribute less to group-level emotion. We demonstrate the effectiveness of our proposed method through experiments using a public dataset.

Lens arrays enable us to easily achieve inexpensive light field acquisition in comparison with massive camera arrays. However, their micro-lenses often cause insufficient exposure to obtain high quality multi-view videos like camera arrays. In this paper, we propose a novel virtual camera array acquiring dynamic light fields as multi-view videos robustly and inexpensively, where mirror arrays consisting of tangent planes to a paraboloid are effectively combined with a single video camera. First, we demonstrate multi-view imaging of not an object but ordinary scenes with sufficient exposure by using a simple horizontal mirror array implemented as a prototype. Next, we theoretically reveal that there exists smooth extension of our mirror array to the vertical direction, that can be implemented easily by 3D printing, CNC machines, and so on. Finally, our design of light field acquisition is improved for effectively utilizing the whole of its mirror array by introducing an one-way mirror to prevent the camera itself from appearing on multi-view images.

We propose a method for heart rate (HR) estimation that is robust to various situations such as bright, low-light, and varying illumination scenes. We capture temporal variations in the pixel values owing to person's cardiac pulse by using a camera that records red, green, and blue (RGB) and near-infrared (NIR) information. The key novelty of our method is to introduce a scheme for adaptive fusion of RGB and NIR signals for HR estimation, by analyzing variations in the background illuminations. RGB signals will be a good cue for HR estimation under bright scenes. In contrast, NIR signals are more reliable in HR estimation than RGB ones in complex illumination scenes, because NIR signals can be captured independent to changes in the background illuminations. By measuring correlations of signals between background and face regions, we adaptively utilize RGB and NIR signals for HR estimation. Experiments demonstrate the effectiveness of our method.

Colorization-based image coding is a technique to compress chrominance information of an image using a colorization technique. The conventional algorithm applies graph Fourier transform to the colorization-based coding. In this algorithm, several pixels on the image are defined as vertices of the graph, and the chrominance values of that pixels are set as graph signals. Then, the graph signal corresponding to the several chrominance values on the image is transformed to the graph spectrum based on the graph Fourier transform, and the graph spectrum is compressed and stored. Because the stored graph spectrum gives the graph signal on the image based on the inverse graph Fourier transform in decoding phase, the color image is recovered from the luminance image and the several chrominance values corresponding to the graph signal. However, high calculation time is required to perform graph Fourier transform, and therefore, this paper proposes a fast graph Fourier transform to improve the conventional colorization-based image coding algorithm. In numerical examples, although the PSNR value is decreased 0.3 dB, the proposed algorithm is 16.8 times faster than the conventional method.

We propose a method for detecting objects that correspond to given three-dimensional (3D) point clouds in a scene. We regard the 3D object detection as a series of optimal matching of the object and scene images that are obtained by projecting point clouds into multiple viewpoints. The key novelty of the proposed method is to introduce a constraint imposed by the spatial relationship among the image-projection directions for the object point clouds, to discover the optimal matching of the projected image sets. This constraint allows to evaluate the appearance consistency of the object in multi-viewpoint scene images. Thus, image-projection directions can be effective cues to detect objects even in cluttered scenes, where previous methods are not effective. We estimate the image-projection directions for the object point clouds by applying principal component analysis to the object point clouds and hence include highly discriminative image features. Then, we back-project reliable matching results, which are retrieved from the image set correspondence, into 3D space to achieve a point-wise object detection. Experiments using public datasets demonstrate the effectiveness and performance of the proposed method.

We propose a method for super-resolution (SR) of low-resolution (LR) color images taken in low-light scenes. Our method is based on multi-frame SR technique, which fuses multiple LR images taken at different camera positions to synthesize a high-resolution color image. Previous methods have implicitly assumed that LR images could be captured with less noise and blur. However, heavy noise and motion blur will be imposed on images taken in low-light scene. They make it difficult to super-resolve low-light images with high quality. To overcome this problems, we utilize a single sensor that captures red, green, blue (RGB) and near-infrared (NIR) information. Since NIR images taken using an NIR flash unit can be captured with less noise, they contribute to effective reduction of image artifacts in super-resolving LR images. Using multiple RGB/NIR raw images, we jointly perform deblurring, denoising and SR of LR images. Experiments using real raw data demonstrate the effectiveness of our method.

Absorption and scattering of light in an underwater scene saliently attenuate red spectrum components. They cause heavy color distortions in the captured underwater images. In this letter, we propose a method for color-correcting underwater images, utilizing a framework of gray information estimation for color constancy. The key novelty of our method is to utilize exposure-bracketing imaging: a technique to capture multiple images with different exposure times for color correction. The long-exposure image is useful for sufficiently acquiring red spectrum information of underwater scenes. In contrast, pixel values in the green and blue channels in the short-exposure image are suitable because they are unlikely to attenuate more than the red ones. By selecting appropriate images (i.e., least over- and under-exposed images) for each color channel from those taken with exposure-bracketing imaging, we fuse an image that includes sufficient spectral information of underwater scenes. The fused image allows us to extract reliable gray information of scenes; thus, effective color corrections can be achieved. We perform color correction by linear regression of gray information estimated from the fused image. Experiments using real underwater images demonstrate the effectiveness of our method.

In this paper, we propose a novel method restoring multi-focus images based on convex optimization with new constraint for fixed pattern noise. Even weak fixed pattern noise on multi-focus images degrades all-in-focus images reconstructed by linear combination of them, especially, when using telecentric optical systems such as microscopes. Our novel method introduces constraint for additive fixed pattern noise into total variation minimization and then it is improved for multiplicative fixed pattern noise. The proposed method suppresses fixed pattern noise on multi-focus images very robustly to avoid such degradation on reconstructed images. Experimental results show that our method achieves high performance compared to simple total variation minimization.

We propose a method for disparity estimation in stereo video. We address the problems associated with spatially-temporally-correlated disparity variations (STCDV). STCDV problems are caused by complex motions, e.g., yaw-rotation, pan-tilt-zoom camera movements, etc. The key novelty of this study is to introduce a spatio-temporal disparity hyperplane (STDH) model. The proposed STDH model represents a hyperplane defined in four-dimensional space spanned by disparity, image plane, and time coordinates. Our STDH model is represented by surface normals varying with the spatially-temporally-correlated changes in disparity. Thus, our STDH model is effective in estimating disparity in a stereo video including STCDVs. We estimate video disparity by incorporating our STDH model into the PatchMatch brief propagation framework. Our experiments demonstrate that the proposed method outperforms other methods.

Color images taken in low light scenes are deteriorated with noise and motion blur. The simultaneous reduction of noise and motion blur from the low-light color images is difficult because the imposed noise hinders accurate motion blur kernel estimation. To overcome this problem, we build a novel imaging system using a single sensor that captures red, green, blue (RGB) and near-infrared (NIR) images. Our imaging system captures low-light scenes with exposure bracketing, which is a technique to acquire multiple images with different exposure times. It thus allows us to obtain the short- and long-exposure RGB/NIR images. Both the short- and long-exposure NIR images taken using an NIR flash unit can be captured with less noise; thus they enable estimation of motion blur kernel accurately. Based on this fact, we perform joint denoising and deblurring of the low-light color image with the estimated motion blur kernel. Our experiments using real raw data captured by our imaging system demonstrate the effectiveness of our method.

We propose a method for color correction of underwater images based on multi-illuminant estimation. We regard the color distortion of underwater images as the color cast that is illuminated by multiple light sources. In order to effectively remove the color distortions from the underwater image, we capture underwater scenes by exposure bracketing imaging. Using multiple images taken with different exposure times, we fuse an image where the attenuation differences in the spectra information of the incoming light are mitigated. We apply a multi-illuminant estimation to the fused image to reconstruct the underwater images so as to be those in a canonical (white) illumination environment. Our experiments demonstrate the effectiveness of our method.

We propose a method for depth upsampling with the aid of high-resolution color image. The key novelty of our method is to exploit a spatio-temporal coherency between the color and depth image sequences. It allows us to perform a depth prediction using the responses from motion estimation in the color image sequence. The predicted depth image is able to estimate the scene boundary regions in the color image; it enables to suppress the influences of color image textures in depth upsampling. We synthesize high-resolution depth images with the help of the estimated scene boundary. Our experiments demonstrate the effectiveness of our method.

Software projects collect and deduplicate vastly numerous crash reports from users to fix bugs efficiently.However, most existing automated methods have performance issues during large-scale clustering.We propose a rerankingbased crash report clustering method.Our method is a combination of two earlier methods.By computing similarity used in ReBucket for the crash reports that are highly similar to the query crash report, the method can process reports with throughput equal to that of PartyCrasher.We also introduce an automatically generated dataset for crash report clustering tasks.The evaluation revealed that our method performs at high processing speed while maintaining high accuracy.

In this paper, we propose a novel method for light field imaging. Previous systems are difficult to obtain multi-viewpoint images (sub-images) at the high-resolution. In order to overcome this problem, we propose a two-layer light field imaging system by using an organic photoelectric conversion film (OPCF). Our imaging system places the OPCF having the green spectral sensitivity onto the micro-lens array of the conventional light field camera. It allows us to obtain the green spectrum information at the full-resolution of the image sensor. In contrast, the other spectra information (red and blue) are coded by the optical system of the light field camera, and are recorded by the image sensor. Using the captured images, we synthesize the full-resolution sub-images. Our experiments using synthetic images demonstrate that our method outperformed other previous methods.

If scene flow expressed in three-dimensional (3D) vector fields is robustly estimated from multi-view or multi-focus images, we can develop advanced 3D motion tracking and motion compensation for 3D video compression. In this study, based on a synthesis of multi-focus images from multi-view images, we propose a novel method for analyzing 3D scene flow accurately at low computational cost as an extension of 2D optical flow estimation. Our method directly estimates 3D scene flow in multi-focus images arranged in a direction orthogonal to them by an extended pyramidal Lucas-Kanade method for 3D. Experimental results show that scene flow estimation is achieved efficiently when using our proposed method. In addition, our method can be easily implemented on a graphics processing unit for real-time applications.

We propose a method for pedestrian detection from aerial images captured by unmanned aerial vehicles (UAVs). Aerial images are captured at considerably low resolution, and they are often subject to heavy noise and blur as a result of atmospheric influences. Furthermore, significant changes to the appearance of pedestrians frequently occur because of UAV motion. In order to address these crucial problems, we propose a cascading classifier that concatenates a pre-trained classifier and an online learning-based classifier. We construct the first classifier using deep belief network (DBN) with an extended input layer. Unlike previous approaches that use raw images as the input layer of the DBN, we exploit multi-scale histogram of oriented gradients (MS-HOG) features. The MS-HOG enables us to supply better and richer information than low-resolution aerial images for constructing a reliable deep structure of DBN, because the dimensions of the input features can be expanded. Furthermore, the MS-HOG effectively extracts the necessary edge information while reducing trivial gradients and noise. The second classifier is based on online learning, and it uses predictions of the target appearance using UAV motions. Predicting the target appearance enables us to collect reliable training samples for the classifier’s online learning process. Experiments using aerial videos demonstrate the effectiveness of the proposed method.

In this paper, we propose a method to enhance the color image of a low-light scene by using a single sensor that simultaneously captures red, green, blue (RGB) and near-infrared (NIR) information. Typical image enhancement methods require two cameras to simultaneously capture color and NIR images. In such cases, meticulous calibration is required to adjust the pixel positions of the two cameras. By contrast, our proposed system is calibration free, but achieves accurate color image restoration. We divide the captured multi-spectral data into RGB and NIR information based on the spectral sensitivity of our imaging system. Using the NIR information for guidance, we reconstruct the corresponding clear color image based on a joint demosaicking and denoising technique. Our experiments show the effectiveness of our method using raw data captured by our imaging system.

We propose a novel method to synthesize a noise- and blur-free color image sequence using near-infrared (NIR) images captured in extremely low light conditions. In extremely low light scenes, heavy noise and motion blur are simultaneously produced in the captured images. Our goal is to enhance the color image sequence of an extremely low light scene. In this paper, we augment the imaging system as well as enhancing the image synthesis scheme. We propose a novel imaging system that can simultaneously capture the red, green, blue (RGB) and the NIR images with different exposure times. An RGB image is taken with a long exposure time to acquire sufficient color information and mitigates the effects of heavy noise. By contrast, the NIR images are captured with a short exposure time to measure the structure of the scenes. Our imaging system using different exposure times allows us to ensure sufficient information to reconstruct a clear color image sequence. Using the captured image pairs, we reconstruct a latent color image sequence using an adaptive smoothness condition based on gradient and color correlations. Our experiments using both synthetic images and real image sequences show that our method outperforms other state-of-the-art methods.

We have been investigating a depth estimation system for use in automobile applications. Conventional methods with stereo camera are too sensitive to slight variations of baseline length due to vibration and temperature. Conversely, the method that uses a monocular camera by focusing cannot provide a balance between wide-area estimation and realtime estimation. Therefore, we proposed a novel method that adopts tilted lens optics. In this method, the plane of sharp focus (POF) lies and the depth of field (DOF) enlarges toward the depth direction. Herein, we can obtained depth values at each pixel from the sharpness values of only two tilted optics images using monocular camera system with spectroscopic.

Noise and blur are annoying, common problems in low-light photography. In this paper, we propose a novel framework to reconstruct a blur and noise-free color image sequence using near-infrared (NIR) images. In extremely low light, previous works may fail in color image restoration because both heavy noise and motion blur are produced simultaneously. To overcome these essential problems, we augment both the imaging system and image synthesis method. Our imaging system captures the color and NIR images with different exposure times. Capturing color image with a long exposure time allows us to mitigate the heavy noise. In contrast, the NIR images are taken with a short exposure time to measure the structure of the scene. By leveraging the captured image pairs, we reconstruct a latent color image sequence using a scale map and color correlation. Our experiments on actual images show that our method outperforms other state-of-the-art methods under extremely low-light condition.

Denoising is important in image processing because degradation by noise affects not only the quality of captured images but also the performance of visual applications that use them. For example, under low light levels, it is difficult to accurately estimate scene depths using noisy stereo images. Conventional methods for denoising find similar regions on an image or among multiple images by block matching(BM) to integrate them for suppressing noise effectively. However, such exhaustive BM incurs considerable costs for real-time applications, in particular, when multi-view images(MVI) are involved. We use view-dependent plane sweeping(PS) for image reconstruction to achieve effective MVI denoising with low computational cost. We use PS for converting MVI to multi-focus images(MFI) to suppress their noise. Then, we find regions in focus on the MFI solely by comparing them with the target view image. Finally, we simply merge the regions to obtain reconstructed images in which their noise is effectively suppressed.

Light-Field enables us to observe scenes from free viewpoints. However, it generally consists of 4-D enormous data, that are not suitable for storing or transmitting without effective compression. 4-D Light-Field is very redundant because essentially it includes just 3-D scene information. Actually, although robust 3-D scene estimation such as depth recovery from Light-Field is not so easy, we successfully derived a method of reconstructing Light-Field directly from 3-D information composed of multi-focus images without any scene estimation. On the other hand, it is easy to synthesize multi-focus images from Light-Field. In this paper, based on the method, we propose novel Light-Field compression via synthesized multi-focus images as effective representation of 3-D scenes. Multi-focus images are easily compressed because they contain mostly low frequency components. We show experimental results by using synthetic and real images. Reconstruction quality of the method is robust even at very low bit-rate.

Light-Field enables us to observe scenes from free viewpoints. However, it generally consists of 4-D enormous data, that are not suitable for storing or transmitting without effective compression. 4-D Light-Field is very redundant because essentially it includes just 3-D scene information. Actually, although robust 3-D scene estimation such as depth recovery from Light-Field is not so easy, a method of reconstructing Light-Field directly from 3-D information composed of multi-focus images without any scene estimation is successfully derived. Previously, based on the method, Light-Field compression via synthesized multi-focus images as effective representation of 3-D scenes was proposed. In this paper, we study efficient compression of multi-focus images synthesized from dense Light-Field by using DWT instead of DCT-based compression in order to suppress degradation such as block noise. Quality of reconstructed Light-Field is evaluated by PSNR and SSIM for analyzing characteristics of residuals. Experimental results reveal that our method is much superior to Light-Field compression using disparity-compensation at low bit-rate.

We previously proposed a method of generating free viewpoint images directly from multi-focus imaging sequences without any depth estimation. It is very effective for the method to be implemented to hardware such as FPGA. However, the number of BlockRAMs on our FPGA limits the image size to 64 × 64 pixels. In this paper, we extend our FPGA-based free viewpoint image reconstruction systems by using an onboard DDR SDRAM and processing the divided blocks of 64 × 64 pixels repeatedly. The system realizes our proposed method even for larger image sizes without great drawbacks. Some experimental results by using synthetic images are shown.

We have been investigating an image sensor in which the 2D motion vector of each pixel is detected at very high frame rates such as those greater than 1000 frames per second. In this sensor, the motion vector is detected by single pixel matching with a limited search range and high inter-frame correlation at a high frame rate. In this paper, we have proposed a new method of on-sensor motion detection for noise reduction and improvement of motion direction accuracy. In this method, we implemented two new functions on the sensor. We fabricated the prototype chip using a 0.35 um CMOS process. Furthermore, we explained the methods for estimating the motion trajectory of this imaging device by using the output from the sensor and for estimating the motion direction of a moving object along the depth direction; these two methods are the applications of this sensor. The simulation results show that the proposed on-sensor motion vector detection is more effective than our previously proposed method.

In this paper, we discuss a method of distance and relative speed estimation for ITS by using a certain amount of focus blur. In this method, we use different focus positions of two cameras for obtaining the amount of focus blur. Next, we propose the method of distance estimation by the amount of focus blur and disparity information. According to the result of simulation, the distance and relative speed were estimated reasonably. In addition, we compose a prototype system for the real-time estimation of distance and relative speed. The system consisted of CMOS sensors designed for this processing, an FPGA, a PC, and other devices. As a result of the implementation of processing, our system was properly validated.

The super-resolution technique based on the integration of successive frames depends on the accuracy of motion estimation. However, the technique requires a considerable amount of calculation and has a tendency to introduce estimation errors for various video sequences. In this paper, we propose a super-resolution method that involves motion estimation on the basis of block matching using high-speed sub-frame images. Because of the high correlation between sub-frame images, it is possible to reduce the computation required and improve the accuracy of motion estimation. The quality of the generated high-resolution image is further improved by the use of estimated motion vectors. We simulated the proposed method by using a high frame rate video sequence and confirmed its effectiveness.

Recently, computer-generated holograms (CGHs) have received considerable attention from researchers. However, computer-generated (CG) objects are usually represented by using multiple mono-color points, and high-resolution color holograms of real scenes are rarely provided. Although we proposed a method for generating holograms using a set of 2-D texture and depth maps of a real scene, the image obtained by optical reconstruction has been displayed in a single color. In this paper, we report that we obtained an accurate reconstruction of an image by using our proposed method against the color hologram anew. We confirmed that our proposed the color hologram properly recorded both color and the 3-D information of the space by an assessment experiment which optically reconstructs the hologram.

We have been investigating image-based rendering (IBR) imaging system by using array of smart image sensors. The imaging system consists of random accessible image sensors and FPGA, arbitrary view images can be obtained in real-time. Although IBR system can generate more realistic images compared to model-based rendering (MBR) system, the generated images are degraded if the depth information for synthesis has some errors. In this paper, we describe a method of depth estimation for the application of the imaging system. It is based on comparing various synthesized images to an actual image obtained by a real sensor repeatedly. By using the depth information, more realistic arbitrary view images can be generated.

Recently, not only still pictures but also moving pictures are displayed on the PDAs or cellular phones. When we take a look such pictures, the picture quality seems to be better than those on the CRTs or the large LCDs. In order to clarify the reason, we have investigated the relationship between the subjective quality of different size of pictures with keeping the viewing distance 6H. After three kinds of pictures (Girl, Mandrill, Milkdrop) are subjectively tested, the MOS using the small sized pictures with several distortions is better than that of the middle and large sized pictures by the psychophysical factors. In this paper, after discussing the effect of picture size on the MOS, we show the experimental results obtained by subjective evaluation with 20 observers. And, the relationship between the MOS and the Weighted SNR compensated by the subjective spatial frequency are given.

We have been investigating the integration of sensing and compression on an image sensor. The compression sensor reduces the number of pixels in the image signal that have to be readout from the sensor. We present a real-time object tracking system using a compression sensor, which has 128/spl times/128 pixels, and FPGA. By using this system, several moving objects can be extracted and tracked independently at 1200 frames/second. We also describe real-time depth estimation by using binocular compression sensors. We show some results obtained by the system.

From a biometric view point, it is preferable to recognize the writer from not only signature but also hand written characters or pictures. In on-line signature verification, we have shown that the combination of pen position, pen pressure and pen inclination information realized a high verification rate (see Hangai, S. et al., IEEE ICME2000, 2000; Yamanaka, S. et al., IEEE ICME2001, 2001). So, we apply the method to the writer verification of arbitrary written characters and pictures. In the verification, the information of direction and inclination of each segmented stroke are mainly used as feature parameters. From experimental results using hand written numeric characters, figures, and pictures by 20 persons, we can realize a verification rate of 92.59%.

A digital cochlea, which has a cascade of 16 filter sections, is realized by 32 commercially available DSP-boards. Each section consists of a travelling waves filter, a velocity transformation filter and a second filter. The artificial cochlea is also applied to spoken word recognition by feeding 16 output signals through a multi-channel A/D converter on PC. From experimental results, it is found that 50 Japanese words uttered by three speakers are recognized with 3% error This means the cochlea extracts feature parameters for speech recognition and shows the possibility of the signal processor for cochlear implants.

A new objective evaluation measure, advanced weighted signal to noise ratio (AWSNR), which reflects human visual characteristics, is proposed. In the measure, subjective observation areas are predicted based on local complexity, local brightness and the human spatial frequency characteristics. From experimental results using 350 pictures, it is found that the correlation coefficient between the MOS (mean opinion score) by 30 testers and the proposed AWSNR becomes 0.91.

We have been investigating a high-speed image-based rendering system. In this area, most of the conventional systems sacrifice spatial or temporal resolution for a heavy amount of input images. When the system uses a camera-array on its input, this problem is more obvious. However, the required image data for the rendering are only a portion of them, determined by the position of the imaginary-view. We propose an image based rendering system, which uses pixel independent random access image sensors to eliminate the bottleneck of the conventional systems. We have developed a prototype of a CMOS image sensor, which has 128 /spl times/ 128 pixels. We verified that the prototype chip readouts externally selected pixels at 60 frames/second.

In signature verification using on-line hand written data, we have cleared that the pen inclination data improves the verification rate[1]. However, the influences of intersession variability on the verification rate have not been considered yet. In order to avoid making error caused by a mismatch between his/her signature and his/her references, we applied some renewal methods to make the reference. From experimental results using signatures over 9 weeks, a reference renewal using authenticated data is effective to maintain high verification rate with 98.5%.

We describe a novel approach to enhance the performance of image sensing by integrating the processing element with an image sensor. We have been investigating a smart sensor which controls the integration time of every pixel independently. The integration time is controlled so that it has higher temporal resolution and wider dynamic range. We describe a new adaptive-integration-time image sensor which has 128/spl times/64 pixels. We have designed the VLSI prototype by using a column parallel architecture. In this new chip, the scheme to control the integration time is extended, and the pixel pitch processing speed and power consumption are much improved in comparison with our previous prototype. A 7-bit address encoder is newly implemented. We show some experimental results obtained with the prototype.

We have been investigating a computational sensor which controls the integration time of every pixel independently. Because the integration time is controlled, higher temporal resolution and wider dynamic range can be achieved. We present a new adaptive integration time image sensor which has 128/spl times/64 pixels. We adopt a column parallel architecture to design the prototype chip. The scheme to control integration time is extended, and pixel pitch, processing speed and power consumption are much improved in comparison with our previous prototype. We show some experimental results obtained with the prototype.

In order to improve the drawing speed without degrading the image quality in computer graphics, reduction of number of polygons that are difficult to detect subjectively is preferable. Especially, the number of vertices of the actual facial image becomes huge and the reduction of vertices without subjective degradation is strongly needed. However, the method of reducing the number of vertices uses evaluation functions such as surface area difference, volume difference, and so on. We have developed an interactive presentation system for evaluating the 3D-polygon model. In the experiment, 6 reduced polygon models (the number of polygons is reduced to 80%, 60%, 40%, 20%, 10% and 5%) using 6 evaluation functions are evaluated on the system.

Signature is widely used to authorize who issued the document. However, signature has ambiguity, and it is difficult to distinguish the authentic signature from the mimicked signature by using bit mapped patterns only. On the other hand, altitude and direction of the gripped pen under signing depends on the shape of writer's hand and the habit of writing. In this paper, we propose a new online writer verification method which uses the pen movement in signing. From the experimental results with 24 writers, a verification rate of 100% was obtained.

We have been investigating an integration of sensing and compression on an image sensor. The compression sensor reduces the number of pixels in the image signal that has to be readout from the sensor. Therefore, the compression sensor can capture the images at the higher pixel rate which the traditional sensor can not handle. In this paper, we present a compression sensor which has 228/spl times/128 pixels. The processing circuits of the sensor can be operated at 5000 frames/second. We also describe the experimental results of two real-time image processing systems by using the compression sensor. They are a reconstruction circuit by using FPGA and a stereo image processing system for tracking of a moving object.

In order to avoid tiresome and tedious evaluation of coded pictures, collaborative subjective evaluation method using Internet is newly proposed, Except for observation conditions, evaluating environment including evaluation time, picture presentation sequence and evaluation word is matched with that of ITU-R recommendation. Experiments open to the students in our university was continued for 4 months on the Internet to get 100 voluntary testers. According to experimental result using 14 coded pictures, the order of pictures sorted by the collaborative MOS agreed with that by the MOS of the ITU-R recommendation when 60 voluntary testers collaborated. It was also found that we could predict the MOS of the ITU-R recommendation from the collaborative MOS with 80 voluntary testers or more.

We propose a motion adaptive sensor for image enhancement and wide dynamic range sensing. The motion adaptive sensor is able to control integration time pixel by pixel. The integration time is determined by saturation and temporal changes of incident light. It is expected to have high temporal resolution in the moving area, high SNR in the static area, and wide dynamic range. We have fabricated a prototype and show some results obtained by our experiments.

We have been investigating a novel integration of sensing and compression on an image sensor. By integration, the number of pixels in the image signal that has to be readout from the sensor can be significantly reduced, and the integration, can consequently increase the pixel rate of the sensor. We present a new compression sensor which has 128/spl times/128 pixels. We have made the prototype based on a column parallel architecture and improved the processing circuits of the new prototype to achieve lower power dissipation and higher processing speed in comparison with our previous prototypes. It is verified that the processing circuits can be operated at 5000 frames/second.

We propose a new sampling control system integrated on an image sensor. Contrary to the conventional random access pixels, the proposed sensor is able to read out spatially variant pixels at high speed, without inputting pixel address for each access. The sampling positions can be changed dynamically by rewriting the sampling position memory. Since the proposed sensor has an array memory that keeps the pixel position to be sampled. The sampling position can be dynamically changed by rewriting the memory array. It can achieve any spatially varying sampling patterns. We have made a first prototype and show results obtained by the prototype.

In order to enhance performance of an image sensor, we have been investigating a novel integration of compression and sensing. By this integration, the image signal that has to be readout from the sensor is significantly reduced. Thus, the integration can consequently leads to high pixel rate sensing. The compression scheme we make use of is conditional replenishment that detects and encodes moving areas. We have developed prototypes based on two different architectures, that are pixel parallel and column parallel architectures. We present the two prototypes and their comparisons, and show the results obtained by them.

In this paper, we propose a novel image sensor which compresses image signals on the sensor plane. Since an image signal is compressed on the sensor plane by making use of the parallel nature of image signals, the amount of signal read out from the sensor can be significantly reduced. Thus, the potential applications of the proposed sensor are high pixel rate cameras and processing systems which require very high speed imaging or very high resolution imaging. The very high bandwidth is the fundamental limitation to the feasibility of those high pixel rate sensors and processing systems. Conditional replenishment is employed for the compression algorithm. In each pixel, current pixel value is compared to that in the last replenished frame. The value and the address of the pixel are extracted and coded if the magnitude of the difference is greater than a threshold. Analog circuits have been designed for processing in each pixel. A first prototype of a VLSI chip has been fabricated. Some results of experiments obtained by using the first prototype are shown in this paper.

We propose a novel concept of an integration of compression and sensing in order to enhance performance of an image sensor. By integrating the compression function on the sensor plane, the image signal that has to be read out from the sensor is significantly reduced. Thus, the integration can consequently increase the pixel rate of the sensor. The compression scheme we make use of is conditional replenishment that detects and encodes moving areas. In this paper, we discuss design and implementation of two architectures for on-sensor compression. One is a pixel parallel approach and the other is a column parallel approach. We describe both approaches and design and a prototype of pixel parallel architecture.

A novel image sensor on which video signals can be compressed is proposed. Since the video signal is compressed on an imager plane by using fast analog processing, the amount of image data read out from the imager can be significantly reduced with very small latency. The proposed system can be potentially applied to high pixel rate cameras such as those for high speed imaging and high resolution imaging. Conditional replenishment is employed for the video compression algorithm. Analog circuits are designed both for processing in each pixel and for controlling the entire data rate. The behavior of the circuit is investigated on the basis of both an analog circuit simulator and a scale-up-circuit. A VLSI chip is designed and is under fabrication.>