This study introduces a novel method to mitigate temporal illumination variations in whisk-broom hyperspectral imaging under varying environmental illumination conditions. Whisk-broom hyperspectral imaging captures high resolution spectra pixel-by-pixel sequentially, a process susceptible to sunlight fluctuations over time, particularly when imaging cultural artifacts outdoors. Despite sunlight's broad spectrum, its variability over time can compromise the quality of hyperspectral images, affecting data analysis. Prior strategies suggested a supplementary single compensating vertical scan alongside the standard row-wise raster scan. However, this strategy fails when the additional single vertical scan is performed near or on a black frame. Building on this, our study proposes using two scans, one performed orthogonally to the traditional row-wise scan, to counteract illumination variations. Furthermore, we formulate the compensation problem in logarithmic space, exploiting the low-dimensional structure of the reflectance and illumination spectra. Using total variation penalization in the cost function enhances smoothness. Our method achieves robust compensation for changes in environmental illumination. We also demonstrate that we can use multiple columns from the column-wise scan without significantly decreasing the compensation quality while reducing acquisition time. We illustrate the application of our methods to hyperspectral images of stained-glass windows of the historic Cathedrale Notre-Dame d'Amiens in France.

We propose a novel method for estimating head orientation from a low-resolution far-infrared (LFIR) image sequence captured by a 32 × 32 FIR sensor array. Estimating head orientation from LFIR images is challenging due to their inherent noise and limited resolution. Manually an-notating head orientation is also difficult, and no suitable dataset currently exists. We propose the LFIR2Head model, which leverages a multimodal-temporal framework for robust estimation. We also introduce the LFIR2Head dataset, which was created using an automatic annotation system. Extensive experiments demonstrate the accuracy of head orientation estimation with the proposed dataset. We con-firmed that head orientation can be estimated accurately from an LFIR image sequence.

The automatic colorization of anime line drawings is a challenging problem in production pipelines. Recent advances in deep neural networks have addressed this problem; however, collectingmany images of colorization targets in novel anime work before the colorization process starts leads to chicken-and-egg problems and has become an obstacle to using them in production pipelines. To overcome this obstacle, we propose a new patch-based learning method for few-shot anime-style colorization. The learning method adopts an efficient patch sampling technique with position embedding according to the characteristics of anime line drawings. We also present a continuous learning strategy that continuously updates our colorization model using new samples colorized by human artists. The advantage of our method is that it can learn our colorization model from scratch or pre-trained weights using only a few pre- and post-colorized line drawings that are created by artists in their usual colorization work. Therefore, our method can be easily incorporated within existing production pipelines. We quantitatively demonstrate that our colorizationmethod outperforms state-of-the-art methods.

Abstract We propose deep depth from focal stack (DDFS), which takes a focal stack as input of a neural network for estimating scene depth. Defocus blur is a useful cue for depth estimation. However, the size of the blur depends on not only scene depth but also camera settings such as focus distance, focal length, and f-number. Current learning-based methods without any defocus models cannot estimate a correct depth map if camera settings are different at training and test times. Our method takes a plane sweep volume as input for the constraint between scene depth, defocus images, and camera settings, and this intermediate representation enables depth estimation with different camera settings at training and test times. This camera-setting invariance can enhance the applicability of DDFS. The experimental results also indicate that our method is robust against a synthetic-to-real domain gap.

Non-line-of-sight (NLOS) imaging is conducted to infer invisible scenes from indirect light on visible objects. The neural transient field (NeTF) was proposed for representing scenes as neural radiance fields in NLOS scenes. We propose NLOS neural implicit surface (NLOS-NeuS), which extends the NeTF to neural implicit surfaces with a signed distance function (SDF) for reconstructing three-dimensional surfaces in NLOS scenes. We introduce two constraints as loss functions for correctly learning an SDF to avoid non-zero level-set surfaces. We also introduce a lower bound constraint of an SDF based on the geometry of the first-returning photons. The experimental results indicate that these constraints are essential for learning a correct SDF in NLOS scenes. Compared with previous methods with discretized representation, NLOS-NeuS with the neural continuous representation enables us to reconstruct smooth surfaces while preserving fine details in NLOS scenes. To the best of our knowledge, this is the first study on neural implicit surfaces with volume rendering in NLOS scenes. Project page: https://yfujimura.github.io/nlos-neus/

In this paper, we propose a new cue of depth sensing using thermal radiation. Our method realizes passive, texture independent, far range, and dark scene applicability, which can broaden the depth sensing subjects. A key ob-servation is that thermal radiation is attenuated by the air and is wavelength dependent. By modeling the wavelength-dependent attenuation by the air and building a multi-spectral LWIR measurement system, we can jointly estimate the depth, temperature, and emissivity of the target. We analytically show the capability of the thermal radiation cue and show the effectiveness of the method in real-world scenes using an imaging system with a few bandpass filters.

The novelty of this paper is the alignment method of narrow field-of-view hyperspectral images to full-view RGB images. The interest is to locate hyperspectral measurements in an environment described by an equirectangular image. But the very different modalities (3 vs. hundreds of channels) and fields-of-view are challenges for accurate alignment. We solve these problems within a dense direct alignment framework that optimizes the warping parameters together with those of a global illumination difference model. Our alignment code is shared with an example dataset available at github.com/jrl-umi3218/hsrgbalign.

A parasitic egg could cause infections and become significant diseases, especially in countries with poor sanitation and hygiene. Its detection which is manually conducted by humans using a microscope could be time-consuming and potent to misclassify. The goal of this research is to get a method that has a good accuracy to automatically detect and identify the parasitic egg. Also, we have a challenge here because the dataset has different resolutions, lighting, and setting conditions. Here we try to utilize the YOLO algorithm by doing image augmentation and restoration to the dataset, we use deep latent space translation for restoration and a simple GrabCut algorithm to generate an augmented image. Furthermore, we apply hyperparameter tuning in image resolution and see how we could utilize the YOLO algorithm to do fast and accurate classification. From the result, we get the combination could give better results in classifying the parasitic egg which the mIoU results reached 71.7 percent.

Abstract We propose a method for eliminating the temporal illumination variations in whisk-broom (point-scan) hyperspectral imaging. Whisk-broom scanning is useful for acquiring a spatial measurement using a pixel-based hyperspectral sensor. However, when it is applied to outdoor cultural heritages, temporal illumination variations become an issue due to the lengthy measurement time. As a result, the incoming illumination spectra vary across the measured image locations because different locations are measured at different times. To overcome this problem, in addition to the standard raster scan, we propose an additional perpendicular scan that traverses the raster scan. We show that this additional scan allows us to infer the illumination variations over the raster scan. Furthermore, the sparse structure in the illumination spectrum is exploited to robustly eliminate these variations. We quantitatively show that a hyperspectral image captured under sunlight is indeed affected by temporal illumination variations, that a Naïve mitigation method suffers from severe artifacts, and that the proposed method can robustly eliminate the illumination variations. Finally, we demonstrate the usefulness of the proposed method by capturing historic stained-glass windows of a French cathedral.

Multispectral photometric stereo (MPS) aims at recovering the surface normal of a scene from a single-shot multi-spectral image, which is known as an ill-posed problem. To make the problem well-posed, existing MPS methods rely on restrictive assumptions, such as shape prior, surfaces having a monochromatic with uniform albedo. This paper alleviates the restrictive assumptions in existing methods. We show that the problem becomes well-posed for a surface with a uniform chromaticity but spatially-varying albedos based on our new formulation. Specifically, if at least three (or two) scene points share the same chromaticity, the proposed method uniquely recovers their surface normals and spectral reflectance with the illumination of more than or equal to four (or five) spectral lights. Besides, our method can be made robust by having many (i.e., 4 or more) spectral bands using robust estimation techniques for conventional photometric stereo. Experiments on both synthetic and real-world scenes demonstrate the effectiveness of our method. Our data and result can be found at https://github.com/GH-HOME/MultispectralPS.git.

Here, we propose a novel method to estimate the parameters of non-planar objects with thin film surfaces. Being able to estimate the optical parameters of objects with thin film surfaces has a wide range of applications from industrial inspections to biological and archaeology research. However, there are many challenging issues that need to be overcome to model such parameters. The appearance of thin film objects is highly dependent on the surface orientation and optical parameters such as the refractive index and film thickness. First, we therefore analyzed the optical parameters of non-planar objects with thin film surfaces. Next, we proposed and implemented an analysis procedure and demonstrated its effectiveness for studying planar objects with thin film surfaces. Finally, we developed a device to acquire the shapes and optical parameters of objects with thin film surfaces using a camera and demonstrated the effectiveness of our method experimentally. Then, we surveyed the errors caused by the light source. We discussed the difference between the theoretically obtained parameters and experimental data obtained using a hyper spectral camera.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, the transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated, and therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

The recent development of 3D printing technology has brought concerns about its potential misuse, such as in copyright infringement and crimes. Although there have been many studies on blind 3D mesh watermarking for the copyright protection of digital objects, methods applicable to 3D printed objects are rare. In this paper, we propose a novel blind watermarking algorithm for 3D printed objects with applications for copyright protection, traitor tracing, object identification, and crime investigation. Our method allows us to embed a few bits of data into a 3D-printed object and retrieve it by 3D scanning without requiring any information about the original mesh. The payload is embedded on the object's surface by slightly modifying the distribution of surface norms, that is, the distance between the surface and the center of gravity. It is robust to resampling and can work with any 3D printer and scanner technology. In addition, our method increases the capacity and resistance by subdividing the mesh into a set of bins and spreading the data over the entire surface to negate the effect of local printing artifacts. The method's novelties include extending the vertex norm histogram to a continuous surface and the use of 3D moments to synchronize a watermark signal in a 3D-printing context. In the experiments, our method was evaluated using a public dataset against center, orientation, minimum and maximum norm misalignments; a printing simulation; and actual print/scan experiments using a standard 3D printer and scanner.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

This paper presents a method of passive non-line-of-sight (NLOS) imaging using polarization cues. A key observation is that the oblique light has a different polarimetric signal. It turns out this effect is due to the polarization axis rotation, a phenomena which can be used to better condition the light transport matrix for non-line-of-sight imaging. Our analysis and results show that the use of a polarization for NLOS is both a standalone technique, as well as an enhancement technique to boost the results of other forms of passive NLOS imaging. We make a surprising finding that, despite 50% light attenuation from polarization optics, the gains from polarized NLOS are overall superior to unpolarized NLOS.

We propose a new blind watermarking algorithm for 3D printed objects that has applications in metadata embedding, robotic grasping, counterfeit prevention, and crime investigation. Our method can be used on fused deposition modeling (FDM) 3D printers and works by modifying the printed layer thickness on small patches of the surface of an object. These patches can be applied to multiple regions of the object, thereby making it resistant to various attacks such as cropping, local deformation, local surface degradation, or printing errors. The novelties of our method are the use of the thickness of printed layers as a one-dimensional carrier signal to embed data, the minimization of distortion by only modifying the layers locally, and one-shot detection using a common paper scanner. To correct encoding or decoding errors, our method combines multiple patches and uses a 2D parity check to estimate the error probability of each bit to obtain a higher correction rate than a naive majority vote. The parity bits included in the patches have a double purpose because, in addition to error detection, they are also used to identify the orientation of the patches. In our experiments, we successfully embedded a watermark into flat surfaces of 3D objects with various filament colors using a standard FDM 3D printer, extracted it using a common 2D paper scanner and evaluated the sensitivity to surface degradation and signal amplitude.

Phase ambiguity is a major problem in the depth measurement in either time-of-flight or phase shifting. Resolving the ambiguity using a low frequency pattern sacrifices the depth precision, and using multiple frequencies requires a number of observations. In this paper, we propose a phase disambiguation method that combines temporal and spatial modulation so that the high depth precision is preserved while the number of observation is small. A key observation is that the phase ambiguities of temporal and spatial domains appear differently with respect to the depth. Using this difference, the phase can disambiguate for a wider range of interest. We develop a prototype to show the effectiveness of our method through real-world experiments.

Active illumination systems which perform disparity gating, or the ability to selectively image photons that arrive from a specified surface geometry some distance away, have recently shown usefulness for robotics, autonomous vehicles, and surveillance applications. In this paper, we present a new technique for sloped disparity gating, capturing a particular set of sloped planar surfaces in a scene, implemented using the synchronization between a raster-scanning projector and the rolling shutter of a camera. We demonstrate how to control the slope and thickness of these planar surfaces using hardware parameters of pixel clock, synchronization delay, and exposure. Finally, we perform applications including real-time image masking and imaging in scattering media with a real hardware prototype in the lab. This work showcases the potential for energy-efficient, geometry-aware disparity gating in the future.

In this paper, we present a practical method for reconstructing the bidirectional reflectance distribution function (BRDF) from multiple images of a real object composed of a homogeneous material. The key idea is that the BRDF can be sampled after geometry estimation using multi-view stereo (MVS) techniques. Our contribution is selection of reliable samples of lighting, surface normal, and viewing directions for robustness against estimation errors of MVS. Our method is quantitatively evaluated using synthesized images and its effectiveness is shown via real-world experiments.

Contact imaging is a compact lensless microscopy technique that allows observation of living cells placed directly on a charge-coupled device sensor. However, captured images exhibit low visibility in general. This work is aimed at integrating computational refocusing and high-frequency illumination into contact imaging to enhance the visibility of the captured images. Refocusing contributes to the synthesis of shallow depth-of-field images from the light field captured by moving a point light source. The visibility may be enhanced via high-frequency illumination. Naïve integration of both techniques requires several observations. Therefore, we propose an efficient integration from reduced observations by reformulating the computational refocusing applied in contact imaging. Furthermore, we developed a prototype system that demonstrates the effectiveness of the proposed method.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. The proposed method is built upon a key observation that the depth measurement by a ToF camera is distorted for objects with certain materials, especially with translucent materials. We show that this distortion is due to the variation of time domain impulse responses across materials and also due to the measurement mechanism of the ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the object material, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as a feature for classification and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluations and real-world experiments, showing its capability of material classification, even for visually indistinguishable objects.

In this research, we propose a novel registration method for three-dimensional (3D) reconstruction from serial section images. 3D reconstructed data from serial section images provides structural information with high resolution. However, there are three problems in 3D reconstruction: non-rigid deformation, tissue discontinuity, and accumulation of scale change. To solve the non-rigid deformation, we propose a novel non-rigid registration method using blending rigid transforms. To avoid the tissue discontinuity, we propose a target image selection method using the criterion based on the blending of transforms. To solve the scale change of tissue, we propose a scale adjustment method using the tissue area before and after registration. The experimental results demonstrate that our method can represent non-rigid deformation with a small number of control points, and is robust to a variation in staining. The results also demonstrate that our target selection method avoids tissue discontinuity and our scale adjustment reduces scale change.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated and, therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

Separation of light transport into direct and indirect paths has enabled new visualizations of light in everyday scenes. However, indirect light itself contains a variety of components from subsurface scattering to diffuse and specular interreflections, all of which contribute to complex visual appearance. In this paper, we present a new imaging technique that captures and analyzes these components of indirect light via light transport between epipolar planes of illumination and rays of received light. This plane-to-ray light transport is captured using a rectified projector-camera system where we vary the offset between projector and camera rows (implemented as synchronization delay) as well as the exposure of each camera row. The resulting delay-exposure stack of images can capture live short and long-range indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties.

Light interacts with a scene in various ways. For scene understanding, a light transport is useful because it describes a relationship between the incident light ray and the result of the interaction. Our goal is to acquire the 4D light transport between the projector and the camera, focusing on direct and short-range transport that include the effect of the diffuse reflections, subsurface scattering, and inter-reflections. The acquisition of the light transport is challenging since the acquisition of the full 4D light transport requires a large number of measurement. We propose an efficient method to acquire short range light transport, which is dominant in the general scene, using synchronized projector-camera system. We show the transport profile of various materials, including uniform or heterogeneous subsurface scattering.

ChromaDepth glasses are microprism glasses that produce a pop-up effect by combining refraction and diffraction. In this paper, we formulate the relation between the pop-up distance and viewing distance using a mathematical model. Using our model and optical measurements, we evaluate the human perception of the pop-up effect for ChromaDepth glasses using a large distant display.

Acquiring accurate scattering properties is important for rendering translucent materials. In particular, the phase function, which determines the distribution of scattering directions, plays a significant role in the appearance of a material. We propose a distinctive scattering theory that approximates the effect of single scattering to acquire the non-parametric phase function from a single image. Furthermore, in various experiments, we measured the phase functions from several real diluted media and rendered images of these materials to evaluate the effectiveness of our theory.

Hyperspectral imaging is a useful technique for various computer vision tasks such as material recognition. However, such technique usually requires an expensive and professional setup and is time-consuming because a conventional hyperspectral image consists of a large number of observations. In this paper, we propose a novel technique of one-shot hyperspectral imaging using faced reflectors on which color filters are attached. The key idea is based on the principle that each of multiple reflections on the filters has a different spectrum, which allows us to observe multiple intensities through different spectra. Our technique can be implemented either by a coupled mirror or a kaleidoscope geometry. Experimental results show that our technique is capable of accurately capturing a hyperspectral image by using a coupled mirror setup which is readily available.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. We use a key observation that the depth measurement by a ToF camera is distorted in objects with certain materials, especially with translucent materials. We show that this distortion is caused by the variations of time domain impulse responses across materials and also by the measurement mechanism of the existing ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the material of the object, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as features and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluation and real-world experiments, showing its capability of even classifying visually similar objects.

This paper describes a method for recovering appearance of inner slices of translucent objects. The appearance of a layered translucent object is the summed appearance of all layers, where each layer is blurred by a depth-dependent point spread function (PSF). By exploiting the difference of low-pass characteristics of depth-dependent PSFs, we develop a multi-frequency illumination method for obtaining the appearance of individual inner slices. Specifically, by observing the target object with varying the spatial frequency of checker-pattern illumination, our method recovers the appearance of inner slices via computation. We study the effect of non-uniform transmission due to inhomogeneity of translucent objects and develop a method for recovering clear inner slices based on the pixel-wise PSF estimates under the assumption of spatial smoothness of inner slice appearances. We quantitatively evaluate the accuracy of the proposed method by simulations and qualitatively show faithful recovery using real-world scenes.

Reconstruction of shapes and appearances of thin film objects can be applied to many fields such as industrial inspection, biological analysis, and archaeologic research. However, it comes with many challenging issues because the appearances of thin film can change dramatically depending on view and light directions. The appearance is deeply dependent on not only the shapes but also the optical parameters of thin film. In this paper, we propose a novel method to estimate shapes and film thickness. First, we narrow down candidates of zenith angle by degree of polarization and determine it by the intensity of thin film which increases monotonically along the zenith angle. Second, we determine azimuth angle from occluding boundaries. Finally, we estimate the film thickness by comparing a look-up table of color along the thickness and zenith angle with captured images. We experimentally evaluated the accuracy of estimated shapes and appearances and found that our proposed method is effective.

This paper presents a method for recovering shape and normal of a transparent object from a single viewpoint using a Time-of-Flight (ToF) camera. Our method is built upon the fact that the speed of light varies with the refractive index of the medium and therefore the depth measurement of a transparent object with a ToF camera may be distorted. We show that, from this ToF distortion, the refractive light path can be uniquely determined by estimating a single parameter. We estimate this parameter by introducing a surface normal consistency between the one determined by a light path candidate and the other computed from the corresponding shape. The proposed method is evaluated by both simulation and real-world experiments and shows faithful transparent shape recovery.

In this paper, we describe a supervised four-dimensional (4D) light field segmentation method that uses a graph-cut algorithm. Since 4D light field data has implicit depth information and contains redundancy, it differs from simple 4D hyper-volume. In order to preserve redundancy, we define two neighboring ray types (spatial and angular) in light field data. To obtain higher segmentation accuracy, we also design a learning-based likelihood, called objectness, which utilizes appearance and disparity cues. We show the effectiveness of our method via numerical evaluation and some light field editing applications using both synthetic and real-world light fields.

Raindrops adhered to a windscreen or window glass can significantly degrade the visibility of a scene. Modeling, detecting and removing raindrops will, therefore, benefit many computer vision applications, particularly outdoor surveillance systems and intelligent vehicle systems. In this paper, a method that automatically detects and removes adherent raindrops is introduced. The core idea is to exploit the local spatio-temporal derivatives of raindrops. To accomplish the idea, we first model adherent raindrops using law of physics, and detect raindrops based on these models in combination with motion and intensity temporal derivatives of the input video. Having detected the raindrops, we remove them and restore the images based on an analysis that some areas of raindrops completely occludes the scene, and some other areas occlude only partially. For partially occluding areas, we restore them by retrieving as much as possible information of the scene, namely, by solving a blending function on the detected partially occluding areas using the temporal intensity derivative. For completely occluding areas, we recover them by using a video completion technique. Experimental results using various real videos show the effectiveness of our method.

This paper describes a method for recovering appearance of inner slices of translucent objects. The outer appearance of translucent objects is a summation of the appearance of slices at all depths, where each slice is blurred by depth-dependent point spread functions (PSFs). By exploiting the difference of low-pass characteristics of depth-dependent PSFs, we develop a multi-frequency illumination method for obtaining the appearance of individual inner slices using a coaxial projector-camera setup. Specifically, by measuring the target object with varying the spatial frequency of checker patterns emitted from a projector, our method recovers inner slices via a simple linear solution method. We quantitatively evaluate accuracy of the proposed method by simulations and show qualitative recovery results using real-world scenes.

Some objects have layered structures in which a dynamic region is covered by a static layer. In this paper, we propose a new experiment for estimating the depth of the dynamic region using speckle analysis. The speckle is caused by the mutual interference of a coherence laser. We use two characteristics of the speckle. One is the temporal stability of the speckle pattern and the other is the wavelength dependency of the transmittance of the laser. We estimate the depth by computing correlations of speckle patterns using multispectral lasers. Experimental results using a simulated skin show that multispectral speckle correlation can be used for analyzing a layered structure.

In this paper we propose a new light transport refocusing method for depth estimation as well as for investigation inside scattering media with unknown scattering properties. Propagated visible light rays through scattering media are utilized in our proposed refocusing method. We use 2D light source to illuminate the scattering media and 2D image sensor for capturing transported rays. The proposed method that uses 4D light transport can clearly visualize shallow depth, as well as deep depth plane of the medium. We apply our light transport refocusing method for depth estimation using conventional depth-from-focus method and for clear visualization by descattering the light rays passing through the medium. To evaluate the effectiveness we have done experiments using acrylic and milk-water type scattering medium in various optical and geometrical conditions. Finally, we show up the results of depth estimation and clear visualization, as well as with numeric evaluation.

The inner structures of an object can be measured by capturing transmissive images. However, the recorded images of a translucent object tend to be unclear due to strong scattering of light inside the object. In this paper, we propose a descattering approach based on Parallel High-frequency Illumination. We show in this paper that the original high-frequency illumination method and the various extended techniques can be uniformly defined as a separation of overlapped and non-overlapped light rays. Also, we show that transmissive light rays do not overlap each other by constructing a parallel projection/measurement system for performing both illumination and observation. We have developed a measurement system that consists of a camera and projector with telecentric lenses and have evaluated descattering effects by extracting transmissive light rays.

In this paper, we propose a method of inertial sensor-based gait authentication by inter-period phase registration of an owner's gallery. In spite of the importance for gait authentication of constructing a gallery of phase-registered gait patterns, previous implementations just relied on simple methods of period detection based on heuristic knowledge such as local peaks/valleys or local auto-correlation of the gait signals. Consequently, we propose to improve a gait gallery by incorporating a phase registration technique which globally optimizes inter-period phase consistency in an energy minimization framework. However, the previous phase registration technique suffers from a phase distortion problem due to ambiguities in the combination of a periodic signal function and a phase evolution function. We present a linear phase evolution prior to constructing an undistorted gait signal for better matching performance. Experiments using real gait signals from 32 subjects show that the proposed methods outperform the latest methods in the field.

We propose a new method to analyze light transport in homogeneous scattering media. The incident light undergoes multiple bounces in translucent objects, and produces a complex light field. Our method analyzes the light transport in two steps. First, single and multiple scattering are separated by projecting high-frequency stripe patterns. Then, multiple scattering is decomposed into each bounce component based on the light transport equation. The light field for each bounce is recursively estimated. Experimental results show that light transport in scattering media can be decomposed and visualized for each bounce.

Although RANSAC is the most widely used robust estimator in computer vision, it has certain limitations making it ineffective in some situations, such as the motion estimation problem, in which uncertainty on the image features changes according to the capturing conditions. The greatest problem is that the threshold used by RANSAC to detect inliers cannot be changed adaptively; instead it is fixed by the user. An adaptive scale algorithm must therefore be applied in such cases. In this paper, we propose a new adaptive scale robust estimator that adaptively finds the best solution with the best scale to fit the inliers, without the need for predefined information. Our new adaptive scale estimator matches the residual probability density from an estimate and the standard Gaussian probability density function to find the best inlier scale. Our algorithm is evaluated in several motion estimation experiments under varying conditions and the results are compared with several of the latest adaptive-scale robust estimators.

We propose a novel wearable imaging system that can capture omnidirectional movies from the viewpoint of the camera wearer. The imaging system solves the problems of resolution uniformity and gaze matching that conventional approaches do not address. We combine cameras with curved mirrors that control the projection of the imaging system to produce uniform resolution. Use of the mirrors also enables the viewpoint to be moved closer to the eyes of the camera wearer, thus reducing gaze mismatching. The optics, including the curved mirror, have been designed to form an objective projection. The capability of the designed optics is evaluated with respect to resolution, aberration, and gaze matching. We have developed a prototype based on the designed optics for practical use. Capability of the prototype and effectiveness of first-person perspective omnidirectional movies were demonstrated through quantitative evaluations and presentation experiments to ordinary people, respectively.

We present a new method of analyzing subsurface scattering occurring in a translucent object from a single image taken under generic illumination. In our method, diffuse subsurface reflectance in the subsurface scattering model can be linearly solved by quantizing the distances between each pair of surface points. Then, the dipole approximation is fit to the diffuse subsurface reflectance. By applying our method to real images, we confirm that the parameters of subsurface scattering can be computed for different materials.

We propose a new egomotion estimation algorithm for a compound omnidirectional camera. Image features are detected by a conventional feature detector and then quickly classified into near and far features by checking infinity on the omnidirectional image of the compound omnidirectional sensor. Egomotion estimation is performed in two steps: first, rotation is recovered using far features; then translation is estimated from near features using the estimated rotation. RANSAC is used for estimations of both rotation and translation. Experiments in various environments show that our approach is robust and provides good accuracy in real-time for large motions.

Measuring BRDF (bi-directional reflectance distribution function) requires huge amounts of time because a target object must be illuminated from all incident angles and the reflected lights must be measured from all reflected angles. In this paper, we present a high-speed method to measure BRDFs using an ellipsoidal mirror and a projector. Our method makes it possible to change incident angles without a mechanical drive. Moreover, the omni-directional reflected lights from the object can be measured by one static camera at once. Our prototype requires only fifty minutes to measure anisotropic BRDFs, even if the lighting interval is one degree.

High dynamic range images (HDRIs) are needed for capturing scenes that include drastic lighting changes. This paper presents a method to improve the dynamic range of a camera by using a reflective liquid crystal. The system consists of a camera and a reflective liquid crystal placed in front of the camera. By controlling the attenuation rate of the liquid crystal, the scene radiance for each pixel is adaptively controlled. After the control, the original scene radiance is derived from the attenuation rate of the liquid crystal and the radiance obtained by the camera. A prototype system has been developed and tested for a scene that includes drastic lighting changes. The radiance of each pixel was independently controlled and the HDRIs were obtained by calculating the original scene radiance from these results.

This paper describes a method of mirror localization to calibrate a catadioptric imaging system. Even though the calibration of a catadioptric system includes the estimation of various parameters, in this paper we focus on the localization of the mirror. Since some previously proposed methods assume a single view point system, they have strong restrictions on the position and shape of the mirror. We propose a method that uses parallel lights to simplify the geometry of projection for estimating the position of the mirror, thereby not restricting the position or shape of the mirror. Further, we omit the translation process between the camera and calibration objects from the parameters to be estimated by observing some parallel lights from a different direction. We obtain the constraints on the projection and compute the error between the model of the mirror and the measurements. The position of the mirror is estimated by minimizing the error. We also test our method by simulation and real experiments, and finally we evaluate the accuracy of our method.

Camera ego-motion consists of translation and rotation, in which rotation can be described simply by distant features. We present a robust rotation estimation using distant features given by our compound omnidirectional sensor. Features are detected by a conventional feature detector, and then distant features are identified by checking the infinity on the omnidirectional image of the compound sensor. The rotation matrix is estimated between consecutive video frames using RANSAC with only distant features. Experiments with various environments show that our approach is robust and also gives reasonable accuracy in real-time.

The HBP (horizontal fixed viewpoint biconical paraboloidal) mirror is an anisotropic convex mirror that has a property of inhomogeneous angular resolution about azimuth angle. In this paper, we investigate the effectiveness of the HBP mirror system for remote surveillance. We developed a real remote surveillance system that is constructed by the HBP mirror system mounted on an electric cart. Through the surveillance experiments, surveyors usually looked almost front views, and they paid attention to interesting objects only when the cart approaches them. Since the HBP mirror system has high resolution in frontal view, it seems to work well in the surveillance. We also constructed a simulational remote surveillance environment in order to quantitatively compare the HBP mirror system with a conventional omnidirectional mirror system under fair experimental conditions. As a practical task, we assumed object searching in a virtually constructed devastated area. We confirmed that objects can be detected earlier and with certainty by the HBP mirror system

If a pattern is projected onto a concave screen,the desired view cannot be correctly observed due to the influence of inter-reflections. This paper proposes a simple but effective technique for photometric compensation in consideration of inter-reflections. The compensation is accomplished by canceling inter-reflections estimated by the radiosity method. The significant advantage of our method is that iterative calculations are not necessary because it analytically solves the inverse problem of inter-reflections.

We present a new video browsing method for multiple videos that are taken at large-scale space for live 3D events such as soccer games. By our method, multiple viewers over a computer network can browse a live 3D event from any viewpoint and each viewer can move his/her viewpoint freely. Our algorithm consists of five steps. Our system first captures videos from multiple cameras, then extracts texture segments from the videos, selects appropriate segments according to a viewpoint which is given by user dynamically, transmits them to users, and lays out the segments in virtual space so that each viewer can see the segments in a virtual environment as if the viewers were in the event. Our 3D video display system requires 10 Mbps at most to browse a soccer game. We conducted experiments at two real soccer stadiums and succeeded in realizing live realistic visualization with free viewpoint at about 26 fps.

A Photometric Image-Based Rendering (PIER) concept is proposed that facilitates the generation of an image with an arbitrary illumination. Based on this concept, we aim to generate both diffuse and specular reflections. It is not necessary to explicitly recover 3D shape and reflection properties of the scene. In order to control appearance changes caused by modifications in the lighting conditions, we utilize a set of real images taken, in a variety of lighting conditions. Since the diffuse and specular reflection components have different characteristics, we separate these components and apply different methods to each. A photometric linearization is introduced to control diffuse reflections as well as for separating the other components. This also facilitates the treatment of attached shadows as a part of the diffuse reflection. A morphing technique is utilized to generate specular reflections. This is an effective technique for dealing with glossy objects, even when the light shape is clearly observed in the specular reflection. Experimental results show that realistic images can be successfully generated using this technique.

The individuality of a human face depends on the fine details of the facial components, and it is necessary to extract and to describe these detailed patterns in order to recognize human faces. We propose a method to describe the eye figure with small parameters by classifying their patterns to typical groups. First, an eye image is divided into parts such as eyelid and inner corner, and a set of 1-dimensional slit projections is obtained from the 2-dimensional intensity array. Then, the principal component analysis is applied to these projections to find the major axes which have typical features. The individuality of each eye is parameterized by the principal component scores. The effectiveness of the description is evaluated by generating sketch images based on the parameters extracted from real eye images.