Separating an image of a scene illuminated by a light source into direct components such as specular reflection and diffuse reflection, and global components such as interreflection and subsurface scattering is important as preprocessing for various computer vision and graphics applications. Conventional methods cannot separate direct and global components from novel viewpoints, and have difficulties in robustly separating those components from a small number of images even from known viewpoints. In this paper, we propose a method for synthesizing the direct and global components of a scene from novel viewpoints by using a relatively small number of images. Specifically, our proposed method uses the multi-view images captured by using a coaxial projector-camera system, and then recovers the density and radiance values of each component on the basis of neural radiance fields (NeRF). We conduct a number of experiments using real images captured with a projector-camera system, and confirm the effectiveness of our method. In addition, we demonstrate that our method is useful for two applications: image-based material editing and 3D shape recovery.

Defocus is a crucial cue for image-based microscale depth estimation, yet its measurement depends on spatial appearance changes, such as texture. We show that passively observed polarization is responsive to small irregularities of the surface visible in the microscopic world and can be leveraged for focus measure as a strong texture. Our key idea is to leverage texture from polarization for blur analysis and accurately estimate the focus level of microscopic polarization images. We further utilize normal cues from polarization to create a prior distribution of the focus level between neighboring pixels. We then interpolatively propagate the focus level of discrete image slices at different focus depths while denoising. We implement our method with a single polarization camera with a microscope and recover the per-pixel depth from the multi-focus images. The reconstructed results demonstrate the effectiveness of our method for various microscale objects regardless of the surface texture.

Synthesizing photo-realistic images of a scene from arbitrary viewpoints and under arbitrary lighting environments is one of the important research topics in computer vision and graphics. In this paper, we propose a method for synthesizing photo-realistic images of a scene with fluorescent objects from novel viewpoints and under novel lighting colors. In general, fluorescent materials absorb light with certain wavelengths and then emit light with longer wavelengths than the absorbed ones, in contrast to reflective materials which preserve wavelengths of light. Therefore, we cannot reproduce the colors of fluorescent objects under arbitrary lighting colors by combining conventional view synthesis techniques with the white balance adjustment of the RGB channels. Accordingly, we extend the novel view synthesis based on the neural radiance fields by incorporating the superposition principle of light; our proposed method captures a sparse set of images of a scene from varying viewpoints and under varying light source colors by using a display-camera system, and then synthesize photo-realistic images of the scene without explicitly modeling the geometric and photometric models of the scene. We conduct a number of experiments using real images, and confirm the effectiveness of our method.

This paper proposes a practical method of microscale 3D shape capturing by a teleidoscopic imaging system. The main challenge in microscale 3D shape reconstruction is to capture the target from multiple viewpoints with a large enough depth-of-field. Our idea is to employ a teleidoscopic measurement system consisting of three planar mirrors and monocentric lens. The planar mirrors virtually define multiple viewpoints by multiple reflections, and the monocentric lens realizes a high magnification with less blurry and surround view even in closeup imaging. Our contributions include, a structured ray-pixel camera model which handles refractive and reflective projection rays efficiently, analytical evaluations of depth of field of our teleidoscopic imaging system, and a practical calibration algorithm of the teleidoscopic imaging system. Evaluations with real images prove the concept of our measurement system.

Artificial light sources are often powered by an electric grid, and then their intensities rapidly oscillate in response to the grid's alternating current (AC). Interestingly, the flickers of scene radiance values due to AC illumination are useful for extracting rich information on a scene of interest. In this paper, we show that the flickers due to AC illumination is useful for intrinsic image decomposition (IID). Our proposed method conducts the light source separation (LSS) followed by the IID under AC illumination. In particular, we reveal the ambiguity in the blind LSS via matrix factorization and the ambiguity in the IID assuming the diffuse reflection model, and then show why and how those ambiguities can be resolved via a physics-based approach. We experimentally confirmed that our method can recover the colors of the light sources, the diffuse reflectance values, and the diffuse and specular intensities (shadings) under each of the light sources, and that the IID under AC illumination is effective for application to auto white balancing.

Separating diffuse and specular reflection components observed on an object surface is important for preprocessing of various computer vision techniques. Conventionally, diffuse-specular separation based on the polarimetric and color clues assumes that the diffuse/specular reflection components are unpolarized/partially polarized under unpolarized light sources. However, the diffuse reflection component is partially polarized in fact, because the diffuse reflectance is maximal when the polarization direction is parallel to the outgoing plane. Accordingly, we propose a method for separating partially-polarized diffuse and specular reflection components on the basis of the polarization reflection model and the dichromatic reflection model. In particular, our method enables us not only to achieve diffuse-specular separation but also to estimate the polarimetric properties of the object surface from a single color polarization image. We experimentally confirmed that our method performs better than the method assuming unpolarized diffuse reflection components.

In this paper, we introduce a novel method for reconstructing surface normals and depth of dynamic objects in water. Past shape recovery methods have leveraged various visual cues for estimating shape (e.g., depth) or surface normals. Methods that estimate both compute one from the other. We show that these two geometric surface properties can be simultaneously recovered for each pixel when the object is observed underwater. Our key idea is to leverage multi-wavelength near-infrared light absorption along different underwater light paths in conjunction with surface shading. Our method can handle both Lambertian and non-Lambertian surfaces. We derive a principled theory for this surface normals and shape from water method and a practical calibration method for determining its imaging parameters values. By construction, the method can be implemented as a one-shot imaging system. We prototype both an off-line and a video-rate imaging system and demonstrate the effectiveness of the method on a number of real-world static and dynamic objects. The results show that the method can recover intricate surface features that are otherwise inaccessible.

Cash flow prediction of a bank is an important task as it is not only related to liquidity risk but is also regulated by financial authorities. To improve the prediction, a graph analysis of bank transaction data is promising, while its size, scale-free nature, and various attributes make the task challenging.In this paper, we propose a graph-based machine learning method for the cash flow prediction t ask. Our contributions are as follows. (i) We introduce an extensible and scalable shared-memory parallel graph analysis platform that supports the vertex-centric, bulk synchronous parallel programming paradigm. (ii) We introduce two novel graph features upon the platform: (ii-a) an internal money flow feature based on the Markov process approximation, and (ii-b) an anomaly score feature derived from other graph features.The proposed method is examined with real bank transaction data. The proposed graph features reduce the error of a long-term (31-day) cash flow prediction by 56 % from that of a non-graph-based time-series prediction model. The graph analysis platform can compute graph features from a graph with 10 × 106nodes and 593 × 106edges in 2 hours 20 minutes.

We introduce a novel method for recovering per-pixel surface normals from a pair of polarization cameras. Unlike past methods that use polarimetric observations as auxiliary features for correspondence matching, we fully integrate them in cost volume construction and filtering to directly recover per-pixel surface normals, not as byproducts of recovered disparities. Our key idea is to introduce a polarimetric cost volume of distance defined on the polarimetric observations and the polarization state computed from the surface normal. We adapt a belief propagation algorithm to filter this cost volume. The filtering algorithm simultaneously estimates the disparities and surface normals as separate entities, while effectively denoising the original noisy polarimetric observations of a quad-Bayer polarization camera. In addition, in contrast to past methods, we model polarimetric light reflection of mesoscopic surface roughness, which is essential to account for its illumination-dependency. We demonstrate the effectiveness of our method on a number of complex, real objects. Our method offers a simple and detailed 3D sensing capability for complex, non-Lambertian surfaces.

The sky exhibits a unique spatial polarization pattern by scattering the unpolarized sun light. Just like insects use this unique angular pattern to navigate, we use it to map pixels to directions on the sky. That is, we show that the unique polarization pattern encoded in the polarimetric appearance of an object captured under the sky can be decoded to reveal the surface normal at each pixel. We derive a polarimetric reflection model of a diffuse plus mirror surface lit by the sun and a clear sky. This model is used to recover the per-pixel surface normal of an object from a single polarimetric image or from multiple polarimetric images captured under the sky at different times of the day. We experimentally evaluate the accuracy of our shape-from-sky method on a number of real objects of different surface compositions. The results clearly show that this passive approach to fine-geometry recovery that fully leverages the unique illumination made by nature is a viable option for 3D sensing. With the advent of quad-Bayer polarization chips, we believe the implications of our method span a wide range of domains.

We introduce a novel 3D sensing method for recovering a consistent, dense 3D shape of a dynamic, non-rigid object in water. The method reconstructs a complete (or fuller) 3D surface of the target object in a canonical frame (e.g., rest shape) as it freely deforms and moves between frames by estimating underwater 3D scene flow and using it to integrate per-frame depth estimates recovered from two near-infrared observations. The reconstructed shape is refined in the course of this global non-rigid shape recovery by leveraging both geometric and radiometric constraints. We implement our method with a single camera and a light source without the orthographic assumption on either by deriving a practical calibration method that estimates the point source position with respect to the camera. Our reconstruction method also accounts for scattering by water. We prototype a video-rate imaging system and show 3D shape reconstruction results on a number of real-world static, deformable, and dynamic objects and creatures in real-world water. The results demonstrate the effectiveness of the method in recovering complete shapes of complex, non-rigid objects in water, which opens new avenues of application for underwater 3D sensing in the sub-meter range.

In this paper, we introduce a novel method for reconstructing surface normals and depth of dynamic objects in water. Past shape recovery methods have leveraged various visual cues for estimating shape (e.g., depth) or surface normals. Methods that estimate both compute one from the other. We show that these two geometric surface properties can be simultaneously recovered for each pixel when the object is observed underwater. Our key idea is to leverage multi-wavelength near-infrared light absorption along different underwater light paths in conjunction with surface shading. We derive a principled theory for this surface normals and shape from water method and a practical calibration method for determining its imaging parameters values. By construction, the method can be implemented as a one-shot imaging system. We prototype both an off-line and a video-rate imaging system and demonstrate the effectiveness of the method on a number of real-world static and dynamic objects. The results show that the method can recover intricate surface features that are otherwise inaccessible.

This paper is aimed at proposing a new structured light pattern for mirror-based multi-view active stereo so that the patterns cast onto the object surface do not interfere even where the object is illuminated by the projector directly and indirectly via mirror. The key idea of our interference-free projection is to encode the projector pixel locations so that they do not collide with the code from other projector pixels by exploiting the epipolar geometry defined by the real and the virtual projectors. We prove that our new encoding does not generate code collisions between the direct and indirect patterns from the real and the virtual projectors respectively. Evaluations using real and synthesized datasets demonstrate that our approach can realize an interference-free projection without using specialized equipment such as orthographic projectors used in the state-of-the-art methods.

We propose a coarse-grained simulation method which takes the effect of memory access contention into account. The method can be used for the evaluation of the execution time of an application program during the system architecture design in an early phase of development. In this phase, information about memory access timings is usually not available. Our method uses a statistical approximation of the memory access timings to estimate their influences on the execution time. We report a preliminary verification of our simulation method by comparing it with an experimental result from an image processing application on a dual-core PC. We find an error of the order of 3 percents on the execution time.

This paper describes a system-level modeling method in UML for performance evaluation of embedded systems. The core technology of this modeling method is reverse modeling based on dynamic analysis. A case study of real MFPs (multifunction peripherals/printers) is presented in this paper to evaluate the modeling method.