This paper presents a novel event camera simulation system fully based on physically based Monte Carlo path tracing with adaptive path sampling. The adaptive sampling performed in the proposed method is based on a statistical technique, hypothesis testing for the hypothesis whether the difference of logarithmic luminances at two distant periods is significantly larger than a predefined event threshold. To this end, our rendering system collects logarithmic luminances rather than raw luminance in contrast to the conventional rendering system imitating conventional RGB cameras. Then, based on the central limit theorem, we reasonably assume that the distribution of the population mean of logarithmic luminance can be modeled as a normal distribution, allowing us to model the distribution of the difference of logarithmic luminance as a normal distribution. Then, using Student's t-test, we can test the hypothesis and determine whether to discard the null hypothesis for event non-occurrence. When we sample a sufficiently large number of path samples to satisfy the central limit theorem and obtain a clean set of events, our method achieves significant speed up compared to a simple approach of sampling paths uniformly at every pixel. To our knowledge, we are the first to simulate the behavior of event cameras in a physically accurate manner using an adaptive sampling technique in Monte Carlo path tracing, and we believe this study will contribute to the development of computer vision applications using event cameras.

Non-line-of-sight imaging (NLOS) is the process of estimating information about a scene that is hidden from the direct line of sight of the camera. NLOS imaging typically requires time-resolved detectors and a laser source for illumination, which are both expensive and computationally intensive to handle. In this paper, we propose an NLOS-based localization and posture classification technique that uses an off-the-shelf projector and camera system. We leverage a message-passing neural network to learn a visible scene geometry and predict the best position to be spotlighted by the projector that can maximize the NLOS signal. The neural network is trained end-to-end and the network parameters are optimized to maximize the NLOS performance. Unlike prior deep-learning-based NLOS techniques that assume planar relay walls, our system allows us to handle line-of-sight scenes where scene geometries are more arbitrary. Our method demonstrates state-of-the-art performance in object localization and position classification using both synthetic and real scenes.

This paper addresses the problem of reconstructing scenes behind optical diffusers, which is common in applications such as imaging through frosted glass. We propose a new approach that exploits specular reflection to capture sharp light distributions with a point light source, which can be used to detect reflections in low signal-to-noise scenarios. In this paper, we propose a rasterizer-based differentiable renderer to solve this problem by minimizing the difference between the captured and rendered images. Because our method can simultaneously optimize multiple observations for different light source positions, it is confirmed that ambiguities of the scene are efficiently eliminated by increasing the number of observations. Experiments show that the proposed method can reconstruct a scene with several mirror-like objects behind the diffuser in both simulated and real environments.

The automatic colorization of anime line drawings is a challenging problem in production pipelines. Recent advances in deep neural networks have addressed this problem; however, collectingmany images of colorization targets in novel anime work before the colorization process starts leads to chicken-and-egg problems and has become an obstacle to using them in production pipelines. To overcome this obstacle, we propose a new patch-based learning method for few-shot anime-style colorization. The learning method adopts an efficient patch sampling technique with position embedding according to the characteristics of anime line drawings. We also present a continuous learning strategy that continuously updates our colorization model using new samples colorized by human artists. The advantage of our method is that it can learn our colorization model from scratch or pre-trained weights using only a few pre- and post-colorized line drawings that are created by artists in their usual colorization work. Therefore, our method can be easily incorporated within existing production pipelines. We quantitatively demonstrate that our colorizationmethod outperforms state-of-the-art methods.

This paper presents an algorithm to obtain an event-based video from noisy frames given by physics-based Monte Carlo path tracing over a synthetic 3D scene. Given the nature of dynamic vision sensor (DVS), rendering event-based video can be viewed as a process of detecting the changes from noisy brightness values. We extend a denoising method based on a weighted local regression (WLR) to detect the brightness changes rather than applying denoising to every pixel. Specifically, we derive a threshold to determine the likelihood of event occurrence and reduce the number of times to perform the regression. Our method is robust to noisy video frames obtained from a few path-traced samples. Despite its efficiency, our method performs comparably to or even better than an approach that exhaustively denoises every frame. Visit our project page for more information: https: //github.com/0V/ESIM-AD.git.

Abstract We propose a method for eliminating the temporal illumination variations in whisk-broom (point-scan) hyperspectral imaging. Whisk-broom scanning is useful for acquiring a spatial measurement using a pixel-based hyperspectral sensor. However, when it is applied to outdoor cultural heritages, temporal illumination variations become an issue due to the lengthy measurement time. As a result, the incoming illumination spectra vary across the measured image locations because different locations are measured at different times. To overcome this problem, in addition to the standard raster scan, we propose an additional perpendicular scan that traverses the raster scan. We show that this additional scan allows us to infer the illumination variations over the raster scan. Furthermore, the sparse structure in the illumination spectrum is exploited to robustly eliminate these variations. We quantitatively show that a hyperspectral image captured under sunlight is indeed affected by temporal illumination variations, that a Naïve mitigation method suffers from severe artifacts, and that the proposed method can robustly eliminate the illumination variations. Finally, we demonstrate the usefulness of the proposed method by capturing historic stained-glass windows of a French cathedral.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, the transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated, and therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

The recent development of 3D printing technology has brought concerns about its potential misuse, such as in copyright infringement and crimes. Although there have been many studies on blind 3D mesh watermarking for the copyright protection of digital objects, methods applicable to 3D printed objects are rare. In this paper, we propose a novel blind watermarking algorithm for 3D printed objects with applications for copyright protection, traitor tracing, object identification, and crime investigation. Our method allows us to embed a few bits of data into a 3D-printed object and retrieve it by 3D scanning without requiring any information about the original mesh. The payload is embedded on the object's surface by slightly modifying the distribution of surface norms, that is, the distance between the surface and the center of gravity. It is robust to resampling and can work with any 3D printer and scanner technology. In addition, our method increases the capacity and resistance by subdividing the mesh into a set of bins and spreading the data over the entire surface to negate the effect of local printing artifacts. The method's novelties include extending the vertex norm histogram to a continuous surface and the use of 3D moments to synchronize a watermark signal in a 3D-printing context. In the experiments, our method was evaluated using a public dataset against center, orientation, minimum and maximum norm misalignments; a printing simulation; and actual print/scan experiments using a standard 3D printer and scanner.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

We propose a new blind watermarking algorithm for 3D printed objects that has applications in metadata embedding, robotic grasping, counterfeit prevention, and crime investigation. Our method can be used on fused deposition modeling (FDM) 3D printers and works by modifying the printed layer thickness on small patches of the surface of an object. These patches can be applied to multiple regions of the object, thereby making it resistant to various attacks such as cropping, local deformation, local surface degradation, or printing errors. The novelties of our method are the use of the thickness of printed layers as a one-dimensional carrier signal to embed data, the minimization of distortion by only modifying the layers locally, and one-shot detection using a common paper scanner. To correct encoding or decoding errors, our method combines multiple patches and uses a 2D parity check to estimate the error probability of each bit to obtain a higher correction rate than a naive majority vote. The parity bits included in the patches have a double purpose because, in addition to error detection, they are also used to identify the orientation of the patches. In our experiments, we successfully embedded a watermark into flat surfaces of 3D objects with various filament colors using a standard FDM 3D printer, extracted it using a common 2D paper scanner and evaluated the sensitivity to surface degradation and signal amplitude.

Active illumination systems which perform disparity gating, or the ability to selectively image photons that arrive from a specified surface geometry some distance away, have recently shown usefulness for robotics, autonomous vehicles, and surveillance applications. In this paper, we present a new technique for sloped disparity gating, capturing a particular set of sloped planar surfaces in a scene, implemented using the synchronization between a raster-scanning projector and the rolling shutter of a camera. We demonstrate how to control the slope and thickness of these planar surfaces using hardware parameters of pixel clock, synchronization delay, and exposure. Finally, we perform applications including real-time image masking and imaging in scattering media with a real hardware prototype in the lab. This work showcases the potential for energy-efficient, geometry-aware disparity gating in the future.

In this paper, we present a practical method for reconstructing the bidirectional reflectance distribution function (BRDF) from multiple images of a real object composed of a homogeneous material. The key idea is that the BRDF can be sampled after geometry estimation using multi-view stereo (MVS) techniques. Our contribution is selection of reliable samples of lighting, surface normal, and viewing directions for robustness against estimation errors of MVS. Our method is quantitatively evaluated using synthesized images and its effectiveness is shown via real-world experiments.

Contact imaging is a compact lensless microscopy technique that allows observation of living cells placed directly on a charge-coupled device sensor. However, captured images exhibit low visibility in general. This work is aimed at integrating computational refocusing and high-frequency illumination into contact imaging to enhance the visibility of the captured images. Refocusing contributes to the synthesis of shallow depth-of-field images from the light field captured by moving a point light source. The visibility may be enhanced via high-frequency illumination. Naïve integration of both techniques requires several observations. Therefore, we propose an efficient integration from reduced observations by reformulating the computational refocusing applied in contact imaging. Furthermore, we developed a prototype system that demonstrates the effectiveness of the proposed method.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. The proposed method is built upon a key observation that the depth measurement by a ToF camera is distorted for objects with certain materials, especially with translucent materials. We show that this distortion is due to the variation of time domain impulse responses across materials and also due to the measurement mechanism of the ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the object material, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as a feature for classification and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluations and real-world experiments, showing its capability of material classification, even for visually indistinguishable objects.

In this research, we propose a novel registration method for three-dimensional (3D) reconstruction from serial section images. 3D reconstructed data from serial section images provides structural information with high resolution. However, there are three problems in 3D reconstruction: non-rigid deformation, tissue discontinuity, and accumulation of scale change. To solve the non-rigid deformation, we propose a novel non-rigid registration method using blending rigid transforms. To avoid the tissue discontinuity, we propose a target image selection method using the criterion based on the blending of transforms. To solve the scale change of tissue, we propose a scale adjustment method using the tissue area before and after registration. The experimental results demonstrate that our method can represent non-rigid deformation with a small number of control points, and is robust to a variation in staining. The results also demonstrate that our target selection method avoids tissue discontinuity and our scale adjustment reduces scale change.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated and, therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

Separation of light transport into direct and indirect paths has enabled new visualizations of light in everyday scenes. However, indirect light itself contains a variety of components from subsurface scattering to diffuse and specular interreflections, all of which contribute to complex visual appearance. In this paper, we present a new imaging technique that captures and analyzes these components of indirect light via light transport between epipolar planes of illumination and rays of received light. This plane-to-ray light transport is captured using a rectified projector-camera system where we vary the offset between projector and camera rows (implemented as synchronization delay) as well as the exposure of each camera row. The resulting delay-exposure stack of images can capture live short and long-range indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties.

Light interacts with a scene in various ways. For scene understanding, a light transport is useful because it describes a relationship between the incident light ray and the result of the interaction. Our goal is to acquire the 4D light transport between the projector and the camera, focusing on direct and short-range transport that include the effect of the diffuse reflections, subsurface scattering, and inter-reflections. The acquisition of the light transport is challenging since the acquisition of the full 4D light transport requires a large number of measurement. We propose an efficient method to acquire short range light transport, which is dominant in the general scene, using synchronized projector-camera system. We show the transport profile of various materials, including uniform or heterogeneous subsurface scattering.

ChromaDepth glasses are microprism glasses that produce a pop-up effect by combining refraction and diffraction. In this paper, we formulate the relation between the pop-up distance and viewing distance using a mathematical model. Using our model and optical measurements, we evaluate the human perception of the pop-up effect for ChromaDepth glasses using a large distant display.

Acquiring accurate scattering properties is important for rendering translucent materials. In particular, the phase function, which determines the distribution of scattering directions, plays a significant role in the appearance of a material. We propose a distinctive scattering theory that approximates the effect of single scattering to acquire the non-parametric phase function from a single image. Furthermore, in various experiments, we measured the phase functions from several real diluted media and rendered images of these materials to evaluate the effectiveness of our theory.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. We use a key observation that the depth measurement by a ToF camera is distorted in objects with certain materials, especially with translucent materials. We show that this distortion is caused by the variations of time domain impulse responses across materials and also by the measurement mechanism of the existing ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the material of the object, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as features and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluation and real-world experiments, showing its capability of even classifying visually similar objects.

This paper describes a method for recovering appearance of inner slices of translucent objects. The appearance of a layered translucent object is the summed appearance of all layers, where each layer is blurred by a depth-dependent point spread function (PSF). By exploiting the difference of low-pass characteristics of depth-dependent PSFs, we develop a multi-frequency illumination method for obtaining the appearance of individual inner slices. Specifically, by observing the target object with varying the spatial frequency of checker-pattern illumination, our method recovers the appearance of inner slices via computation. We study the effect of non-uniform transmission due to inhomogeneity of translucent objects and develop a method for recovering clear inner slices based on the pixel-wise PSF estimates under the assumption of spatial smoothness of inner slice appearances. We quantitatively evaluate the accuracy of the proposed method by simulations and qualitatively show faithful recovery using real-world scenes.

This paper presents a method for recovering shape and normal of a transparent object from a single viewpoint using a Time-of-Flight (ToF) camera. Our method is built upon the fact that the speed of light varies with the refractive index of the medium and therefore the depth measurement of a transparent object with a ToF camera may be distorted. We show that, from this ToF distortion, the refractive light path can be uniquely determined by estimating a single parameter. We estimate this parameter by introducing a surface normal consistency between the one determined by a light path candidate and the other computed from the corresponding shape. The proposed method is evaluated by both simulation and real-world experiments and shows faithful transparent shape recovery.

In this paper, we describe a supervised four-dimensional (4D) light field segmentation method that uses a graph-cut algorithm. Since 4D light field data has implicit depth information and contains redundancy, it differs from simple 4D hyper-volume. In order to preserve redundancy, we define two neighboring ray types (spatial and angular) in light field data. To obtain higher segmentation accuracy, we also design a learning-based likelihood, called objectness, which utilizes appearance and disparity cues. We show the effectiveness of our method via numerical evaluation and some light field editing applications using both synthetic and real-world light fields.

This paper describes a method for recovering appearance of inner slices of translucent objects. The outer appearance of translucent objects is a summation of the appearance of slices at all depths, where each slice is blurred by depth-dependent point spread functions (PSFs). By exploiting the difference of low-pass characteristics of depth-dependent PSFs, we develop a multi-frequency illumination method for obtaining the appearance of individual inner slices using a coaxial projector-camera setup. Specifically, by measuring the target object with varying the spatial frequency of checker patterns emitted from a projector, our method recovers inner slices via a simple linear solution method. We quantitatively evaluate accuracy of the proposed method by simulations and show qualitative recovery results using real-world scenes.

We propose novel load balancers for key-value data stores running as a service over geographically distributed data centers. We assume that the key-value store has the following properties: (i) The requests (e.g., put/write, get/read, copy, etc.) can be directed to any one of the cluster nodes regardless of which cluster nodes store the information; (ii) Different objects can have different number of replicas and applications can set their desired consistency levels; and (iii) it is possible for intermediate (authorized) proxies to query/learn cluster topology and key space to node mappings. We propose two load balancing strategies that would be useful under these assumptions: (i) Static strategy that makes assignment decisions solely based on where the majority of the hosting nodes are located for a given key. (ii) Dynamic strategy where proxy nodes actively probe the performance of different cluster nodes as a function of their location (i.e., which data center they belong) and key-specific operational parameters (e.g., consistency level and replication factor) to select the best cluster node to serve each request. Our evaluations show that 2× gains for average read/write delays against random load balancers, while dynamic approach significantly outperforming the static approach between 60th and 90th percentiles.

In this paper, we discuss problems in densely deployed wireless networks. Particularly, we focus on wireless local area networks (WLANs) because they have enabled us to provide seamless and high capacity wireless access easily and inexpensively. However, recently, channel interference between different services has become a serious problem because access points (APs) of WLANs are located too densely. In the carrier sense multiple access with collision avoidance used in WLANs, the hidden terminal (HT) and the exposed terminal (ET) problems occur depending on the distance between stations and the carrier sensing range. In the higher dense deployment mentioned above, the HT and ET problems occur complicatedly. Therefore, we propose an AP cooperation system that detects the HT and ET problems between stations (STAs). In our system, APs are operated with time synchronization and obtain the information of connected STAs from received frames. The HT and ET problems are identified from the integration of the information obtained at different APs. The effectiveness is verified by simulations.

The enhancements of the transmission speed in wireless access networks and mobile-device capacity enable us to use data/audio streaming/video streaming multicast service in large scale networks via their personal mobile devices. Peer-to-peer (P2P) architecture ensures scalability and robustness more easily and more economically than server-client architecture; as the number of nodes in a P2P network increases, the amount of workload per node decreases and lessens the impact of node failure. However, mobile users feel much larger psychological cost due to strict limitations on bandwidth, processing power, memory capacity, and battery life, and they want to minimize their contributions to these services. Therefore, the issue of how we can reduce this psychological cost remains. In this paper, we consider how effective a social networking service is as a platform for mobile P2P multicast. We model users' cooperative behaviors in mobile P2P multicast streaming, and propose a social-network based P2P streaming architecture for mobile networks. We also measured the psychological forwarding cost of real users in mobile P2P multicast streaming through an emulation experiment, and verify that our social-network based mobile P2P multicast streaming improves service quality by reducing the psychological forwarding cost using multi-agent simulation.

Wireless local area networks (WLANs) have allowed us to deploy seamless and high capacity wireless systems easily and inexpensively. However, recently, channel interference between different services has become a serious problem because access points (APs) of WLANs are located too densely. In the carrier sense multiple access with collision avoidance used in WLANs, the hidden terminal (HT) problems occur depending on the distance between stations and the carrier sensing range. In the higher dense deployment mentioned above, the HT problems occur complicatedly. In this paper, we propose an AP cooperation system that detects the HT problems between stations (STAs). In our system, APs obtain the information of connected STAs from received frames and the HT problems are identified from the integration of the information obtained at different APs. The effectiveness is verified by simulations.

We propose a new automatic head modeling system to generate individualized head models which can express person-specific facial expressions. The head modeling system consists of two core processes. The head modeling process with the proposed automatic mesh completion generates a whole head model only from facial range scan data. The key shape generation process generates key shapes for the generated head model based on physics-based facial muscle simulation with an individual muscle layout estimated from subject's facial expression videos. Facial animations considering personal characteristics can be synthesized using the individualized head model and key shapes. Experimental results show that the proposed system can generate head models where 84% of subjects can identify themselves. Therefore, we conclude that our head modeling system is effective to games and entertainment systems like a Future Cast System.

Enhancements of transmission speed and mobile node capability in wireless access networks enable people to watch movies and share large files with friends via their personal mobile devices. Considering the increased demand for such services in the next five to ten years, we will need to introduce P2P architectures to mobile networks to handle the large number of requests from mobile nodes. However, in mobile P2P networks, since negative effects caused by instable links and high leave rates are propagated to down-stream nodes, it is difficult to ensure service quality and long service lifetime. To solve this problem, we describe a novel node-allocation framework using multi-dimensional context parameters of mobile nodes, which includes available bandwidth, disconnection rate, and remaining battery capacity. Taking into account the significance of each parameter, our framework integrates the context parameters into a single parameter called Relay Ability. Each node is allocated to the P2P topology in accordance with its relay ability. To test our method, we applied our architecture to overlay multicast and several of results from comparative evaluations through computer simulation.

The growth in broadband access and increases in terminal capability have enabled us to provide large-scale multimedia services. The traditional unicast architecture cannot support such a service because the transportation from a content server to many end hosts causes the overload to a content server. Overlay multicast techniques have been proposed to solve this problem. This high-layer technology allows us to provide largescale multicast services even for mobile networks. Also, the recent rapid growth of mobile broadband brought the large-scale mobile multicast services into reality. However, stream relays by users are indispensable in overlay multicast. Moreover, in mobile environments, users' willingness to relay depends on user states such as remaining battery charge, movement conditions, and the relationship with the relay requestor. In this paper, we investigate user actions and states in mobile overlay networks and model user actions as agents. Then, we discuss how user actions affect the performance of the mobile overlay network and which user states we should pay attention to in order to build effective networks through multiagent simulations.

With the rapid growth of the wireless network and the function of mobile terminal, the services which need the wide bandwidth like video streaming are expected to be used extensively over wireless networks. Overlay multicast in which a terminal relays the content to other terminals is able to deliver it to enormous terminals. However, due to instantaneous disconnection and rapid bandwidth decrease in wireless channels, we need a different architecture from the ones used for fixed environments. In this paper, we propose a new overlay-topology design, where we use the instability information of mobile terminals such as the disconnection rate and the changing range of physical transmission rate determined by the physical conditions including radio signal strength and handover. Moreover, to reduce the performance degradation caused by instantaneous disconnection and rapid bandwidth decrease and to obtain the route-diversity gain, the proposed method assigns more logical links to the terminals that likely suffer from the impact, which results in the improvement of the received stream qualities of the terminals. This paper also shows its superiority to the conventional methods in terms of service quality and robustness from the simulation results using a wireless channel-quality model we newly propose.

In wireless local area networks (WLANs), the demand of the multimedia applications such as voice telephony and peer-to-peer content distribution has followed the necessity of quality-of-service (QoS) control for uplink flows. However, in uplink WLANs, the hidden-station problem causes difficulties in the QoS control because of unfair collision probability. In this paper, we point out this hidden-station problem and clarify the following unfairness between user datagram protocol (UDP) and transmission control protocol (TCP) uplink flows: 1) the effect of collision caused by hidden-station relationship on throughput and 2) the instability of the throughput depending on the number of hidden stations. To solve these problems, we propose a mechanism that first groups stations according to the hidden-station relationship and type of transport protocol they use then assigns a transmission permitted period to each group. Our proposed mechanism eliminates the collisions between hidden stations and provides a flexible control over bandwidth allocation to UDP and TCP. Its performance is shown through simulation.