This study introduces a novel method to mitigate temporal illumination variations in whisk-broom hyperspectral imaging under varying environmental illumination conditions. Whisk-broom hyperspectral imaging captures high resolution spectra pixel-by-pixel sequentially, a process susceptible to sunlight fluctuations over time, particularly when imaging cultural artifacts outdoors. Despite sunlight's broad spectrum, its variability over time can compromise the quality of hyperspectral images, affecting data analysis. Prior strategies suggested a supplementary single compensating vertical scan alongside the standard row-wise raster scan. However, this strategy fails when the additional single vertical scan is performed near or on a black frame. Building on this, our study proposes using two scans, one performed orthogonally to the traditional row-wise scan, to counteract illumination variations. Furthermore, we formulate the compensation problem in logarithmic space, exploiting the low-dimensional structure of the reflectance and illumination spectra. Using total variation penalization in the cost function enhances smoothness. Our method achieves robust compensation for changes in environmental illumination. We also demonstrate that we can use multiple columns from the column-wise scan without significantly decreasing the compensation quality while reducing acquisition time. We illustrate the application of our methods to hyperspectral images of stained-glass windows of the historic Cathedrale Notre-Dame d'Amiens in France.

We propose a novel method for estimating head orientation from a low-resolution far-infrared (LFIR) image sequence captured by a 32 × 32 FIR sensor array. Estimating head orientation from LFIR images is challenging due to their inherent noise and limited resolution. Manually an-notating head orientation is also difficult, and no suitable dataset currently exists. We propose the LFIR2Head model, which leverages a multimodal-temporal framework for robust estimation. We also introduce the LFIR2Head dataset, which was created using an automatic annotation system. Extensive experiments demonstrate the accuracy of head orientation estimation with the proposed dataset. We con-firmed that head orientation can be estimated accurately from an LFIR image sequence.

The automatic colorization of anime line drawings is a challenging problem in production pipelines. Recent advances in deep neural networks have addressed this problem; however, collectingmany images of colorization targets in novel anime work before the colorization process starts leads to chicken-and-egg problems and has become an obstacle to using them in production pipelines. To overcome this obstacle, we propose a new patch-based learning method for few-shot anime-style colorization. The learning method adopts an efficient patch sampling technique with position embedding according to the characteristics of anime line drawings. We also present a continuous learning strategy that continuously updates our colorization model using new samples colorized by human artists. The advantage of our method is that it can learn our colorization model from scratch or pre-trained weights using only a few pre- and post-colorized line drawings that are created by artists in their usual colorization work. Therefore, our method can be easily incorporated within existing production pipelines. We quantitatively demonstrate that our colorizationmethod outperforms state-of-the-art methods.

Abstract We propose deep depth from focal stack (DDFS), which takes a focal stack as input of a neural network for estimating scene depth. Defocus blur is a useful cue for depth estimation. However, the size of the blur depends on not only scene depth but also camera settings such as focus distance, focal length, and f-number. Current learning-based methods without any defocus models cannot estimate a correct depth map if camera settings are different at training and test times. Our method takes a plane sweep volume as input for the constraint between scene depth, defocus images, and camera settings, and this intermediate representation enables depth estimation with different camera settings at training and test times. This camera-setting invariance can enhance the applicability of DDFS. The experimental results also indicate that our method is robust against a synthetic-to-real domain gap.

Non-line-of-sight (NLOS) imaging is conducted to infer invisible scenes from indirect light on visible objects. The neural transient field (NeTF) was proposed for representing scenes as neural radiance fields in NLOS scenes. We propose NLOS neural implicit surface (NLOS-NeuS), which extends the NeTF to neural implicit surfaces with a signed distance function (SDF) for reconstructing three-dimensional surfaces in NLOS scenes. We introduce two constraints as loss functions for correctly learning an SDF to avoid non-zero level-set surfaces. We also introduce a lower bound constraint of an SDF based on the geometry of the first-returning photons. The experimental results indicate that these constraints are essential for learning a correct SDF in NLOS scenes. Compared with previous methods with discretized representation, NLOS-NeuS with the neural continuous representation enables us to reconstruct smooth surfaces while preserving fine details in NLOS scenes. To the best of our knowledge, this is the first study on neural implicit surfaces with volume rendering in NLOS scenes. Project page: https://yfujimura.github.io/nlos-neus/

In this paper, we propose a new cue of depth sensing using thermal radiation. Our method realizes passive, texture independent, far range, and dark scene applicability, which can broaden the depth sensing subjects. A key ob-servation is that thermal radiation is attenuated by the air and is wavelength dependent. By modeling the wavelength-dependent attenuation by the air and building a multi-spectral LWIR measurement system, we can jointly estimate the depth, temperature, and emissivity of the target. We analytically show the capability of the thermal radiation cue and show the effectiveness of the method in real-world scenes using an imaging system with a few bandpass filters.

The novelty of this paper is the alignment method of narrow field-of-view hyperspectral images to full-view RGB images. The interest is to locate hyperspectral measurements in an environment described by an equirectangular image. But the very different modalities (3 vs. hundreds of channels) and fields-of-view are challenges for accurate alignment. We solve these problems within a dense direct alignment framework that optimizes the warping parameters together with those of a global illumination difference model. Our alignment code is shared with an example dataset available at github.com/jrl-umi3218/hsrgbalign.

A parasitic egg could cause infections and become significant diseases, especially in countries with poor sanitation and hygiene. Its detection which is manually conducted by humans using a microscope could be time-consuming and potent to misclassify. The goal of this research is to get a method that has a good accuracy to automatically detect and identify the parasitic egg. Also, we have a challenge here because the dataset has different resolutions, lighting, and setting conditions. Here we try to utilize the YOLO algorithm by doing image augmentation and restoration to the dataset, we use deep latent space translation for restoration and a simple GrabCut algorithm to generate an augmented image. Furthermore, we apply hyperparameter tuning in image resolution and see how we could utilize the YOLO algorithm to do fast and accurate classification. From the result, we get the combination could give better results in classifying the parasitic egg which the mIoU results reached 71.7 percent.

Abstract We propose a method for eliminating the temporal illumination variations in whisk-broom (point-scan) hyperspectral imaging. Whisk-broom scanning is useful for acquiring a spatial measurement using a pixel-based hyperspectral sensor. However, when it is applied to outdoor cultural heritages, temporal illumination variations become an issue due to the lengthy measurement time. As a result, the incoming illumination spectra vary across the measured image locations because different locations are measured at different times. To overcome this problem, in addition to the standard raster scan, we propose an additional perpendicular scan that traverses the raster scan. We show that this additional scan allows us to infer the illumination variations over the raster scan. Furthermore, the sparse structure in the illumination spectrum is exploited to robustly eliminate these variations. We quantitatively show that a hyperspectral image captured under sunlight is indeed affected by temporal illumination variations, that a Naïve mitigation method suffers from severe artifacts, and that the proposed method can robustly eliminate the illumination variations. Finally, we demonstrate the usefulness of the proposed method by capturing historic stained-glass windows of a French cathedral.

Multispectral photometric stereo (MPS) aims at recovering the surface normal of a scene from a single-shot multi-spectral image, which is known as an ill-posed problem. To make the problem well-posed, existing MPS methods rely on restrictive assumptions, such as shape prior, surfaces having a monochromatic with uniform albedo. This paper alleviates the restrictive assumptions in existing methods. We show that the problem becomes well-posed for a surface with a uniform chromaticity but spatially-varying albedos based on our new formulation. Specifically, if at least three (or two) scene points share the same chromaticity, the proposed method uniquely recovers their surface normals and spectral reflectance with the illumination of more than or equal to four (or five) spectral lights. Besides, our method can be made robust by having many (i.e., 4 or more) spectral bands using robust estimation techniques for conventional photometric stereo. Experiments on both synthetic and real-world scenes demonstrate the effectiveness of our method. Our data and result can be found at https://github.com/GH-HOME/MultispectralPS.git.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, the transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated, and therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

The recent development of 3D printing technology has brought concerns about its potential misuse, such as in copyright infringement and crimes. Although there have been many studies on blind 3D mesh watermarking for the copyright protection of digital objects, methods applicable to 3D printed objects are rare. In this paper, we propose a novel blind watermarking algorithm for 3D printed objects with applications for copyright protection, traitor tracing, object identification, and crime investigation. Our method allows us to embed a few bits of data into a 3D-printed object and retrieve it by 3D scanning without requiring any information about the original mesh. The payload is embedded on the object's surface by slightly modifying the distribution of surface norms, that is, the distance between the surface and the center of gravity. It is robust to resampling and can work with any 3D printer and scanner technology. In addition, our method increases the capacity and resistance by subdividing the mesh into a set of bins and spreading the data over the entire surface to negate the effect of local printing artifacts. The method's novelties include extending the vertex norm histogram to a continuous surface and the use of 3D moments to synchronize a watermark signal in a 3D-printing context. In the experiments, our method was evaluated using a public dataset against center, orientation, minimum and maximum norm misalignments; a printing simulation; and actual print/scan experiments using a standard 3D printer and scanner.

The decomposition of light transport into direct and global components, diffuse and specular interreflections, and subsurface scattering allows for new visualizations of light in everyday scenes. In particular, indirect light contains a myriad of information about the complex appearance of materials useful for computer vision and inverse rendering applications. In this paper, we present a new imaging technique that captures and analyzes components of indirect light via light transport using a synchronized projector-camera system. The rectified system illuminates the scene with epipolar planes corresponding to projector rows, and we vary two key parameters to capture plane-to-ray light transport between projector row and camera pixel: (1) the offset between projector row and camera row in the rolling shutter (implemented as synchronization delay), and (2) the exposure of the camera row. We describe how this synchronized rolling shutter performs illumination multiplexing, and develop a nonlinear optimization algorithm to demultiplex the resulting 3D light transport operator. Using our system, we are able to capture live short and long-range non-epipolar indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties. In particular, we show the utility of indirect imaging for capturing and analyzing the hidden structure of veins in human skin.

We propose a new blind watermarking algorithm for 3D printed objects that has applications in metadata embedding, robotic grasping, counterfeit prevention, and crime investigation. Our method can be used on fused deposition modeling (FDM) 3D printers and works by modifying the printed layer thickness on small patches of the surface of an object. These patches can be applied to multiple regions of the object, thereby making it resistant to various attacks such as cropping, local deformation, local surface degradation, or printing errors. The novelties of our method are the use of the thickness of printed layers as a one-dimensional carrier signal to embed data, the minimization of distortion by only modifying the layers locally, and one-shot detection using a common paper scanner. To correct encoding or decoding errors, our method combines multiple patches and uses a 2D parity check to estimate the error probability of each bit to obtain a higher correction rate than a naive majority vote. The parity bits included in the patches have a double purpose because, in addition to error detection, they are also used to identify the orientation of the patches. In our experiments, we successfully embedded a watermark into flat surfaces of 3D objects with various filament colors using a standard FDM 3D printer, extracted it using a common 2D paper scanner and evaluated the sensitivity to surface degradation and signal amplitude.

Phase ambiguity is a major problem in the depth measurement in either time-of-flight or phase shifting. Resolving the ambiguity using a low frequency pattern sacrifices the depth precision, and using multiple frequencies requires a number of observations. In this paper, we propose a phase disambiguation method that combines temporal and spatial modulation so that the high depth precision is preserved while the number of observation is small. A key observation is that the phase ambiguities of temporal and spatial domains appear differently with respect to the depth. Using this difference, the phase can disambiguate for a wider range of interest. We develop a prototype to show the effectiveness of our method through real-world experiments.

Active illumination systems which perform disparity gating, or the ability to selectively image photons that arrive from a specified surface geometry some distance away, have recently shown usefulness for robotics, autonomous vehicles, and surveillance applications. In this paper, we present a new technique for sloped disparity gating, capturing a particular set of sloped planar surfaces in a scene, implemented using the synchronization between a raster-scanning projector and the rolling shutter of a camera. We demonstrate how to control the slope and thickness of these planar surfaces using hardware parameters of pixel clock, synchronization delay, and exposure. Finally, we perform applications including real-time image masking and imaging in scattering media with a real hardware prototype in the lab. This work showcases the potential for energy-efficient, geometry-aware disparity gating in the future.

In this paper, we present a practical method for reconstructing the bidirectional reflectance distribution function (BRDF) from multiple images of a real object composed of a homogeneous material. The key idea is that the BRDF can be sampled after geometry estimation using multi-view stereo (MVS) techniques. Our contribution is selection of reliable samples of lighting, surface normal, and viewing directions for robustness against estimation errors of MVS. Our method is quantitatively evaluated using synthesized images and its effectiveness is shown via real-world experiments.

Contact imaging is a compact lensless microscopy technique that allows observation of living cells placed directly on a charge-coupled device sensor. However, captured images exhibit low visibility in general. This work is aimed at integrating computational refocusing and high-frequency illumination into contact imaging to enhance the visibility of the captured images. Refocusing contributes to the synthesis of shallow depth-of-field images from the light field captured by moving a point light source. The visibility may be enhanced via high-frequency illumination. Naïve integration of both techniques requires several observations. Therefore, we propose an efficient integration from reduced observations by reformulating the computational refocusing applied in contact imaging. Furthermore, we developed a prototype system that demonstrates the effectiveness of the proposed method.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. The proposed method is built upon a key observation that the depth measurement by a ToF camera is distorted for objects with certain materials, especially with translucent materials. We show that this distortion is due to the variation of time domain impulse responses across materials and also due to the measurement mechanism of the ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the object material, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as a feature for classification and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluations and real-world experiments, showing its capability of material classification, even for visually indistinguishable objects.

In this research, we propose a novel registration method for three-dimensional (3D) reconstruction from serial section images. 3D reconstructed data from serial section images provides structural information with high resolution. However, there are three problems in 3D reconstruction: non-rigid deformation, tissue discontinuity, and accumulation of scale change. To solve the non-rigid deformation, we propose a novel non-rigid registration method using blending rigid transforms. To avoid the tissue discontinuity, we propose a target image selection method using the criterion based on the blending of transforms. To solve the scale change of tissue, we propose a scale adjustment method using the tissue area before and after registration. The experimental results demonstrate that our method can represent non-rigid deformation with a small number of control points, and is robust to a variation in staining. The results also demonstrate that our target selection method avoids tissue discontinuity and our scale adjustment reduces scale change.

We present a novel time-resolved light transport decomposition method using thermal imaging. Because the speed of heat propagation is much slower than the speed of light propagation, transient transport of far infrared light can be observed at a video frame rate. A key observation is that the thermal image looks similar to the visible light image in an appropriately controlled environment. This implies that conventional computer vision techniques can be straightforwardly applied to the thermal image. We show that the diffuse component in the thermal image can be separated and, therefore, the surface normals of objects can be estimated by the Lambertian photometric stereo. The effectiveness of our method is evaluated by conducting real-world experiments, and its applicability to black body, transparent, and translucent objects is shown.

Separation of light transport into direct and indirect paths has enabled new visualizations of light in everyday scenes. However, indirect light itself contains a variety of components from subsurface scattering to diffuse and specular interreflections, all of which contribute to complex visual appearance. In this paper, we present a new imaging technique that captures and analyzes these components of indirect light via light transport between epipolar planes of illumination and rays of received light. This plane-to-ray light transport is captured using a rectified projector-camera system where we vary the offset between projector and camera rows (implemented as synchronization delay) as well as the exposure of each camera row. The resulting delay-exposure stack of images can capture live short and long-range indirect light transport, disambiguate subsurface scattering, diffuse and specular interreflections, and distinguish materials according to their subsurface scattering properties.

Light interacts with a scene in various ways. For scene understanding, a light transport is useful because it describes a relationship between the incident light ray and the result of the interaction. Our goal is to acquire the 4D light transport between the projector and the camera, focusing on direct and short-range transport that include the effect of the diffuse reflections, subsurface scattering, and inter-reflections. The acquisition of the light transport is challenging since the acquisition of the full 4D light transport requires a large number of measurement. We propose an efficient method to acquire short range light transport, which is dominant in the general scene, using synchronized projector-camera system. We show the transport profile of various materials, including uniform or heterogeneous subsurface scattering.

ChromaDepth glasses are microprism glasses that produce a pop-up effect by combining refraction and diffraction. In this paper, we formulate the relation between the pop-up distance and viewing distance using a mathematical model. Using our model and optical measurements, we evaluate the human perception of the pop-up effect for ChromaDepth glasses using a large distant display.

Acquiring accurate scattering properties is important for rendering translucent materials. In particular, the phase function, which determines the distribution of scattering directions, plays a significant role in the appearance of a material. We propose a distinctive scattering theory that approximates the effect of single scattering to acquire the non-parametric phase function from a single image. Furthermore, in various experiments, we measured the phase functions from several real diluted media and rendered images of these materials to evaluate the effectiveness of our theory.

This paper presents a material classification method using an off-the-shelf Time-of-Flight (ToF) camera. We use a key observation that the depth measurement by a ToF camera is distorted in objects with certain materials, especially with translucent materials. We show that this distortion is caused by the variations of time domain impulse responses across materials and also by the measurement mechanism of the existing ToF cameras. Specifically, we reveal that the amount of distortion varies according to the modulation frequency of the ToF camera, the material of the object, and the distance between the camera and object. Our method uses the depth distortion of ToF measurements as features and achieves material classification of a scene. Effectiveness of the proposed method is demonstrated by numerical evaluation and real-world experiments, showing its capability of even classifying visually similar objects.

In this paper, we present a regression for predicting 3D rigid transformations from real-valued vectors. We use a unit dual quaternion to represent the transformation. The regression is formulated as blending unit dual quaternions. To formulate it in a closed form, we introduce an approximation based on error metrics according to geometric algebra. Finally, we take an articulated motion and an elastic deformation as examples to present the descriptive power of our method in modeling the motion and the deformation.

Image restoration is a fundamental problem in the field of image processing. The key objective of image restoration is to recover clean images from images degraded by noise and blur. Recently, a family of new statistical techniques called variational Bayes (VB) has been introduced to image restoration, which enables us to automatically tune parameters that control restoration. While information from one image is often insufficient for high-quality restoration, however, current state-of-the-art methods of image restoration via VB approaches use only a single-degraded image to recover a clean image. In this paper, we propose a novel method of multiframe image restoration via a VB approach, which can achieve higher image quality while tuning parameters automatically. Given multiple degraded images, this method jointly estimates a clean image and other parameters, including an image warping parameter introduced for the use of multiple images, through Bayesian inference that we enable by making full use of VB techniques. Through various experiments, we demonstrate the effectiveness of our multiframe method by comparing it with single-frame one, and also show the advantages of our VB approach over non-VB approaches.

In this paper, we describe a supervised four-dimensional (4D) light field segmentation method that uses a graph-cut algorithm. Since 4D light field data has implicit depth information and contains redundancy, it differs from simple 4D hyper-volume. In order to preserve redundancy, we define two neighboring ray types (spatial and angular) in light field data. To obtain higher segmentation accuracy, we also design a learning-based likelihood, called objectness, which utilizes appearance and disparity cues. We show the effectiveness of our method via numerical evaluation and some light field editing applications using both synthetic and real-world light fields.

Sensing the intention of a user’s forthcoming action is a necessary function for systems that assist human physical activity. In this article, a strategy for recipe guidance systems that can predict the forthcoming intended subtask in a cooking task is investigated. The focus is on user accessing objects, that is, touching and releasing objects. Touching can indicate the start of the forthcoming subtask and releasing can indicate the end of the task. The main difficulty lies in the fact that humans may move objects because they are in the way and use cooking tools that are unanticipated by an assistive system. In such cases, the accessed object should not indicate the forthcoming subtask. A method is proposed to track the progress of a task based on the object access history. This enables to eliminate object accesses that are out of context. Simultaneously, the method predicts the forthcoming subtask based on a combination of progress and materials rather than tools and materials. Then, a guidance system that runs as a web service is developed. In experiments, real cooking activities navigated by this system are observed. The Wizard of OZ method is utilized to simulate a system that detects object accesses. The experimental results show that 73.6% accuracy is achieved in the selection of the displayed information. This result supports the use of “access to objects” realize effective intention-sensing systems.

In this paper, we adopt the coded aperture technique to the alignment process for industrial machinery. As a special setting for assembly and inspection, such machinery uses an illumination of narrow wavelength range for imaging with less aberration. This leads to significant influence of light diffraction on the image restoration. Although the diffraction was treated as negligible in most previous studies of coded aperture since they were carried out for natural images, the aperture patterns of them do not achieve enough accuracy for the alignment. We optimize the aperture pattern by performing a simulation of light diffraction, and we experimentally show that it achieves better performance in the alignment.

Most vision-based 3D acquisition methods including both passive and active methods have a limitation in that cameras must be able to observe the surface to be measured. If this is not possible, that is to say, if the surface is occluded, most of the methods cannot acquire the surface shape. In this paper, we present a method that can acquire the 3D points on the occluded surfaces. The main idea is to observe the scattering of reflected light in a participating medium instead of observing only the reflected light. We place the target in a participating medium, specifically a liquid tank filled with a participating medium, and irradiate a laser beam on it. Even if the reflecting point is occluded, our method can acquire the 3D position of occluded reflecting points from the scattering. Experimental results showed that our method can measure 3D reflecting points on several objects even if the objects are completely occluded.

In this paper we describe a method for generating a procedural text given its flow graph representation. Our main idea is to automatically collect sen-tence skeletons from real texts by re-placing the important word sequences with their type labels to form a skeleton pool. The experimental results showed that our method is feasible and has a potential to generate natural sentences. 1

This article discusses the problem of recognizing groups of people in conversation with each other in an open space. Previous work on this problem takes an approach based on the knowledge that people in the same conversation group often makes a circular formation called an F-formation, by referring to the work in social psychology. Since the F-formation describes spatial and orientational characteristics of the lower bodies of people, their positions and orientations need to be obtained for employing the approach. However, it is difficult to observe lower bodies by cameras, especially for people in a circular formation, due to occlusions between their bodies. We propose to recognize conversation groups while estimating the positions and orientations of the lower bodies of the people only from the information of their head positions and facial orientations observable from a single camera, by considering that the lower bodies of the people are arranged in a circular pattern if they are in the same conversation group.

Most shape and reflectance acquisition methods use the light reflected from objects' surfaces. When the reflection cannot be observed, e.g., when the object's surface is black matte or highly specular, its shape and reflectance are difficult to acquire. In this paper, we propose a method that can measure shape and reflectance with another approach. Our method involves the use of the scattering of reflected light in a participating media instead of using only the light reflection. We place the target in a participating medium and focus a laser beam on it. Cameras can observe the scattering of reflected light toward all reflection angles even if they can only observe light reflected toward the cameras. Experimental results showed that our method can acquire the shape and reflectance of various surfaces.

In this paper, we propose a novel method of skeleton estimation for the purpose of constructing and manipulating individualized hand models via a data glove. To reconstruct actual hand accurately, we derive motion constraints in fully 6-DOF at the joints without assuming either a center of rotation or a joint axis. The constraints are derived by regression analysis on the configuration of the finger segments with respect to the sensor data. In order to acquire pairs of the configuration and the sensor data, we introduce graspable reference objects for reproducing postures with and without the data glove. To achieve accurate regression on fewer samples, we introduce a regression model according to a detailed investigation using a fused motion capture system, which enabled us to perform optical motion capture and sensor data collection simultaneously. The effectiveness of the method is demonstrated through practical applications involving grasped reference objects.

This paper discusses virtual object manipulation using a human hand in an AR environment. For this purpose, we need to measure the configuration, which includes the position, pose and posture, of the human hand. However, if we employ a data glove for the measurement, the glove appears in the video images synthesized by AR. If we otherwise employ a vision-based approach, the obtained configuration often includes larger errors due to occlusions among fingers. Moreover, the configuration of the human hand in anyway often includes some cognitive errors in manipulating the virtual object, which is only visible in the synthesized videos. To cope with these problems, we propose to replace the human hand with a data glove with a virtual hand in the video and adjust its configuration so that it properly grasps the virtual objects without errors.

This summary introduces the aim of the CEA'13 workshop and the list of papers presented in the workshop.

Person re-identification is one of the most challenging issues in network-based surveillance. The difficulties mainly come from the great appearance variations induced by illumination, camera view and body pose changes. Maybe influenced by the research on face recognition and general object recognition, this problem is habitually treated as a verification or classification problem, and much effort has been put on optimizing standard recognition criteria. However, we found that in practical applications the users usually have different expectations. For example, in a real surveillance system, we may expect that a visual user interface can show us the relevant images in the first few (e.g. 20) candidates, but not necessarily before all the irrelevant ones. In other words, there is no problem to leave the final judgement to the users. Based on such an observation, this paper treats the re-identification problem as a ranking problem and directly optimizes a listwise ranking function named Mean Reciprocal Rank (MRR), which is considered by us to be able to generate results closest to human expectations. Using a maximum-margin based structured learning model, we are able to show improved re-identification results on widely-used benchmark datasets.

In this paper, we propose a real-time system for measuring the consumption of various types of seasonings. In our system, all seasonings are placed on a scale, and we continuously take images of these items using a camera. Our system estimates the consumption of each condiment by calculating the difference between the weight when the seasoning was picked up and the weight when it was placed back on the scale. Our system identifies the type of seasoning that was used by determining whether or not the seasoning was present on the scale. By using our system, users can automatically log their usage of seasoning. Then, they can adjust the seasoning according to their desired taste.

This paper presents a method for recognizing recipe ingredients based on the load on a chopping board when ingredients are cut. The load is measured by four sensors attached to the board. Each chop is detected by indentifying a sharp falling edge in the load data. The load features, including the maximum value, duration, impulse, peak position, and kurtosis, are extracted and used for ingredient recognition. Experimental results showed a precision of 98.1% in chop detection and 67.4% in ingredient recognition with a support vector machine (SVM) classifier for 16 common ingredients.

This paper describes our work in computer vision to track food materials in the food preparation process. Tracking such food materials is difficult, because they are often hidden when moved by hand. Furthermore, their appearance may change in hand when they are cut or peeled. For tracking these objects in such situations, we propose a novel method that matches an object on a cooking table to one grasped in the past. We use the following three criteria to match the objects even when they are cut or peeled: the similarity in their appearance, the validity of their change in appearance, and the grasped order. We experimentally evaluated our method by applying it to the scenes of cutting and peeling food materials. As a result, we achieved an accuracy of 83.6% in matching the objects.

3D shapes are reconstructed from silhouettes obtained by multiple cameras with the volume intersection method. In recent work, methods of integrating silhouettes in time sequences have been proposed. The number of silhouettes can be increased by integrating silhouettes in multiple frames. The silhouettes of a rigid object in multiple frames are integrated with its rigid motion. This motion is often estimated with 3D feature points extracted from silhouettes. When the estimated motion has large error, shapes are reconstructed with missing parts. This error is given by the incomplete extraction of 3D feature points, which is caused by additional and missing regions of extracted silhouettes. We cannot prevent silhouettes from being extracted with the additional and missing regions in real environments. Here, we propose an intelligent method of integrating incomplete silhouettes where outcrop points, which are 3D feature points for estimating motion, play an important role. The reconstructed shape can be evaluated referring to how many outcrop points have been included in the reconstructed shape of another frame. Although the evaluation does not represent the accuracy of estimated motion directly, it does guarantee that outstanding parts will be preserved in the reconstructed shape. Silhouettes in multiple frames can be integrated with fewer missing and additional parts based on this evaluation.