Swarm User Interfaces allow dynamic arrangement of user environments through the use of multiple mobile robots, but their operational range is typically confined to a single plane due to constraints imposed by their two-wheel propulsion systems. We present corobos, a proof-of-concept design that enables these robots to cooperatively transition between table (horizontal) and wall (vertical) surfaces seamlessly, without human intervention. Each robot is equipped with a uniquely designed slope structure that facilitates smooth rotation when another robot pushes it toward a target surface. Notably, this design relies solely on passive mechanical elements, eliminating the need for additional active electrical components. We investigated the design parameters of this structure and evaluated its transition success rate through experiments. Furthermore, we demonstrate various application examples to showcase the potential of corobos in enhancing user environments.

The imperceptible color vibration method, which embeds imperceptible patterns in display images through rapid chromatic changes, enables the presentation of information to machines while maintaining the viewing experience of users. However, when markers are embedded in video images, decoding becomes challenging during periods with significant interframe differences, which adds noise to the decoding. In this study, we enhanced the signal-point distance by embedding two quadrature chromatic changes within the color space across the entire image. Additionally, we employed soft-decision decoding, which involves accumulating multiple confidence values per pixel, thereby enabling robust decoding during periods with significant interframe differences in video images. Our evaluation confirmed that the proposed method significantly improves the decoding rate compared to conventional techniques. Furthermore, we measured the time required for decoding and verified the effectiveness of the proposed soft-decision decoding method.

We present PneuModule, a tangible interface platform that enables users to reconfigure physical controls on pressure-sensitive touch surfaces using pneumatically-actuated inflatable pin arrays. PneuModule consists of a main module and extension modules. The main module is tracked on the touch surface and forwards continuous inputs from attached multiple extension modules to the touch surface. Extension modules have distinct mechanisms for user input, which pneumatically actuates the inflatable pins at the bottom of the main module through internal air pipes. The main module accepts multi-dimensional inputs since each pin is individually inflated by the corresponding air chamber. Also, since the extension modules are swappable and identifiable owing to the marker design, users can quickly customize the interface layout. We contribute to design details of inflatable pins and diverse pneumatic input control design examples for PneuModule. We also showcase the feasibility of PneuModule through a series of evaluations and interactive prototypes.

We present ForceStamps, fiducial markers for supporting rapid prototyping of physical control interfaces on pressure-sensitive touch surfaces. We investigate marker design options for supporting various physical controls, with focusing on creating dedicated footprints and maintaining the structural stability. ForceStamps can be persistently tracked on surfaces along with the force information and other attributes. Designers without knowledge of electronics can rapidly prototype physical controls by attaching mechanisms to ForceStamps, while manipulating the haptic feedback with buffer materials. The created control widgets can be spatially configured on the touch surface to make an interface layout. We showcase a variety of example controls created with ForceStamps. In addition, we report on our analysis of a two-day musical instrument design workshop to explore the affordances of ForceStamps for making novel instruments with diverse interaction designs.

Open-set classification is a problem of handling `unknown' classes that are not contained in the training dataset, whereas traditional classifiers assume that only known classes appear in the test environment. Existing open-set classifiers rely on deep networks trained in a supervised manner on known classes in the training set; this causes specialization of learned representations to known classes and makes it hard to distinguish unknowns from knowns. In contrast, we train networks for joint classification and reconstruction of input data. This enhances the learned representation so as to preserve information useful for separating unknowns from knowns, as well as to discriminate classes of knowns. Our novel Classification-Reconstruction learning for Open-Set Recognition (CROSR) utilizes latent representations for reconstruction and enables robust unknown detection without harming the known-class classification accuracy. Extensive experiments reveal that the proposed method outperforms existing deep open-set classifiers in multiple standard datasets and is robust to diverse outliers.

Multi-task learning improves generalization performance in neural networks by sharing knowledge among related tasks. Existing models are for task combinations annotated on the same dataset; research on how to utilize the knowledge of successful single-task convolutional neural networks (CNNs) that are trained on individual datasets is limited. We propose a cross-connected CNN, an architecture that connects single-task CNNs through convolutional layers that transfer useful information to their counterparts. We evaluated the architecture with a combination of detection and segmentation using datasets of two targets: pedestrians and wild birds. Experiments demonstrate how well our CNN learns general representations from multi-task learning.

The projection of invisible data on visible images can facilitate seamless interactive projection, since data embedded in regular images is unobtrusive to human viewers. However, the previous techniques sacrificed one of the following key goals: 1) calibration-free setup; 2) full-color projection; or 3) high contrast image. In this paper, we propose a Pixel-level Infrared Light Communication (PILC) projector that achieves all these requirements by adding an infrared light source to the full-color projector. To provide a proof of concept, we built a functional prototype, evaluated its performance, and presented a basic application.

We present an accurate stereo matching method using local expansion moves based on graph cuts. This new move-making scheme is used to efficiently infer per-pixel 3D plane labels on a pairwise Markov random field (MRF) that effectively combines recently proposed slanted patch matching and curvature regularization terms. The local expansion moves are presented as many -expansions defined for small grid regions. The local expansion moves extend traditional expansion moves by two ways: localization and spatial propagation. By localization, we use different candidate -labels according to the locations of local -expansions. By spatial propagation, we design our local -expansions to propagate currently assigned labels for nearby regions. With this localization and spatial propagation, our method can efficiently infer MRF models with a continuous label space using randomized search. Our method has several advantages over previous approaches that are based on fusion moves or belief propagation; it produces submodular moves deriving a subproblem optimality; it helps find good, smooth, piecewise linear disparity maps; it is suitable for parallelization; it can use cost-volume filtering techniques for accelerating the matching cost computations. Even using a simple pairwise MRF, our method is shown to have best performance in the Middlebury stereo benchmark V2 and V3.

In face-to-face situations, tabletop displays have been used to share information among multiple users. The use of mid-air images is promising for using the space above a tabletop. However, in our previous research, the mid-air images on a tabletop were fixed in the depth direction. To design a digital PONG-like game, the mid-air image needs to move freely horizontally on the table.

We propose a passive mid-air display in which an image is made visible by illuminating an empty space with a flashlight. In this study, we design an optical configuration that reacts to the light source of the user. This design has an inconsistency of illumination that surprises the user. We confirm the basic operational principle and measure the brightness. We also observe user behavior to consider the possibility of this technique for entertainment. With this work, we make three research contributions. First, we design a passive mid-air display that can be used for entertainment displays. Second, we confirm its operational principle and measure the brightness. Third, we observe user behavior. We also summarize the advantages of our display as well as the issues that remain.

Video records are sometimes used during meetings nowadays in order to compensate for the incompleteness of hand-written minutes. To use video effectively to record the minutes of a meeting, indices are considered necessary in addition to standard functions of movie players. Some systems can form indices by sensing human actions in the world, but such indices are not directly relevant to the contents of the discussions that occur in meetings. In this study, we propose an automated meeting index system based on people's discussions during daily meetings in the world. In our system, indices based on the contents of discussion can be automatically formulated in meetings if only paper, writing implements, and a camera are arranged in a certain manner. To this end, we develop trackable sticky notes, called "Tracky Notes," as indices. Tracky Notes can be tracked with visible markers through a camera, and is robust against occlusion. We then propose a meeting viewer where Tracky Notes are used as indices. Finally, we report the result of a user study conducted in a university classroom. The study revealed the appropriate region of visible markers and the robustness against occlusion of Tracky Notes. We also measured the effectiveness of our system comparing with standard functions of movie players.

Nodding has various communicative functions in humans, such as agreement, emphasis and turn-taking and can also create various positive impressions in communication by the person exhibiting the behavior. The ultimate aim of our research is to facilitate communication by implementing nodding behavior in robots. This study analyzed videos of human conversations in groups and focused on three aspects of people's nodding behavior as they listened to others: 1) Time period to complete a nodding cycle and each nods, 2) Time delay before initiating a nodding response, and 3) Number of continuous nods used at one time. We found that: 1) The mode time period to a nod was 0.27 s, with 96% of all nods occurring within 0.17~0.57 s. 2) The mode time delay before initiating a nodding response was 0.30 s, with 95% of all nods occurring within -0.78~1.4 s. 3) Fewer than six continuous nods were used 97% of the time - one nod, 55%, two nods, 24%, three nods, 12%, four nods, 3.0%, and five nods, 2.1%. Ultimately, the research findings serve as guidelines for implementing accurate human nodding behavior in robots.

This paper tackles the problem of bird detection in large landscape images for applications in the wind energy industry. While significant progress in image recognition has been made by deep convolutional neural networks (CNNs), small object detection remains a problem. To solve it, we follow the idea that a detector can be tuned to small objects of interest and semantic segmentation methods can be complementary used to recognize large background areas. Specifically, we train a CNN-based detector, fully convolutional networks, and a superpixel-based semantic segmentation method. The results of the three methods are combined by using support vector machines to achieve high detection performance. Experimental results on a bird image dataset show the high precision and effectiveness of the proposed method.

For glass-free mixed reality (MR), mid-air imaging is a promising way of superimposing a virtual image onto a real object. We focus on attaching virtual images to non-static real life objects. In previous work, moving the real object causes latency in the superimposing system, and the virtual image seems to follow the object with a delay. This is caused by delays due to sensors, displays and computational devices for position sensing, and occasionally actuators for moving the image generation source. In order to avoid this problem, this paper proposes to separate the object-anchored imaging effect from the position sensing. Our proposal is a retro-reflective system called "SkyAnchor," which consists of only optical devices: two mirrors and an aerial-imaging plate. The system reflects light from a light source anchored under the physical object itself, and forms an image anchored around the object. This optical solution does not cause any latency in principle and is effective for high-quality mixed reality applications. We consider two types of light sources to be attached to physical objects: reflecting content from a touch table on which the object rests, or attaching the source directly on the object. As for position sensing, we utilize a capacitive marker on the bottom of the object, tracked on a touch table. We have implemented a prototype, where mid-air images move with the object, and whose content may change based on its position.

We propose an efficient and general purpose energy optimization method for binary variable energies used in various low-level vision tasks. Our method can be used for broad classes of higher-order and pairwise non-submodular functions. We first revisit a submodular-supermodular procedure (SSP) [19], which is previously studied for higher-order energy optimization. We then present our method as generalization of SSP, which is further shown to generalize several state-of-the-art techniques for higher-order and pairwise non-submodular functions [2, 9, 25]. In the experiments, we apply our method to image segmentation, deconvolution, and binarization, and show improvements over state-of-the-art methods.

Wind turbines have become significant collision risks to endangered birds. To estimate the impact of bird strikes and take action against them, we need an automated bird monitoring system that detects and categorizes bird species. Towards this goal, we constructed an image dataset as a benchmark for ecological investigations of birds. The dataset consists of time-series images of a scene in a wind farm, bounding boxes around the aerial objects in the images, and species annotations. Bird experts searched for and annotated the images of birds, and thus, even birds that appeared to be very small in the whole image could be specified in detail. In total, 32,000 birds are annotated. In collecting the images, we had to monitor birds in the distance to cover enough area, but even with a telephoto setup, the average size of the birds in the image is around 25 pixels. This means we need a means of recognition that works on very low-resolution images. As an application of the dataset, we conducted experiments in two-class categorization (birds and non-birds), to serve as basis for detection and further categorization. We found that the dataset is a challenging recognition task because of its low resolution and hard negative samples.

We propose design methods that process the surface of Urushi, a Japanese traditional coating material, with computer technology and ultraviolet (UV) light control. Urushi changes their physical condition by absorbing UV light energy. Our research introduces design method with this physical feature for Personal Fabrication by utilizing UV laser and galvanometer mirror or UV lamp and paper.

We present an accurate and efficient stereo matching method using locally shared labels, a new labeling scheme that enables spatial propagation in MRF inference using graph cuts. They give each pixel and region a set of candidate disparity labels, which are randomly initialized, spatially propagated, and refined for continuous disparity estimation. We cast the selection and propagation of locally-defined disparity labels as fusion-based energy minimization. The joint use of graph cuts and locally shared labels has advantages over previous approaches based on fusion moves or belief propagation, it produces submodular moves deriving a subproblem optimality, enables powerful randomized search, helps to find good smooth, locally planar disparity maps, which are reasonable for natural scenes, allows parallel computation of both unary and pairwise costs. Our method is evaluated using the Middlebury stereo benchmark and achieves first place in sub-pixel accuracy.

We propose an image segmentation method that divides an image into foreground and background regions when the approximate color distributions for these regions are given.Our approach was inspired by global consistency measures that directly evaluate the similarity between a given distribution and the distribution of the resulting segmentation, which were recently proposed in order to overcome the limitations of traditional pixelwise (local) consistency measures.The main feature of our proposal is that it uses two (foreground and background) input distributions, which increases the robustness compared to previous studies.To achieve this, we formulated a new mathematical model that describes the consistencies between the two input distributions and the segmentation, in which weighting parameters for the two distribution matching terms are set to be approximately proportional to the size of the foreground and background areas.We call this dual distribution matching (DDM).We also derived an optimization method that uses graph cuts.Experimental results that show the effectiveness of our method and comparisons between local and global consistency measures are presented.

This paper addresses the problem of image segmentation with a reference distribution. Recent studies have shown that segmentation with global consistency measures outperforms conventional techniques based on pixel-wise measures. However, such global approaches require a precise distribution to obtain the correct extraction. To overcome this strict assumption, we propose a new approach in which the given reference distribution plays a guiding role in inferring the latent distribution and its consistent region. The inference is based on an assumption that the latent distribution resembles the distribution of the consistent region but is distinct from the distribution of the complement region. We state the problem as the minimization of an energy function consisting of global similarities based on the Bhattacharyya distance and then implement a novel iterated distribution matching process for jointly optimizing distribution and segmentation. We evaluate the proposed algorithm on the GrabCut dataset, and demonstrate the advantages of using our approach with various segmentation problems, including interactive segmentation, background subtraction, and co-segmentation.

Free-viewpoint image synthesis (FVIS) refers to the process of generating novel viewpoint images from a set of multi-view images. Most of the conventional FVIS methods were based on image blending, so that they are subject to a fundamental limitation in resolution: the output resolution is lower than or at most equal to that of the input images. A reasonable approach to overcome this limitation is to replace image blending with reconstruction-based super-resolution. Following this idea, we propose a new FVIS method named as super-resolution plane sweeping by extending general plane sweeping methods. We also propose an adaptive weighting scheme to make super-resolution reconstruction operate only on the pixels where it improve the quality. Experimental results with real images are presented to show the effectiveness of our method.

An image/video communication scenario with super-resolution (SR) decoding, where the decoded images are upsampled using SR reconstruction at the receiver side, is analyzed from a theoretical perspective. To formulate the rate-distortion performance for such cases, we propose a new numerical model that combines a frequency-domain SR model and a rate-distortion theory for lossy image compression. We considered several factors that affect the reconstruction quality, and revealed that SR decoding performs better in low bitrates. We also conducted real-image simulations and confirmed that both the numerical analysis and real-image simulations exhibit quite similar tendencies, which supports the effectiveness of our numerical model.

SolaColor is a scheme for creating a place (namely, a floor space) whose color is varied in response to sunlight. The aim of this work is to create sustainable spatial rendition by utilizing sunlight. A feature of this spatial-rendition scheme is rendering places by color--rather than contrast---of lighting. By means of SolaColor, areas in sunlight are colored in pink, while shaded areas are colored white. When this scheme is practically applied, time-varying patterns appear in synchronization with the movement of the sun. When a person or object enters a space created by SolaColor, white shadows oriented according to the position of the sun are formed in the manner of a sundial. The scheme can be implemented by utilizing photochromic material and optical design, and it can be introduced in any location under sunlight without the need for an electrical-power supply.

This paper addresses the problem of interactive image segmentation with a user-supplied object bounding box. The underlying problem is the classification of pixels into foreground and background, where only background information is provided with sample pixels. Many approaches treat appearance models as an unknown variable and optimize the segmentation and appearance alternatively, in an expectation maximization manner. In this paper, we describe a novel approach to this problem: the objective function is expressed purely in terms of the unknown segmentation and can be optimized using only one minimum cut calculation. We aim to optimize the trade-off of making the foreground layer as large as possible while keeping the similarity between the foreground and background layers as small as possible. This similarity is formulated using the similarities of distant pixel pairs. We evaluated our algorithm on the GrabCut dataset and demonstrated that high-quality segmentations were attained at a fast calculation speed.

A new model of decomposing an image hierarchically into direction-adaptive subbands using pixel-wise direction estimation is presented. For each decomposing operation, an input image is divided into two parts: a base image subsampled from the input image and subband components. The subband components consist of residuals of estimating the pixels skipped through the subsampling, which ensures the invertibility of the decomposition. The estimation is performed in a direction-adaptive way, whose optimal direction is determined by a L1 norm criterion for each pixel, aiming to achieve good energy compaction that is suitable for image coding. Furthermore, since the L1 norms are obtained from the base image alone, we do not need to retain the directional information explicitly, which is another advantage of our model. Experimental results show that the proposed model can achieve lower entropy than conventional Haar or D5/3 discrete wavelet transform in case of lossless coding.

This paper presents a method that jointly performs synthesis of free-viewpoint images and object segmentation from that viewpoint. This method works efficiently and online by sharing a certain calculation process among rendering and segmentation steps. Since the segmentation is performed for arbitrary viewpoints directly, the extracted object can be superimposed onto another 3-D scene with geometric consistency. Experimental results using a 25-camera array show the effectiveness of our method.

The system described in this paper provides a real-time 3D visual experience by using an array of 64 video cameras and an integral photography display with 60 viewing directions. The live 3D scene in front of the camera array is reproduced by the full-color, full-parallax autostereoscopic display with interactive control of viewing parameters. The main technical challenge is fast and flexible conversion of the data from the 64 multicamera images to the integral photography format. Based on image-based rendering techniques, our conversion method first renders 60 novel images corresponding to the viewing directions of the display, and then arranges the rendered pixels to produce an integral photography image. For real-time processing on a single PC, all the conversion processes are implemented on a GPU with GPGPU techniques. The conversion method also allows a user to interactively control viewing parameters of the displayed image for reproducing the dynamic 3D scene with desirable parameters. This control is performed as a software process, without reconfiguring the hardware system, by changing the rendering parameters such as the convergence point of the rendering cameras and the interval between the viewpoints of the rendering cameras.

In this paper, we consider the rate-distortion performance of a multi- view image set with subsampling of the viewpoints. As a basic analysis, we compare two scenarios: (i) all images would be coded and transmitted in even quality, and (ii) a half of the images would be discarded by subsampling at the sender side, and the remaining half would be coded and transmitted. In the second scenario, the discarded images would be reconstructed at the receiver side using some view interpolation technology. We first introduce a theoretical model describing the rate-distortion performance of the scenarios above. Then, we present numerical simulations and experiments, showing that which scenario yields better performance depends on the bitrate, the properties of the image set, and the accuracy of the view interpolation.

This paper discusses a view interpolation problem where a new view would be synthesized at the midst of two parallel views. We first introduce a new theoretical model that describes the relation between accuracy of disparities and the quality of the interpolated view. Then, we derive the optimal prefilter that minimizes the power of errors on the interpolated view. Our approach is inspired by the rate-distortion theory for video coding, and can be also regarded as an attempt to reconsider the plenoptic sampling theory. Finally, we present numerical simulations and experimental results to validate our theory.

This paper discusses a system in which multi-view images are captured and encoded in a distributed fashion and a viewer synthesizes a novel view from this data. We developed an efficient method for such system that combines decoding and rendering process to directly synthesize the novel image without reconstructing all the input images. Our method jointly performs disparity compensation in decoding process and geometry estimation in rendering process, because they are essentially equivalent if the camera parameters for the input images are known. It achieves low-complexity for both encoder and decoder in distributed multi-view coding system. Experimental results show superior coding performance of our method compared to a conventional intra-coding method especially at low bit rate.

This paper introduces a new theoretical model on the rate-distortion performance in transmitting multi-view image data. To reduce the data amount, a practical solution is just decreasing the number of images by subsampling. The questions we focus on are (i) how much bit-rate can be reduced, and (ii) how much additional distortion would be caused, by the subsampling of images. In our theoretical model, the rate-distortion theory and the plenoptic sampling theory are combined to consider the relation between the sampling condition (the cameras' interval) and the compression efficiency. Numerical simulations which show the theoretical lower bounds for (i) and (ii) are presented with discussions.

This paper proposes a view-dependent light field coding scheme using some image-based rendering techniques prior to coding. The proposed coder first synthesizes an image at a given viewpoint, which is called a representative viewpoint, and then predicts all input images by using the synthesized image as a reference. It can produce a view-dependent scalable bitstream. This means that the quality of synthesized views around the representative viewpoint is kept high even at extremely low bit rates, and the quality of views away from there is improved according to the increase of the bit rate. Our experimental results show that this coding scheme also achieves good coding efficiency for both multi-camera images and integral photography, which are common light field representations.

A light field means a 4-D function which characterizes the flow of light rays from a target scene, and used for image-based rendering. This paper presents a novel theoretical framework which considers the aliasing problem in dealing with discrete light field data. We introduce a new scheme called the aliasing separation to isolate the additional aliasing component caused by subsampling, and give a new perspective for how to optimize the reconstruction filter to interpolate light field data without aliasing artifacts. This optimization is closely related with depth estimation. Both the focus measure for light field rendering and the multiple baseline stereo can be derived from our theoretical framework.

Light field rendering (LFR) is an image-based rendering method for synthesizing free-viewpoint images from a set of multi-view images. In LFR, no/little knowledge of geometry is required, since scene objects are assumed to be at a constant depth. This assumption causes focus-like effects on synthetic images, in which only the objects near to the assumed depth are synthesized clearly and sharply. In order to generate all-in-focus synthetic images, the authors proposed a focus measurement method which uses several kinds of reconstruction filters. Using this method, we can detect in-focus parts on multiple images that are synthesized with different assumed depths, and integrate them into one final image. Though this algorithm was shown to be effective for some real scenes, the optimal combination of reconstruction filters has not been discussed yet. In this paper, we introduce a detailed spatial domain analysis of this focus measurement method, and compare three combinations by the theoretical analysis and an experiment.

This paper introduces a novel image-based rendering method which uses inputs from unstructured cameras and synthesizes free-viewpoint images of high quality. Our method uses a set of depth layers in order to deal with scenes with large depth ranges. To each pixel on the synthesized image, the optimal depth layer is assigned automatically based on the on-the-fly focus measurement algorithm that we propose. We implemented this method efficiently on a PC and achieved nearly interactive frame-rates.

Abstract This paper proposes a novel and general method of glare generation based on wave optics. A glare image is regarded as a result of Fraunhofer diffraction, which is equivalent to a 2D Fourier transform of the image of given apertures or obstacles. In conventional methods, the shapes of glare images are categorized according to their source apertures, such as pupils and eyelashes and their basic shapes (e.g. halos, coronas, or radial streaks) are manually generated as templates, mainly based on statistical observation. Realistic variations of these basic shapes often depend on the use of random numbers. Our proposed method computes glare images fully automatically from aperture images and can be applied universally to all kinds of apertures, including camera diaphragms. It can handle dynamic changes in the position of the aperture relative to the light source, which enables subtle movement or rotation of glare streaks. Spectra can also be simulated in the glare, since the intensity of diffraction depends on the wavelength of light. The resulting glare image is superimposed onto a given computer‐generated image containing high‐intensity light sources or reflections, aligning the center of the glare image to the high‐intensity areas. Our method is implemented as a multipass rendering software. By precomputing the dynamic glare image set and putting it into texture memory, the software runs at an interactive rate.

The goal of our research is to support cooperative work performed by stakeholders sitting around a table. To support such cooperation, various table-based systems with a shared electronic display on the tabletop have been developed. These systems, however, suffer the common problem of not recognizing shared information such as text and images equally because the orientation of their view angle is not favorable. To solve this problem, we propose the Lumisight Table. This is a system capable of displaying personalized information to each required direction on one horizontal screen simultaneously by multiplexing them and of capturing stakeholders' gestures to manipulate the information.

Light field rendering is a fundamental method for synthesizing free-viewpoint images from a set of multiviewpoint images. In the simplest case, the scene structure is approximated by a simple plane: a focal plane. This approximation leads to focus-like effects on synthetic images where the focused depth is determined by the focal plane. A serious problem is that the range of the focused depth is too small in most practical cases. In this paper, we propose a focus measure that is specialized for synthetic images by light field rendering. When a set of differently-focused images is generated at a given viewpoint, the proposed focus measure enables us to obtain a depth map and an all in-focus image. Our approach has some remarkable differences from other related techniques, such as depth-from-stereo and depth-from-focus methods. Experimental results show that the proposed method effectively enhances PSNR of the final synthetic images.

This paper proposes a general method of glare generation based on wave optics. A glare image is regarded as a result of Fraunhofer diffraction, which is equivalent to a 2D Fourier transform of the image of given apertures or obstacles. In conventional methods, the shapes of glare images are categorized according to their source apertures, such as pupils and eyelashes and their basic shapes (e.g. halos, coronas or radial streaks) are manually generated as templates, mainly based on statistical observation. Realistic variations of these basic shapes often depend on the use of random numbers. Our proposed method computes glare images fully automatically from aperture images and can be applied universally to all kinds of apertures, including camera diaphragms. It can handle dynamic changes in the position of the aperture relative to the light source, which enables subtle movement or rotation of glare streaks. Spectra can also be simulated in the glare, since the intensity of diffraction depends on the wavelength of light. The resulting glare image is superimposed onto a given computer-generated image containing high intensity light sources or reflections, aligning the center of the glare image to the high intensity areas. Our method is implemented as a multipass rendering software. By precomputing the dynamic glare image set and putting it into texture memory, the software runs at an interactive rate.

This paper focuses on the sampling problem in light field rendering (LFR) that is a fundamental approach to image based rendering. Quality of LFR depends on a light ray database generated from pre-acquired images, since image synthesis is a process of gathering appropriate light ray data from the database. For improving the quality, interpolation of light ray data is effective. It is based on an assumption that scene objects are placed on a plane called "focal plane". According to the depth of the focal plane (distance between cameras and focal plane), focus-like effect would appear on the synthesized images. In this paper, we formulate the depth of field In light field rendering to address the range of depth where scene objects are rendered in focus. In our theory, the plenoptic sampling theory is generalized to compare with some other related works. Our representation is intuitive and useful for quantitative analysis of LFR.

This paper proposes real-time interactive systems illustrating your shadows cast on the floor, which we name "graphic shadow." They will make you experience an exciting space of interaction performing an illusion on your own shadows.

We demonstrate a real-time interactive system illuminating your shadows cast on the floor. A user occluding the projector light may create undesirable shadow in front-projector-based systems. We overcome and turn this problem to our advantage with an elaborately designed optics including multi-projection techniques. The systems will make you experience an exciting space of interaction performing an illusion on your shadow.

It is obvious that faster processors are essential for virtual reality applications. Recently, we have a new generation of graphics hardware which can enhance the reality of interactive graphics. Although it is designed for rendering purpose, its programmability is expected to be effective for other general applications. This paper evaluates its performance as an image processing unit especially for image filtering and stereo matching. First, we focus on image filtering, and compare GPU (ATI RADEON 9700 Pro) implementations to optimized CPU (Intel Pentium 4 3.06GHz) ones. Experimental results show that, for linear filtering, GPUs are from three to six times faster than CPUs. Then, we implement a block-based stereo matching algorithm using filters and depth-test-based optimization on the GPU. This implementation is shown to be twice faster than a CPU implementation. Finally, a system based on this algorithm is constructed to estimate depth from two video sequences in real time.

We propose the concepts of virtual light and virtual shadow with the aim of achieving a mixed reality environment focused on shadows. In this proposal, we divide the concept of virtual shadow into four categories, and among them, implement four types of interactive applications: (a) real to virtual shadows for rigid objects; (b) real to virtual shadows for non-rigid objects; (c) image-based virtual to virtual shadows, and (d) virtual to real shadows. In these applications, we can see the shadow of a real object projected onto the virtual world and vice versa. These proposed concepts should contribute to the realization of a mixed reality environment that provides a novel sense of interaction.

We have been investigating a high-speed image-based rendering system. In this area, most of the conventional systems sacrifice spatial or temporal resolution for a heavy amount of input images. When the system uses a camera-array on its input, this problem is more obvious. However, the required image data for the rendering are only a portion of them, determined by the position of the imaginary-view. We propose an image based rendering system, which uses pixel independent random access image sensors to eliminate the bottleneck of the conventional systems. We have developed a prototype of a CMOS image sensor, which has 128 /spl times/ 128 pixels. We verified that the prototype chip readouts externally selected pixels at 60 frames/second.

This paper presents a method to create a panorama disparity map of a natural environment from stereo images captured by our original multi-purpose 3-camera panorama capturing system which features accurate frame synchronization between 3 channels and can be used outdoors through battery operation. For robust determination of correspondences of a stereo pair, we make use of a census transform, a kind of non-parametric local transform. The census transform summarizes local image structure as a bit string, and gives very good stereo-matching results without such processes as image pyramids, bi-directional search, complicated matching evaluation, and too much edge detection dependent interpolation. To interpolate unknown disparities, we introduced a process influenced by the K-means algorithm. Then, use of 3-channel images and overlapping of the maps give multiple disparity values for one pixel, which enables screening of very confident values. We have obtained very good results on such complex scenes as contain branches and leaves of trees.

Multi-media ambiance communication refers to a means of shared-space communication, that makes use of actual moving pictures captured by video camera, combined with the laws of perspective, as used in painting and the visual characteristics of human beings, to establish a photo-realistic quality three-dimensional image space that users can naturally feel part of. We aim to enable this ambiance communication by basing the shared-space on actual moving pictures rather than on an accurate three-dimensional image space, such as is used in computer graphics.

In this paper, an efficient analysis method for global head motion, and a synthesis method of 3-D face scenes using model-based image coding techniques on the Internet browser are presented. By applying the techniques of model-based image coding in facial image sequence, human face or anthropomorphic agents can be synthesized with real facial motion on the internet browser using VRML and/or JAVA. While typical model-based image coding produces synthetic face scenes in 2-D space at the receiver terminals, the proposed scheme can produce synthetic face scenes in 3-D space interactively supporting some visual standard. This enables users to synthesize 3-D face scenes at any terminals. In this sense, it improves the portability of synthesis in model-based image coding. To speed up the computation in the analysis process, a new method that generates referential frame dynamically is proposed. To produce a 3-D face model which is suitable for the purpose of application, an idea of automatic face modeling using different 3-D face models is presented. Experimental results show the usefulness of applying the techniques of model-based image coding to the analysis of facial motion and synthesis of 3-D face scene on the internet browser.

Multi-screen projection systems are widely used in the field of virtual reality. Within the system bounded by screens, users can view the panoramic stereoscopic images projected onto the screens. In such an immersive environment, however, just one viewer can feel the natural depth sensation. The authors propose a novel approach to realize a multi-user system, in which several users can feel the adequate depth sensation simultaneously. Experimental results show the potential applicability of the proposed method to the next-generation virtual environment.

For the 3-D image communication, a flexible interpolation technique of a multi-viewpoint image set (MVIS) is required. If we could recover the 3-D structure from an MVIS, interpolative images would be obtained. This approach, however, can not be easily applied to arbitrary objects because of the difficulty in stereo-matching. In this paper, the self-similarity modeling which has been studied in the field of data compression is chosen as a modeling scheme which is scalable to any resolution, and the interpolation is virtually achieved by enhancing the resolution of the viewpoint-axis of an MVIS. This new approach is free from stereo-matching and is shown to achieve interpolative images with an SNR of about 35 dB.>

This paper deals with two important problems for the 3-D image communication: (i) how to synthesize intermediate-viewing-angle images of a multi-view 3-D image, and (ii) how to compress the tremendous amount of data required for the stereoscopic visual effect. In order to synthesize a 3-D image having a larger number of views from a 3-D image having fewer views, we propose a new concept of view interpolation technique. The resolution of the viewpoint-axis of a multi-view 3-D image is enhanced for the purpose of increasing the number of viewpoints. We present a new data compression method which is scalable to any resolution, i.e. scalable to any number of views. For this purpose, the fractal coding scheme with the feature of resolution independence is examined, and is shown to be applicable to 3-D image communication.>