Abstract This paper proposes a novel and general method of glare generation based on wave optics. A glare image is regarded as a result of Fraunhofer diffraction, which is equivalent to a 2D Fourier transform of the image of given apertures or obstacles. In conventional methods, the shapes of glare images are categorized according to their source apertures, such as pupils and eyelashes and their basic shapes (e.g. halos, coronas, or radial streaks) are manually generated as templates, mainly based on statistical observation. Realistic variations of these basic shapes often depend on the use of random numbers. Our proposed method computes glare images fully automatically from aperture images and can be applied universally to all kinds of apertures, including camera diaphragms. It can handle dynamic changes in the position of the aperture relative to the light source, which enables subtle movement or rotation of glare streaks. Spectra can also be simulated in the glare, since the intensity of diffraction depends on the wavelength of light. The resulting glare image is superimposed onto a given computer‐generated image containing high‐intensity light sources or reflections, aligning the center of the glare image to the high‐intensity areas. Our method is implemented as a multipass rendering software. By precomputing the dynamic glare image set and putting it into texture memory, the software runs at an interactive rate.

This paper proposes a general method of glare generation based on wave optics. A glare image is regarded as a result of Fraunhofer diffraction, which is equivalent to a 2D Fourier transform of the image of given apertures or obstacles. In conventional methods, the shapes of glare images are categorized according to their source apertures, such as pupils and eyelashes and their basic shapes (e.g. halos, coronas or radial streaks) are manually generated as templates, mainly based on statistical observation. Realistic variations of these basic shapes often depend on the use of random numbers. Our proposed method computes glare images fully automatically from aperture images and can be applied universally to all kinds of apertures, including camera diaphragms. It can handle dynamic changes in the position of the aperture relative to the light source, which enables subtle movement or rotation of glare streaks. Spectra can also be simulated in the glare, since the intensity of diffraction depends on the wavelength of light. The resulting glare image is superimposed onto a given computer-generated image containing high intensity light sources or reflections, aligning the center of the glare image to the high intensity areas. Our method is implemented as a multipass rendering software. By precomputing the dynamic glare image set and putting it into texture memory, the software runs at an interactive rate.

This paper focuses on the sampling problem in light field rendering (LFR) that is a fundamental approach to image based rendering. Quality of LFR depends on a light ray database generated from pre-acquired images, since image synthesis is a process of gathering appropriate light ray data from the database. For improving the quality, interpolation of light ray data is effective. It is based on an assumption that scene objects are placed on a plane called "focal plane". According to the depth of the focal plane (distance between cameras and focal plane), focus-like effect would appear on the synthesized images. In this paper, we formulate the depth of field In light field rendering to address the range of depth where scene objects are rendered in focus. In our theory, the plenoptic sampling theory is generalized to compare with some other related works. Our representation is intuitive and useful for quantitative analysis of LFR.

This paper proposes real-time interactive systems illustrating your shadows cast on the floor, which we name "graphic shadow." They will make you experience an exciting space of interaction performing an illusion on your own shadows.

We demonstrate a real-time interactive system illuminating your shadows cast on the floor. A user occluding the projector light may create undesirable shadow in front-projector-based systems. We overcome and turn this problem to our advantage with an elaborately designed optics including multi-projection techniques. The systems will make you experience an exciting space of interaction performing an illusion on your shadow.

It is obvious that faster processors are essential for virtual reality applications. Recently, we have a new generation of graphics hardware which can enhance the reality of interactive graphics. Although it is designed for rendering purpose, its programmability is expected to be effective for other general applications. This paper evaluates its performance as an image processing unit especially for image filtering and stereo matching. First, we focus on image filtering, and compare GPU (ATI RADEON 9700 Pro) implementations to optimized CPU (Intel Pentium 4 3.06GHz) ones. Experimental results show that, for linear filtering, GPUs are from three to six times faster than CPUs. Then, we implement a block-based stereo matching algorithm using filters and depth-test-based optimization on the GPU. This implementation is shown to be twice faster than a CPU implementation. Finally, a system based on this algorithm is constructed to estimate depth from two video sequences in real time.

We propose the concepts of virtual light and virtual shadow with the aim of achieving a mixed reality environment focused on shadows. In this proposal, we divide the concept of virtual shadow into four categories, and among them, implement four types of interactive applications: (a) real to virtual shadows for rigid objects; (b) real to virtual shadows for non-rigid objects; (c) image-based virtual to virtual shadows, and (d) virtual to real shadows. In these applications, we can see the shadow of a real object projected onto the virtual world and vice versa. These proposed concepts should contribute to the realization of a mixed reality environment that provides a novel sense of interaction.

In this paper, an efficient analysis method for global head motion, and a synthesis method of 3-D face scenes using model-based image coding techniques on the Internet browser are presented. By applying the techniques of model-based image coding in facial image sequence, human face or anthropomorphic agents can be synthesized with real facial motion on the internet browser using VRML and/or JAVA. While typical model-based image coding produces synthetic face scenes in 2-D space at the receiver terminals, the proposed scheme can produce synthetic face scenes in 3-D space interactively supporting some visual standard. This enables users to synthesize 3-D face scenes at any terminals. In this sense, it improves the portability of synthesis in model-based image coding. To speed up the computation in the analysis process, a new method that generates referential frame dynamically is proposed. To produce a 3-D face model which is suitable for the purpose of application, an idea of automatic face modeling using different 3-D face models is presented. Experimental results show the usefulness of applying the techniques of model-based image coding to the analysis of facial motion and synthesis of 3-D face scene on the internet browser.

Multi-screen projection systems are widely used in the field of virtual reality. Within the system bounded by screens, users can view the panoramic stereoscopic images projected onto the screens. In such an immersive environment, however, just one viewer can feel the natural depth sensation. The authors propose a novel approach to realize a multi-user system, in which several users can feel the adequate depth sensation simultaneously. Experimental results show the potential applicability of the proposed method to the next-generation virtual environment.

The model-based coding is a new semantic-based coding which utilizes a common knowledge database being both at a transmitter and its receiver.We have proposed the interactive model-based coding for the face metaphor user interface applications such as multimedia e-mail, WWW and agents.Three points are addressed in this paper.(l)We have constructed a framework of interactive model-based coding tool in network communications environment.(2)We show that motion parameter description for face expressions should have both CGbased synthesis and natural image-based synthesis.The corresponding descriptions by relative and absolute way are necessary for sufficient face metaphor expressions.(3)We have developed the encoder and decoder.The motion parameters are transmitted from experimental server to a receiver.The decoded motion picture can be demonstrated both by a videotape and by a PC on-line browsing through the internet.

This paper presents a new coding method representing moving images with planes. For the purpose of handling 3-D motion and depth information, we extend the layered representation and propose a new planar representation, called "setting representation." 3-D descriptions are efficient for handling images, compared with 2-D descriptions. The estimation method of 3-D structured information and the integration method of 3-D motion are also addressed. It is not easy, however, to extract consistent 3-D information under various conditions. On the other hand, even 2-D information is enough for describing certain parts of the image. We combine the layered and the setting representations and propose a new video coding method based on 2-D/3-D structured information.

This paper is concerned with the data compression and interpolation of multi-view images. We propose the affine-based disparity compensation based on a geometric relationship. We first investigate the geometric relationship between the point in object space and its projection onto a view image. Then, we propose the disparity compensation based on the affine transform, which utilize the geometric constraints between view images. In this scheme, multi-view images are compressed into the structure and texture of the triangular patches. This scheme not only compresses the multi-view image but also synthesize the view images from any viewpoints in the viewing zone, because the geometric relationship is taken into account. Finally, we report an experiment, where 19 view images were used as the original multi-view image and the amount of data was reduced to 1/19 with an SNR of 34 dB.>

This paper presents an automated approach to detecting scene changes and activities which are meaningful to the user. We show that scene changes and activities may by treated as as a collection of motion discontinuities. We then discuss how this formulation call be used to transform video streams into features such as the sign of the Gaussian and mean curvature of spatiotemporal surfaces. The measurable features may be used to partition the video stream, mark occurrence in database timeline, and characterize shots in a video database. We present video segmentation experiments involving real video data.>

For the 3-D image communication, a flexible interpolation technique of a multi-viewpoint image set (MVIS) is required. If we could recover the 3-D structure from an MVIS, interpolative images would be obtained. This approach, however, can not be easily applied to arbitrary objects because of the difficulty in stereo-matching. In this paper, the self-similarity modeling which has been studied in the field of data compression is chosen as a modeling scheme which is scalable to any resolution, and the interpolation is virtually achieved by enhancing the resolution of the viewpoint-axis of an MVIS. This new approach is free from stereo-matching and is shown to achieve interpolative images with an SNR of about 35 dB.>

This paper presents an algorithmic procedure for the generation of a 3D wireframe model from range image. The basic idea is to represent a surface by dominant points which have important shape attributes. To determine dominant points, we propose a criterion based on 3D invariant characteristics of surfaces, which we call mean-square curvature. This quantity has some desirable properties and, compared to conventional Gaussian curvature or mean curvature, is more suitable for dominant point selection. It also allows a physical explanation. The 3D wireframe model is constructed by detecting dominant points followed by triangulating them in 3D space. The algorithms are tested on a real range image and the results are shown.>

We present a novel approach for the interpolation of intermediate images from images captured at adjacent viewpoints. The key insight of this approach is that no matter where a camera moves about a scene, for any particular feature, the lines of sight from the camera's principal point, through that feature in space all intersect at the feature. From mathematical duality, the duals of these lines of sight lie along a feature trajectory whose dual is the scene point. Coupling this duality with several constraints imposed on the image acquisition process, intermediate views can be generated from adjacent views by interpolating lines of sight along the feature trajectories. Pixel-by-pixel correspondence between views is simpler than in a stereo matching system because the densely sampled views change minimally from frame to frame. Visibility determination of holes and overlaps is based on the grouping of segmented regions of epipolar plane images (EPIs) into aspects. Once the aspects are recovered, we use the aspect hierarchy to establish a layered structure of feature trajectories and their visual events.>

This paper deals with two important problems for the 3-D image communication: (i) how to synthesize intermediate-viewing-angle images of a multi-view 3-D image, and (ii) how to compress the tremendous amount of data required for the stereoscopic visual effect. In order to synthesize a 3-D image having a larger number of views from a 3-D image having fewer views, we propose a new concept of view interpolation technique. The resolution of the viewpoint-axis of a multi-view 3-D image is enhanced for the purpose of increasing the number of viewpoints. We present a new data compression method which is scalable to any resolution, i.e. scalable to any number of views. For this purpose, the fractal coding scheme with the feature of resolution independence is examined, and is shown to be applicable to 3-D image communication.>

This paper proposes new methods for analyzing image sequences and updating textures of the three-dimensional (3-D) facial model. It also describes a method for synthesizing various facial expressions. These three methods are the key technologies for the model-based image coding system. The input image analysis technique directly and robustly estimates the 3-D head motions and the facial expressions without any two-dimensional (2-D) entity correspondences. This technique resolves the 2-D correspondence mismatch errors and provides quality reproduction of the original images by fully incorporating the synthesis rules. To verify the analysis algorithm, the paper performs quantitative and subjective evaluations. It presents two methods for updating the texture of the facial model to improve the quality of the synthesized images. The first method focuses on the facial parts with large change of brightness according to the various facial expressions for reducing the transmission bit rates. The second method focuses on all changes of brightness caused by the 3-D head motions as well as the facial expressions. The transmission bit rates are estimated according to the update methods. For synthesizing the output images, it describes rules that simulate the facial muscular actions because the muscles cause the facial expressions. These rules more easily synthesize the high-quality facial images that represent the various facial expressions.>

Given the small bit allocation for motion information in very low bit-rate coding, motion compensation using the block matching (BMA) algorithm fails to maintain an acceptable level of prediction errors. The reason is that the motion model, or spatial transformation, assumed in block matching cannot approximate the motion in the real world precisely with a small number of parameters (i.e. motion vectors). To develop an effective motion compensation method for very low bit-rate video coding, The authors address the issues of 1) adopting more sophisticated spatial transformations than block matching, and 2) developing a motion estimation algorithm that is suitable for these spatial transformations. The spatial transformations discussed are affine transformation, bilinear transformation and perspective transformation. Two new motion estimation algorithms, a matching-based algorithm and its fast version, are presented. The performance of the motion compensation methods, which combine the spatial transformations and the proposed motion estimation algorithms, is evaluated theoretically and experimentally using the following criteria: prediction error, amount of motion information, and computational cost. The simulation results show that in the proposed method, the amount of motion information required to achieve a fixed level of prediction error is reduced to one fourth that of block matching. The authors combine the designed motion compensation method with the discrete cosine transform to simulate coding of CIF format images at the transmission rate of 16 kb/s.>

A new representation of dynamic scene objects to facilitate the compression and indexing of image sequences is presented. Exploiting the spatiotemporal (ST) piecewise-continuity of visual cues, the dynamic scene objects are represented as a collection of motion discontinuities. Each discontinuity signals either the start of a different motion or the occurrence of occlusion. It is shown that the sign of the Gaussian and mean curvature of the ST surfaces is sufficient for qualitatively representing the motion discontinuities. An estimation technique is developed to compute the sign of surface curvature from ST volume data, and the transformation rules that relate the image motion to the type of surface patch are described.>

A method is proposed which analyzes and synthesizes image sequences, and updates a three-dimensional facial model in knowledge-based image coding. Rules for synthesizing the output images are formulated to simulate the facial muscular actions. The input image analysis technique estimates the head motion and the facial actions directly and robustly without any correspondences. This technique also provides good reproduction of the original images because it is incorporated with the synthesis rules. The presented method for updating the facial model improves analysis accuracy and quality of the synthesized images.>

A motion-based approach is presented to recovering the nonrigid deformation of free-form curved objects. Protrusions and indentations of contours are typically the results of altering curvature structure along the symmetry axes. The recovery task is formulated as a search for a set of lines of curvature on the spatiotemporal surface. These lines of curvature correspond to the tracings of curvature extrema of the plane contour as it is drawn across the time axis. Three-dimensional steerable filters are developed to compute the lines of curvature. The filters analyze the gradient of Gaussian curvature, remove noise by angularly adaptive filtering, and select the dominant orientation direction of the lines of curvature.>

An incremental approach to understanding 3-D structure and motion of rigid as well as nonrigid objects from image sequences is presented. To be applicable to natural imagery, the recovery process should meet two requirements: robustness to noise and the ability to cope with deformable objects. These requirements can be met by using a smoothness-of-motion constraint. The smoothness-of-motion constraint provides a framework for temporal integration of motion information over a longer sequence. The 3-D structure and motion information is smoothly and successively updated to agree with the observed transformation in the range. A model of rigid and nonrigid motion is introduced, and the smoothness-of-motion constraint is formulated as a stabilizing function. Some preliminary results are also given.>

Automatic media conversion schemes from speech to a facial image and a construction of a real-time image synthesis system are presented. The purpose of this research is to realize an intelligent human-machine interface or intelligent communication system with synthesized human face images. A human face image is reconstructed on the display of a terminal using a 3-D surface model and texture mapping technique. Facial motion images are synthesized by transformation of the 3-D model. In the motion driving method, based on vector quantization and the neural network, the synthesized head image can appear to speak some given words and phrases naturally, in synchronization with voice signals from a speaker.>

An image coding method that combines model-based coding with conventional waveform coding is presented. Despite its potential to realize image communication at very low rates, model-based coding still has many glitches that need to be solved before any practical use. The major problems are the difficulty in modeling unknown objects and the presence of analysis errors. To cope with these difficulties, waveform coding is incorporated into model-based coding. The incorporated waveform coder can code unmodeled objects and cancel errors of the analysis system. A general model of model-based/waveform hybrid coding is discussed. Preliminary simulation results of a model-based/MC-DCT hybrid coding system show that this hybrid coding method is effective at very low transmission rates, such as 16 kb/s.>

An automatic field motion image synthesis scheme (driven by speech) and a real-time image synthesis design are presented. The purpose of this research is to realize an intelligent human-machine interface or intelligent communication system with talking head images. A human face is reconstructed on the display of a terminal using a 3-D surface model and texture mapping technique. Facial motion images are synthesized naturally by transformation of the lattice points on 3-D wire frames. Two driving motion methods, a text-to-image conversion scheme and a voice-to-image conversion scheme, are proposed. In the first method, the synthesized head image can appear to speak some given words and phrases naturally. In the second case, some mouth and jaw motions can be synthesized in synchronization with voice signals from a speaker. Facial expressions other than mouth shape and jaw position can be added at any moment, so it is easy to make the facial model appear angry, to smile, to appear sad, etc., by special modification rules. These schemes were implemented on a parallel image computer system. A real-time image synthesizer was able to generate facial motion images on the display at a TV image video rate.>

An image sequence coding scheme based on the extracted information of 3-D structure and motion is presented. The goal is to achieve efficient and high-quality visual telecommunications through the use of 3-D scene characteristics inherent in image sequences. The structure and motion of rigid and nonrigid objects are robustly estimated from the displacement vectors between frames. As an image source model in the image coding process, the 3-D information provides more efficient representations of image sequences because they are derived directly from the scene characteristics. Experimental results are given and suggest that the scheme can be a viable solution in realizing high-quality visual communications.>

Automatic facial motion image synthesis schemes and a real-time system design are presented. The purpose of this schemes is to realize an intelligent human-machine interface or intelligent communication system with talking head images. Human's face is reconstructed with 3D surface model and texture mapping technique on the display of terminal. Facial motion images are synthesized naturally by transformation of the lattice points on wire frames. Two types of motion drive methods, text to image conversion and speech to image conversion are proposed in this paper. In the former manner, synthesized head can speak some given texts naturally and in the latter case, some mouth and jaw motions can be synthesized in time to speech signal of behind speaker. These schemes were implemented to a parallel image computer and a real-time image synthesizer could output facial motion images to the display as fast as video rate.

The work herein, applying the concept of irreversible coding via copying to low-rate video compression, has devised some new variants of universal pattern-matching interframe coding (PMIC), which produces the extra effect of generalizing the definition of a search area used in the conventional block-matching motion compensation. We have experimentally shown the performance gain provided by the generalization within the framework of irreversible coding via copying, and demonstrated that PMIC is useful and potential as a basic means for low-rate video compression.

The authors propose and compare two types of model-based facial motion coding schemes, i.e. synthesis by rules and synthesis by parameters. In synthesis by rules, facial motion images are synthesized on the basis of rules extracted by analysis of training image samples that include all of the phonemes and coarticulation. This system can be utilized as an automatic facial animation synthesizer from text input or as a man-machine interface using the facial motion image. In synthesis by parameters, facial motion images are synthesized on the basis of a code word index of speech parameters. Experimental results indicate good performance for both systems, which can create natural facial-motion images with very low transmission rate. Details of 3-D modeling, algorithm synthesis, and performance are discussed.>

The authors have developed a novel universal pattern-matching image compression scheme (PMC) that avoids the pattern-set mismatch problem. The spatial vector quantization approach was enhanced by self-organization of pattern sets and the encoding of future subblocks by copying from a frame memory containing the recently decoded subblocks. Computer simulations on still images have demonstrated that this technique could very efficiently compress sampled images, especially at rates under 1.0 bit per pixel.>

A new vector quantization scheme in discrete cosine transform domain, named DCT-VQ and its application to color image coding are described. In this scheme, DCT-domain is partitioned into vectors which are normalized and vector-quantized using universal vector quantizers designed with multidimensional Laplacian distribution. Adaptive coding scheme is also introduced to obtain better reconstruction of images. The color image coder employs the above scheme and encodes separately three components converted from R,G,B signals. The simulations have shown that adaptive DCT-VQ exhibits better performance than a conventional adaptive cosine transform coding with scalar quantization. The decomposition of DCT-block into vectors results in much less complex coder than a vector quantizer in original space domain.

A new nonlinear digital filter which separates nonstationary waves such as spikes from stationary background waves of the EEG is proposed. This filter is composed of a prediction filter and a simple nonlinear function, and separates the nonstationary waves by processing the prediction error with the nonlinear function. This filter aims at real-time analysis of EEG, hence the algorithm of this filter is deliberately kept very simple, and the parameters in it are set adaptively. Some examples of separating spikes from epileptic EEG data are shown.

This paper describes a knowledge based isolated Japanese word recognition algorithm. The program is written with Prolog/KR [4] and has two basic inference processes, i.e., a Bottom-up search and a Top-down search. In the Bottom-up process, a segmentation and a vowel decision are performed and some target word patterns are generated. The Top-down process includes a consonant decision using a score of each candidate word calculated based on the Fuzzy Set Theory [1]. In the vowel inference, a template matching is applied mainly. In the segmentation, heuristic rules based on the spectrum transition and the wave form are used. But in the consonant inference, each rule has a hierarchy structure and it is defined automatically in the form of the multi-valued threshold function from learning data. This system can treat an obscure information about the consonant classification and to select the most effective decision rule in order to simplify the understanding process. The truth rate of consonant recognition is better than using statistical method.

An efficient discrete cosine transform image coding system using the gain/shape vector quantizers (DCT-G/S VQ) is presented. In the coding system, AC transform coefficients in a subblock are partitoned into several bands according to the Schaming's method, and the normalized AC transform coefficients of each band are quantized with the gain/shape vector quantizer designed on a spherically symmetric probability model. In addition, an adaptive DCT-G/S VQ (A-DCT-G/S VQ) is presented by incorporating a modification of the recursive quantization technique in the DCT-G/S VQ. The coding systems are simulated on color images, and their performance is compared to that of previously reported discrete cosine transform coding systems using the Max-type scalor quantizers.

A new transmission scheme for channels with intersymbol interference named "matched-transmission (MT) technique" is proposed. An implementation of the MT system is presented and its relations with conventional techniques are described. It is shown that the partial response scheme is a special case of our new technique. The performance of the MT technique is discussed from the view point of the output signal-to-noise ratio and the system is shown to attain asymptotically the bound performance predicted by the information theory.