Humans recognize perceived continuous information by dividing it into significant segments such as words and unit motions. We believe that such unsupervised segmentation is also an important ability that robots need to learn topics such as language and motions. Hence, in this paper, we propose a method for dividing continuous time-series data into segments in an unsupervised manner. To this end, we proposed a method based on a hidden semi-Markov model with Gaussian process (GP-HSMM). If Gaussian processes, which are nonparametric models, are used, unit motion patterns can be extracted from complicated continuous motion. However, this approach requires the number of classes of segments in the time-series data in advance. To overcome this problem, in this paper, we extend GP-HSMM to a nonparametric Bayesian model by introducing a hierarchical Dirichlet process (HDP) and propose the hierarchical Dirichlet processes-Gaussian process-hidden semi-Markov model (HDP-GP-HSMM). In the nonparametric Bayesian model, an infinite number of classes is assumed and it becomes difficult to estimate the parameters naively. Instead, the parameters of the proposed HDP-GP-HSMM are estimated by applying slice sampling. In the experiments, we use various synthetic and motion-capture data to show that our proposed model can estimate a more correct number of classes and achieve more accurate segmentation than baseline methods.

Humans interact with one another daily and learn from their experiences via observation and interaction. To create robots that can coexist with humans, it is important that they learn how to appropriately interact in the human community. In this paper, we propose a novel model, the coupled Gaussian process hidden semi-Markov model (GP-HSMM), which enables robots to learn rules of interaction between two persons by observing them in an unsupervised manner. The continuous motions of the persons are segmented into discrete actions based on GP-HSMM, and the relationships between the actions are extracted. Moreover, all corresponding actions are not simultaneously conducted by two persons during actual interaction. Thus, the coupled GP-HSMM accounts for these lags. We conduct experiments using the motion data of interaction games. Experimental results showed that the coupled GP-HSMM can estimate actions, lags between them and their relationships.

We propose a method for a robot to form various concepts. The robot uses its embodiment to obtain visual, auditory, and haptic information by grasping, shaking, and observing objects. At the same time, a user teaches the robot object features through speech. From these kinds of information, the robot can form object concepts. The information obtained by the robot is converted into Bag-of-Words representations, which are classified into categories. The proposed method is based on a stochastic model, and objects can be classified by estimating their parameters. We introduce the Chinese restaurant process into the multimodal hierarchical Dirichlet process. This model is an infinite mixture of models and enables the robot to form various concepts such as object type, color, and so on. Therefore, the robot can form not only object concepts but also various concepts that are not represented by object concepts (e.g., color). Also, because the proposed method is based on a stochastic model, it makes it possible for the robot to estimate category and unobserved information of unseen objects. We implement the proposed method on a robot and show that it can form various concepts and perform various estimations for unseen objects.

A theoretical analysis on interframe predictive coding is presented in which special attention is paid on two issues that are practically important. First, the displacements between the target and reference images are usually given with limited accuracy due to quantization. Second, when the displacements are provided in subpixel accuracies, interpolation between the pixels is necessary to produce an estimation image. Our formulation takes an exhaustive approach in which we faithfully follow the process of subpixel interpolations with limited displacement accuracies. Our analysis can be used to determine how accurate the displacement vectors should be and which subpixel interpolation method should be used to optimize the rate-distortion performance.

A novel block-based bag of words (BBoW)???method is proposed for robust face recognition. In our approach, a face image is partitioned into multiple blocks, dense SIFT features are then calculated and vector quantized into different codewords on each block respectively. Finally, histograms of codeword distribution on each local block are concatenated to represent the face image. Experimental results on AR database show that only using one neutral expression frame per person for training, our method can obtain excellent face recognition results on face images with extreme expressions, variant illumination, and partial occlusions. Our method also achieves an average recognition rate of 100% on XM2VTS database.

Object tracking is often disturbed by visual occlusion. To handle this problem, we have previously proposed the tracking method by the particle filter, which switches tracking targets autonomously. This method enables the tracker to track the occluded target indirectly by switching its target to the occluder effectively. However, the color-based target model used in this method often causes inaccurate tracking because the model with only one color distribution is not necessarily sufficient. In this paper, we propose a method for online object modeling using a set of color distributions and a set of SIFT features. Since the proposed model has more color information and local texture information, it enables the tracker to recognize the target more robustly. Furthermore, this model can be created dynamically and updated in an online fashion using the graph cuts technique during tracking. Consequently it can be applied to the previously proposed tracking method with autonomous switching of targets. Experimental results show the effectiveness of the proposed method.

Active Shape Model (ASM) is a powerful statistical tool for image interpretation, especially in face alignment. In the standard ASM, local appearances are described by intensity profiles, and the model parameter estimation is based on the assumption that the profiles follow a Gaussian distribution. It suffers from variations of poses, illumination and expressions. In this paper, an improved ASM framework, GentleBoost based SIFT-ASM is proposed. Local appearances of landmarks are originally represented by SIFT (Scale-Invariant Feature Transform) descriptors, which are gradient orientation histograms based representations of image neighborhood. They can provide more robust and accurate guidance for search than grey-level profiles. Moreover, GentleBoost classifiers are applied to model and search the SIFT features instead of the unnecessary assumption of Gaussian distribution. Experimental results show that SIFT-ASM significantly outperforms the original ASM in aligning and localizing facial features.

A novel framework of facial appearance and shape information extraction for facial expression recognition is proposed. For appearance extraction, a facial-component-based bag of words method is presented. We segment face images into 4 component regions, and sub-divide them into 4×4 sub-regions. Dense SIFT (scale-invariant feature transform) features are calculated over the sub-regions and vector quantized into 4×4 sets of codeword distributions. For shape extraction, PHOG (pyramid histogram of orientated gradient) descriptors are computed on the 4 facial component regions to obtain the spatial distribution of edges. Our framework provides holistic characteristics for the local texture and shape features by enhancing the structure-based spatial information, and makes the local descriptors be possible to be used in facial expression recognition for the first time. The recognition rate achieved by the fusion of appearance and shape features at decision level using the Cohn-Kanade database is 96.33%, which outperforms the state of the arts.

Human-friendly robots will work mainly in usual living spaces for humans, such as offices, kitchens and living rooms. In these environments, there are many objects which can cause the visual occlusion. So it will often occur that the robot cannot observe some object by the occlusion while the user can observe it, or vice versa, according to their positions. In these situations, the robot is required to recognize this "invisibility," that is, to interact with the user considering the possibility that there is some object which the user can observe but the robot cannot or vice versa. In this paper, we propose a method for estimation of robot's and user's views and blind regions from images obtained by a stereo camera. Our method is one of the fundamental techniques to realize the robot which can handle the occlusion problems mentioned above. Experimental results in real occlusion environments show the effectiveness of our method.

Tracking humans' position is a useful skill for the coming generation of mobile robot. It is a challenging problem of planning and control in dynamic environment. We propose the omni-directional estimation method of speaker's position using the combination of audio and visual information. Estimation of the position of the sound is carried out to calculate the difference of arrival time from sound source to multi-channel microphones. The robust human template matching on the omni-directional image is employed to combine the result of sound source estimation to realize a highly accurate estimation of speaker's location. In our experiments, the systems were implemented and tested on an omni-directional robot at our laboratory. The results show that we are able to reliably detect and track moving objects in natural environment.

A skin-color extraction algorithm is proposed to detect human faces in color images with varying illumination condition and presence of complex background. The approach is based on both a Gaussian mixture model of human skin-color distribution and image segmentation using all automatic and adaptive multi-thresholding technique. Detected regions are then refined by morphological operations. Experimental results on images presenting a wide range of variations in lighting condition, face orientation, scale, pose, facial expression and background, demonstrate the efficiency of our skin-segmentation algorithm. Using additional information about facial features, our method becomes an efficient step in localizing candidate faces for a face detection system.

This paper proposes a speech recognition and an enhancement system for noisy car environments based on a microphone array. In the system, multiple microphones axe arranged in 2-dimensional space, surrounding the interior of a car, and the speaker's location is first estimated by our proposed HE (harmonic enhanced) 2-D MUSIC (MUltiple SIgnal Classification). Then, 2-D delay and sum (DS) is applied to enhance the target speech. Such pre-processing makes robust speech recognition in noisy car environments possible. In the proposed system, not only a driver, but also a fellow passenger can control car electronics by their voices no matter where they are. This is an advantage of the system as well. The results of the simulation and the preliminary experiment in a real car environment are presented to confirm the validity of our proposed system.

In this paper, generalized linear phase lapped orthogonal transforms with unequal length basis functions (GULLOT) are considered. The length of each basis of the proposed GULLOT can be different from each other, while all the bases of the conventional GenLOT are of equal length. In order to apply the GULLOT to subband image coding, we also investigate the size-limited structure to process the finite length signal which is important in practice.

In this paper, an efficient analysis method for global head motion, and a synthesis method of 3-D face scenes using model-based image coding techniques on the Internet browser are presented. By applying the techniques of model-based image coding in facial image sequence, human face or anthropomorphic agents can be synthesized with real facial motion on the internet browser using VRML and/or JAVA. While typical model-based image coding produces synthetic face scenes in 2-D space at the receiver terminals, the proposed scheme can produce synthetic face scenes in 3-D space interactively supporting some visual standard. This enables users to synthesize 3-D face scenes at any terminals. In this sense, it improves the portability of synthesis in model-based image coding. To speed up the computation in the analysis process, a new method that generates referential frame dynamically is proposed. To produce a 3-D face model which is suitable for the purpose of application, an idea of automatic face modeling using different 3-D face models is presented. Experimental results show the usefulness of applying the techniques of model-based image coding to the analysis of facial motion and synthesis of 3-D face scene on the internet browser.

Multi-screen projection systems are widely used in the field of virtual reality. Within the system bounded by screens, users can view the panoramic stereoscopic images projected onto the screens. In such an immersive environment, however, just one viewer can feel the natural depth sensation. The authors propose a novel approach to realize a multi-user system, in which several users can feel the adequate depth sensation simultaneously. Experimental results show the potential applicability of the proposed method to the next-generation virtual environment.

The model-based coding is a new semantic-based coding which utilizes a common knowledge database being both at a transmitter and its receiver.We have proposed the interactive model-based coding for the face metaphor user interface applications such as multimedia e-mail, WWW and agents.Three points are addressed in this paper.(l)We have constructed a framework of interactive model-based coding tool in network communications environment.(2)We show that motion parameter description for face expressions should have both CGbased synthesis and natural image-based synthesis.The corresponding descriptions by relative and absolute way are necessary for sufficient face metaphor expressions.(3)We have developed the encoder and decoder.The motion parameters are transmitted from experimental server to a receiver.The decoded motion picture can be demonstrated both by a videotape and by a PC on-line browsing through the internet.

This paper presents a new coding method representing moving images with planes. For the purpose of handling 3-D motion and depth information, we extend the layered representation and propose a new planar representation, called "setting representation." 3-D descriptions are efficient for handling images, compared with 2-D descriptions. The estimation method of 3-D structured information and the integration method of 3-D motion are also addressed. It is not easy, however, to extract consistent 3-D information under various conditions. On the other hand, even 2-D information is enough for describing certain parts of the image. We combine the layered and the setting representations and propose a new video coding method based on 2-D/3-D structured information.

A new hybrid coding method for transmitting videoconferencing images at a bit rate 384 kbits/s is proposed. Considering the characteristics of motion-compensated interframe prediction errors for typical videoconferencing scenes, a filter is introduced to separate pulsive components on which conventional discrete cosine transform (DCT) coding method does not work well. These separated pulsive components are coded by using scalar quantization (SQ). The remainder are DCT coded. For DCT coefficients, an adaptive coding method based on the classification of DCT coefficients is applied in order to improve the coding performance. Since the proposed method employs both DCT coding and SQ of prediction errors, it is named "DCT/SQ coding method." Experimental results show that the DCT/SQ coding method is effective to reduce so-called mosquito effects, and thus it can improve the quality of decoded images.