We present a novel framework to automatically generate natural gesture motions accompanying speech from audio utterances. Based on a Bi-Directional LSTM Network, our deep network learns speech-gesture relationships with both backward and forward consistencies over a long period of time. Our network regresses a full 3D skeletal pose of a human from perceptual features extracted from the input audio in each time step. Then, we apply combined temporal filters to smooth out the generated pose sequences. We utilize a speech-gesture dataset recorded with a headset and marker-based motion capture to train our network. We validated our approach with a subjective evaluation and compared it against "original" human gestures and "mismatched" human gestures taken from a different utterance. The evaluation result shows that our generated gestures are significantly better than the "mismatched" gestures with respect to time consistency. The generated gesture also shows marginally significant improvement in terms of semantic consistency when compared to "mismatched" gestures.

In this research, we take a first step in generating motion data for gestures directly from speech features. Such a method can make creating gesture animations for Embodied Conversational Agents much easier. We implemented a model using Bi-Directional LSTM taking phonemic features from speech audio data as input to output time sequence data of rotations of bone joints. We assessed the validity of the predicted gesture motion data by evaluating the final loss value of the network, and evaluating the impressions of the predicted gesture by comparing it with the actual motion data that accompanied the audio data used for input and motion data that accompanied a different audio data. The results showed that the accuracy of the prediction for the LSTM model was better than a simple RNN model. In contrast, the impressions evaluation of the predicted gesture was rated lower than the original and mismatched gestures, although individually some predicted gestures were rated the same degree as the mismatched gestures.

We present a method that estimates graspability measures on a single depth map for grasping objects randomly placed in a bin. Our method represents a gripper model by using two mask images, one describing a contact region that should be filled by a target object for stable grasping, and the other describing a collision region that should not be filled by other objects to avoid collisions during grasping. The graspability measure is computed by convolving the mask images with binarized depth maps, which are thresholded differently in each region according to the minimum height of the 3D points in the region and the length of the gripper. Our method does not assume any 3-D model of objects, thus applicable to general objects. Our representation of the gripper model using the two mask images is also applicable to general grippers, such as multi-finger and vacuum grippers. We apply our method to bin picking of piled objects using a robot arm and demonstrate fast pick-and-place operations for various industrial objects.

In this paper we present a robust alignment algorithm for correcting the effects of out-of-plane rotation to be used for automatic alignment of the Computed Tomography (CT) volumes and the generally low quality fluoroscopic images for radiotherapy applications. Analyzing not only in-plane but also out-of-plane rotation effects on the Dignitary Reconstructed Radiograph (DRR) images, we develop simple alignment algorithm that extracts a set of implicit features from DRR. Using these SIFT-based features, we align DRRs with the fluoroscopic images of the patient and evaluate the alignment accuracy. We compare our approach with traditional techniques based on gradient-based operators and show that our algorithm performs faster while in most cases delivering higher accuracy.

This paper presents a novel background subtraction method for detecting foreground objects in dynamic scenes involving swaying trees and fluttering flags. Most methods proposed so far adjust the permissible range of the background image variations according to the training samples of background images. Thus, the detection sensitivity decreases at those pixels having wide permissible ranges. If we can narrow the ranges by analyzing input images, the detection sensitivity can be improved. For this narrowing, we employ the property that image variations at neighboring image blocks have strong correlation, also known as "cooccurrence". This approach is essentially different from chronological background image updating or morphological postprocessing. Experimental results for real images demonstrate the effectiveness of our method.

This paper describes video coding and segmentation techniques that can be used to achieve significant increase in storage capacity. Specifically, we examine the possibility to use object- based coding for efficient long-term archiving of surveillance video. We consider surveillance systems with many camera sources in which we are required to store several months of video data for each source, thus storage capacity is a major concern. The paper considers several automatic segmentation algorithms. With each algorithm, we analyze the shape coding overhead and implication on overall storage requirements, as well as the effect each algorithm has on the reconstructed quality of frames. Additionally, this paper reviews techniques to dynamically control the temporal rate of objects in the scene and perform bit allocation. Experimental results show that up to 90% savings in storage can be achieved with the proposed method compared to frame-based video coding techniques. The cost for this savings is that the accuracy of the background is compromised; however, we feel that this is satisfactory for the application under consideration.

In this paper, we propose a practical object recognition system which consists of two functional modules. The first is object extraction module using a range image, and the second is a precise position measurement module using a grayscale intensity image. Both high-reliability and high-accuracy can be achieved by effective image integration. We also propose an idea of stereo vision with random-dot pattern projection as an effective way to obtain a range image.

Background subtraction is a useful and effective method for detecting moving objects in video images. Since this method assumes that image variations are caused only by moving objects (i.e., the background scene is assumed to be stationary), however, its applicability is limited. In this paper, we propose a background subtraction method that robustly handles various changes in the background. The method learns the chronological changes in the observed scene's background in terms of distributions of image vectors. The method operates the subtraction by evaluating the Mahalanobis distances between the averages of such image vectors and newly observed image vectors. The method we propose herein expresses actual changes in the background using a multi-dimensional image vector space. This enables the method to detect objects with the correct sensitivity. We also introduce an eigenspace to reduce the computational cost. We describe herein how approximate Mahalanobis distances are obtained in this eigenspace. In our experiments, we confirmed the proposed method's effectiveness for real world scenes.

This paper presents our real-time vision system for robots. The system employs a new internal image representation, in which the scene is encoded into three intensity level images. This representation is generated by Laplacian-Gaussian filtering followed by dual-thresholding. We refer to this image as three-level broad-edge representation. It reduces the computational cost of the normalized cross correlation. Thus the system performs the searching visual task as well as the object tracking task, which is not reliably achieved by the mean absolute error based correlators. Our prototype system achieves 32/spl times/32 pixel normalized cross correlation based matching from the 128/spl times/128 pixel search area in 2 ms under 9 MHz pixel clock image processor. This speed is fast enough for searching and tracking a single object at video frame rate.

Genetic labeling is a new labeling algorithm using Genetic Algorithm (GA). Although several applications of GA for low-level image processing such as line detection have been studied, they still require much computing time. We apply GA to the labeling for scene interpretation. The chromosome coding method we proposed is such that each bit represents the existence of an object. Genetic operation enables efficient labeling based on the building block hypothesis. We have developed a vision system for depalletizing robot using this technique. Object candidates are properly labeled, and the position of cartons is recognized. Through real image experiments, we estimated that genetic labeling is about 100 times faster than an improved enumerating method. Also we have proven that the reliability and speed of this system is practical.>

Proposes a statistical model for the perception of motion transparency and coherence which is given by a two stage process for the extraction of the optical flow and the velocity histogram. In the first stage, a sequence of images is divided into regions; in each region, an optical flow or a normal component of the optical flow is computed. In the second stage, in each of the regions mentioned above, a local velocity histogram is computed and the local histogram is summarised into a global histogram. As for experiment, the authors describe the perception of motion coherence and transparency for synthetic images, such as line patterns and filled regions. They then extend the model for real scenes and apply it to the analysis of the motion of pedestrians.>