Humans routinely pay attention to important emotion information from visual and audio modalities without considering multimodal alignment issues, and recognize emotions by integrating important multimodal information at a certain interval. In this paper, we propose a multiple attention fusion network (MAFN) with the goal of improving emotion recognition performance by modeling human emotion recognition mechanisms. MAFN consists of two types of attention mechanisms: the intra-modality attention mechanism is applied to dynamically extract representative emotion features from a single modal frame sequences; the inter-modality attention mechanism is applied to automatically highlight specific modal features based on their importance. In addition, we define a multimodal domain adaptation method to have a positive effect on capturing interactions between modalities. MAFN achieved 58.65% recognition accuracy with the AFEW testing set, which is a significant improvement compared with the baseline of 41.07%.

Empathic conversations have increasingly been important for dialogue systems to improve the users' experience, and increase their engagement with the system, which is difficult for many existing monotonous systems. Existing empathic dialogue systems are designed for limited domain dialogues. They respond fixed phrases toward observed user emotions. In open domain conversations, however, generating empathic responses for a wide variety of topics is required. In this paper, we draw on psychological studies about empathy, and propose an empathic dialogue system in open domain conversations. The proposed system generates empathic utterances based on observed emotions in user utterances, thus is able to build empathy with users. Our experiments have proven that users were able to feel more empathy from the proposed system, especially when their emotions were explicitly expressed in their utterances.

In this paper, we propose a method to reduce the number of parameters based on a pre-trained deep convolutional neural network (DNN) for a facial expression recognition (FER) task. In order to maintain the high level of accuracy of the pre-trained network and save computational resources through reducing the number of parameters, we first build and train a high accuracy DNN model and extract the layers which contain the most representative facial expression features; then, we connect these layers with lightweight depthwise separable convolutions and global max pooling layers as the re-training network. We train and compare the re-training network with different DNN architectures on two FER datasets (FER2013, AffectNet). The results demonstrate that our method retains its pre-trained high accuracy with fewer model parameters than current state-of-the-art DNN methods.

In this paper, we propose an autonomous and adaptive recommendation system that relies on the user's mood and implicit feedback to recommend songs without any prior knowledge about the user preferences. Our method builds autonomously a latent factor model from the available online data of many users (generic song map per mood) based on the associations extracted between user, song, user mood and song emotion. It uses a combination of the Reinforcement Learning (RL) framework and Page-Hinkley (PH) test to personalize the general song map for each mood according to user implicit reward. We conduct a series of tests using LiveJournal two-million (LJ2M) dataset to show the effect of mood in music recommendation and how the proposed solution can improve the performance of music recommendation over time compared to other conventional solutions in terms of hit rate and Fl score.

In this paper, we design a crowd-powered system to efficiently collect data for training dialogue systems. Conventional systems assign dialogue roles to a pair of crowd workers, and record their interaction on an online chat. In this framework, the pair is required to work simultaneously, and one worker must wait for the other when he/she is writing a message, which decreases work efficiency. Our proposed system allows multiple workers to create dialogues in an asynchronous manner, which relieves workers from time restrictions. We have conducted an experiment using our system on a crowdsourcing platform to evaluate the efficiency and the quality of dialogue collection. Results show that our system can reduce the necessary time to input a message by 68% while maintaining quality.

Many older adults are interested in smartphones but encounter difficulties in self-instruction and need support, especially text input. Voice input is a useful option for text input, but also presents some difficulties for older adults.In this paper, we propose a tutoring system for voice input that detects input stumbles using a statistical approach and provides instructions to overcome them. We construct the tutoring system based on the data from a user study with novice older adults. In an evaluation experiment, the number of input stumble and the sentence completion time of the participants using the tutoring system were significantly smaller than those without it. The results showed that the tutoring system resulted in the improvement of the efficiency of voice input for novice older adults.

We built a personalized system that can predict the upcoming user mood even in days without text-type tweets. We, first, study the correlation of three types of features (cyber, social and physical) with a user mood. Then, we use these features to train a predictive system. The results suggest a statistically significant correlation between user mood and his cyber, social and physical activities distributed among different OSNs, which lead to a low RMSE in our predictive system compared to linguistic-based systems when non-text content is available.

The increasing popularity of mobile crowdsourcing platforms has enabled crowd workers to accept jobs wherever/whenever they are, and also provides opportunity for task requesters to order time/location specific tasks to workers. Since workers on mobile platforms are working on the go, the situation of the workers is expected to influence their performance. However, the effects of mobile worker situations to task performance is an uninvestigated area. In this paper, our research question is, "do worker situations affect task completion, price and quality on mobile crowdsourcing platforms?" We draw on economics and psychology research to examine whether worker situations such as busyness, fatigue and presence of companions affect their performance. Our three-week between-subjects field experiment revealed that worker busyness caused 30.1% relative decrease of task completion rate. Mean accepted task price increased by 7.6% when workers are with companions. Worker fatigue caused 37.4% relative decrease of task quality.

Activity recognition from smart devices and wearable sensors is an active area of research due to the widespread adoption of smart devices and the benefits it provide for supporting people in their daily lives. Many of the available datasets for fine-grained primitive activity recognition focus on locomotion or sports activities with less emphasis on real-world day-to-day behavior. This paper presents a new dataset for activity recognition in a realistic unmodified kitchen environment. Data was collected using only smart-watches from 10 lay participants while they prepared food in an unmodified rented kitchen. The paper also providing baseline performance measures for different classifiers on this dataset. Moreover, a deep feature learning system and more traditional statistical features based approaches are compared. This analysis shows that - for all evaluation criteria - data-driven feature learning allows the classifier to achieve best performance compared with hand-crafted features.

This paper examines the effects of objective feedback of facial expression recognition on the operators of a video support chat system. We have built a facial expression feedback (FEF) system, which recognizes the users' facial expressions, and displays a visual summary of the recognition results to the user, in order to raise self-awareness of his/her expressions during video communication. We evaluated our system in a supposed simulation of a call center, in which operators provide assistance to customers about how to use their smartphones. Experimental results have verified that our approach is useful to help operators improve their self-awareness of facial expression in the video support chat.

This research proposes a novel method to extract image regions of products from an advertisement video, by analyzing features which are completely independent from the target object. Namely, we focus on how each product is emphasized in the video production, and propose the utilization of low-level visual features which leverage the technical know-how of video producers. By using such features, our proposal can achieve highly accurate detection of temporal and spatial locations of the advertised products, regardless of the product domain. Evaluation of the proposed method has been conducted with the actual advertisement video, in which accuracy of 79.4% in F-measure has been achieved.

This research proposes a novel method to present "thumbnails" of episodes of digitized comics, in order to improve the efficiency of comic search. Comic episode thumbnails are generated based on image analysis technologies developed especially for comic images. Namely, the following procedures are developed for our system: automatic comic frame segmentation, text balloon extraction, and a linear regression based model to calculate the importance score of each extracted frame. The system then selects frames from each episode with high importance score, and aligns the selected frames to create the episode thumbnail, which is presented to the system user as a compact preview of the episode. User experiments conducted with actual Japanese comic images prove that the proposed method significantly decreases the time necessary to search for specific episodes from a large scaled comic data collection.

Utilization of web images with social tags as training data has been a major trend for the development of automatic image tagging/classification systems. While the amount of information available on web sites such as Flickr is abundant, the majority of information obtained from such sites is in English, the dominant language on the web. This linguistic unbalance is expected to affect auto-tagging results in a negative way for non-English users, who demand image tags in their native languages. The objective of this research is to develop an image auto-tagging system which can generate tags in languages other than English. This paper examines the effect of linguistic unbalance in training data to construct auto-tagging systems, which aim to generate tags in minor languages. Furthermore, we propose methods which utilize an auto-tagging model generated on English training data, and convert the auto-tagging results to the target language. Subjective evaluations show that the proposed method is capable of generating auto-tagging results with better quality than conventional approaches.

In this research, we propose a method to automatically generate a landmark identification system for geo-tagged photographs, based on analysis of various data collected from the Web. The method first conducts Web analysis based on three major procedures: (1) Automatic extraction of points-of-interest (POIs) based on geographical clustering of geo-tagged images, (2) Retrieval of landmark candidates for each extracted POI from search results of map search API, and (3) Collection and feature extraction of Web images of the landmark candidates. The system then identifies the landmark which appears in the query geo-tagged photograph, by comparing the location and content-based features of the query with the information accumulated by the previous procedures. Experimental results show that the proposed method is capable to construct a highly accurate landmark identification system by leveraging Web information.

Automatic beat estimation is an essential technology not only for fundamental analysis of music, but also for the development of advanced music applications such as DJ mixing. In this paper, we propose an autocorrelation based beat estimation method, which, unlike existing methods, is capable of accurately detecting the position of beats, even from songs with drastic change of tempo. Our proposal consists of two major steps. First, the approximate positions of the beats are pre-estimated by a long-sized analysis window. Next, the preestimated beats are verified by using a short analysis window. These steps are repetitively executed until the beat estimation results converge. Experiments conducted with audio signals and songs with artificially applied tempo changes have proved that our proposal can detect beats with high accuracy.

The demand of browsing information from general Web pages using a mobile phone is increasing. However, since the majority of Web pages on the Internet are optimized for browsing from PCs, it is difficult for mobile phone users to obtain sufficient information from the Web. Therefore, a method to reconstruct PC-optimized Web pages for mobile phone users is essential. An example approach is to segment the Web page based on its structure, and utilize the hierarchy of the content element to regenerate a page suitable for mobile phone browsing. In our previous work, we have examined a robust automatic Web page segmentation scheme which uses the distance between content elements based on the relative HTML tag hierarchy, i.e., the number and depth of HTML tags in Web pages. However, this scheme has a problem that the content-distance based on the order of HTML tags does not always correspond to the intuitional distance between content elements on the actual layout of a Web page. In this paper, we propose a hybrid segmentation method which segments Web pages based on both the content-distance calculated by the previous scheme, and a novel approach which utilizes Web page layout information. Experiments conducted to evaluate the accuracy of Web page segmentation results prove that the proposed method can segment Web pages more accurately than conventional methods. Furthermore, implementation and evaluation of our system on the mobile phone prove that our method can realize superior usability compared to commercial Web browsers.

This paper proposes a feature space modification method for feature extraction of music, which is effective for the development of a content-based music information retrieval (MIR) system based on user preferences. The proposed method conducts clustering of all songs in the music collection, and utilizes the resulting cluster IDs as training data for feature space modification, and is capable to automatically generate a feature space which is suitable to the content of any music collection. Experiment results prove that the proposed method improves accuracy of user preference based MIR

This paper describes our new algorithm for shot boundary detection and its evaluation. We adopt a 2-stage data fusion approach with SVM technique to decide whether a boundary exists or not within a given video sequence. This approach is useful to avoid huge feature space problems, even when we adopt many promising features extracted from a video sequence. We also introduce a novel feature to improve detection. The feature consists of two kinds of values extracted from a local frame sequence. One is the image difference between the target frame and that synthesized from the neighbors. The other is the difference between neighbors. This feature can be extracted quickly with a least-square technique. Evaluation of our algorithm is conducted with the TRECVID evaluation framework. Our system obtained a high performance at a shot boundary detection task in TRECVID2005

In this paper, camera motion detection methods using a background image generated by video mosaicing based on the correlation between feature points on a frame pair are described. In this method, a telop (video caption) removal method, iterative foreground and background image separation method and appropriate frame pair selection from consecutive frames are introduced to generate background images accurately. Parameters indicating the location of each frame on the background image are retrieved and used to detect the camera motion. Except for the simple threshold-based method, a method using hidden Markov models (HMMs) is introduced to detect variable length camera motion based on the maximum likelihood criterion. The effectiveness of the proposed method is evaluated by using a TRECVID 2005 low-level feature extraction task

This paper proposes an adaptive system for video news story tracking based on the earth mover's distance (EMD). When an interesting story appears in the news, it is flagged manually as a topic for tracking. Our system then tracks the events as they unfold over time and present accumulated results to the user for feedback. This feedback is used to adapt the topic model to changes in the tracked story. EMD provides the system with a robust way of performing many-to-many matching of news stories independent of the temporal order of their contents. This is particularly suitable in the news genre as stories often are subjected to video editing between shows. Experiments have been run with a range of topics and show promising results

Numerous efforts on content-based music information retrieval have been presented in recent years. However, the object of such existing research is to retrieve a specific song from a large music database. In this research, we propose a music retrieval method which retrieves songs based on the user's musical preferences. This enables users to discover new songs which they are expected to like. Since music preferences are expected to be highly ambiguous, we propose the implementation of relevance feedback methods to improve the performance of our music information retrieval method. In order to reduce the burden of users to input learning data to the system, we also propose a method to generate user profiles based on genre preferences, and refinement of such profiles based on relevance feedback. Evaluation experiments are conducted based on a corpus of music data with user ratings. Results of these experiments prove the effectiveness of our method.

No abstract available.

No abstract available.

Document filtering is a task to retrieve documents relevant to a user's profile from a flow of documents. Generally, filtering systems calculate the similarity between the profile and each incoming document, and retrieve documents with similarity higher than a threshold. However, many systems set a relatively high threshold to reduce retrieval of non-relevant documents, which results in the ignorance of many relevant documents. In this paper, we propose the use of a non-relevant information profile to reduce the mistaken retrieval of non-relevant documents. Results from experiments show that this filter has successfully rejected a sufficient number of non-relevant documents, resulting in an improvement of filtering performance.

This paper describes controlling robot's gaze which has relation to smoothness of turn-taking in communication. We considered the role of gaze in dialogues between human beings and examined it by simulation and our humanoid. Also we analyzed the features of gaze movement in dialogues by plural persons and confirmed that controlling gaze is efficient in confirmation of communication channel by implementing it on the humanoid.