Sit-to-stand and stand-to-sit motions are important in daily activities. However, elderly individuals often find these motions difficult to perform with declining lower limb strength, which causes a considerable reduction to their quality of life. In this study, a sensing method for controlling robotic assistive devices was proposed. This method utilizes electromyographic measurements and a deep neural network to predict motion initiation, and it estimates the timing of triggering assistive devices. Experimental results indicate that four muscle synergy patterns are required to represent the sit-to-stand and stand-to-sit motions together, with two of them being shared between both movements. Subsequently, a long short-term memory network was designed to forecast these two motions, and the result indicates that the prediction accuracy reached 92.95% ± 0.83% with forecasting time of 300 ms.

During video lectures, learners may have attentional modes such as “follow a lecturer's guide (speech and pointers),” “look ahead of spoken parts and actively check slide content,” and “roughly browse a slide.” The dynamic change of these modes is useful to characterize personal and/or temporary viewing styles. This paper presents a method to analyze video viewing styles through gaze behavioral data by using a probabilistic generative model with a latent mode variable. In our experiments, we show that the model can infer viewers' temporal mode patterns and successfully characterize task-dependent viewing situations.

While many spoken dialog systems are recently developed, users need to summarize and convey what they want the system to do clearly. However, in a human dialog, a speaker often summarize what to say incrementally, provided that there is a good listener who responds to the speaker's utterances at appropriate timing. We consider that generating backchannel responses, where appropriate, overlapped with the user's utterances is crucial for an artificial listener system that can promote user's utterances since such overlaps are the norm in human dialogs. Toward the goal to realize such a listener system, in this paper, we propose a voting-based algorithm of predicting the end of utterances early (i.e., before the utterances end) using audio-visual information. In the evaluation, we demonstrate the effectiveness of using audio-visual information and the applicability of the voting-based prediction algorithm with some early results.

Generation of natural human motion is one of key techniques for multimodal dialogue systems with a human-like avatar. In particular, natural and expressive lip motion synthesis is necessary to make conversation between a user and an avatar richer. However, such expressive lip motion is often difficult to be generated automatically because it can be changed depending on phonemic context and prosody. To address this difficulty, we introduce a novel motion generation method on the basis of the modulation of a set of dynamic models learned from neutral motion data. As a suitable model for lip motion generation, we adopt a hybrid dynamical system, which consists of linear dynamical systems for each motion unit and a symbolic automaton for switching between these units. We show that, from the viewpoint of control theory, it is possible to modulate linear dynamical systems for various types of motion. Early results demonstrate the applicability of the proposed method using lip motion synthesis for simple phoneme sequences.

When we choose items among alternatives, we sometimes face a problem of mismatch between what we actually want and selected items. Therefore, if an interactive system can probe our interests from several modalities (e.g, eye movements and speech recognition) and decrease these mismatch, the system can be helpful for decision making with a satisfaction. In order to build such interactive decision support systems, the systems need to estimate both users' interests (selection criteria for that decision) and users' knowledge about the content domain. Here, not only users' knowledge but also users' selection criteria can be changed; for example, users' selection criteria converge as a reaction to system's recommendation. Therefore, the system needs to understand the dynamics of users' selection criteria in order to choose appropriate actions. What makes more difficult is that the dynamics of users' internal states themselves can change depend on a phase of decision making. Therefore, we need to address (a) how to represent users' internal states, (b) how to estimate and trace temporal changes of users' internal state, and (c) how to trace users' decision phase so that system can decide actions for decision assistance. In order to tackle these problems, we propose a novel representation of users' internal state, which consists of selection criteria and structures of users' knowledge about the content domain and a method to estimate these selection criteria from gaze behavior. In addition, we consider the multiscale dynamics of users' internal states so that the system can trace users' decision phase as temporal changes of dynamics of users' internal states.