Sit-to-stand and stand-to-sit motions are important in daily activities. However, elderly individuals often find these motions difficult to perform with declining lower limb strength, which causes a considerable reduction to their quality of life. In this study, a sensing method for controlling robotic assistive devices was proposed. This method utilizes electromyographic measurements and a deep neural network to predict motion initiation, and it estimates the timing of triggering assistive devices. Experimental results indicate that four muscle synergy patterns are required to represent the sit-to-stand and stand-to-sit motions together, with two of them being shared between both movements. Subsequently, a long short-term memory network was designed to forecast these two motions, and the result indicates that the prediction accuracy reached 92.95% ± 0.83% with forecasting time of 300 ms.

Transmission latency presents a significant challenge when operating remote equipment, such as a robotic arm. To address this, we developed a platform and conducted experiments to reduce transmission latency to near-zero levels. These experiments employed Long Short-Term Memory (LSTM) to anticipate future motion trends, leveraging both the controller's movement variables and Electromyography (EMG) data from the operator's arm muscles. Our findings indicate the potential to decrease transmission latency by approximately 500ms. Additionally, our research confirms a direct correlation between prediction accuracy and the brevity of prediction time, suggesting that shorter prediction times yield more accurate results when using EMG. In the context of video transmission for a remotely located robotic arm, we applied video prediction techniques using the Predictive Coding Network (PredNet) to counter network latency. Our results suggest that these predictive methods can effectively compensate for a latency period of 300ms, thereby highlighting their potential for reducing transmission latency in remote robotic operations.

Recently, a worker’s subjective satisfaction, in other words Quality-of-Working Life (QWL), has attracted more attention than productivity or efficiency. To provide QWL-oriented working support in a factory manufacturing environment, this study proposes a framework for recognizing manual assembly behaviors that may reflect a worker’s inner state or physical condition. First, a new set of interactions is defined to describe the behavioral fluctuations and diversity that appear even in the same assembly task. We expand the conventional interaction definitions for manufacturing analysis in three ways: 1) we add primitive interactions that qualify the fundamental interactions, 2) we install a spatial attribute into the interaction definition, and 3) we allow the simultaneous occurrence of multiple interactions. Additionally, an image-based automatic recognition technique is designed to detect the newly defined interactions. Through experimental evaluations for a compressor attachment task, we found various differences in manual assembly behaviors and confirmed that they can be distinguished using the recognized interactions.

A smile is a representative expression of happiness or high quality-of-life; however, automatic recognition of a smile according to happiness remains a challenging task. Because expressions of happiness are strongly dependent upon physical condition and occurrence of other emotions, and similar facial expression often occur under different emotions, we consider that there is no absolute visual pattern of a smile corresponding to happiness. Therefore, in this study, we assumed that a “smile with happiness” is observed as the temporal ascent in the degree of smiling and attempted to recognize this by capturing changes in facial expression within temporally sequential images. As an implementation of this scheme, we proposed a Siamese-structure deep neural network to compare facial expressions in two input images and estimate the existence of smile ascension or descension. For primal analysis of the proposed network, we developed a unique smiling dataset containing image pairs with various changes in smiling degree, including slight changes. The results demonstrated that the proposed method achieved nearly perfect recognition with > 0.95 accuracy when recognizing changes in the degree of smiling that humans certainly recognize. Attention regions that contributed to the predicted labels were concentrated on the mouth, cheeks, and tail of the eyes, which indicates a reasonable function for recognizing changes in smiling degree was constructed by the proposed method.

This paper introduces a novel form of communication via a combination of muscle sensing by electromyography and stimulation via a skin-stretcher device as a motion monitoring system. After sensing muscle activity through electromyography, the skin-stretcher device provides a skin sensation that confidentially informs or induces movements of the user who wears the device. This paper also introduces methods for translating muscle activities to the skin-stretch sensations, and additional filtering to improve the performance. In this study, we conducted preliminary experiments that demonstrate the potential of our system design.

Discovering semantic knowledge is significant for understanding and interpreting how people interact in a meeting discussion. In this paper, we propose a mining method to extract frequent patterns of human interaction based on the captured content of face-to-face meetings. Human interactions, such as proposing an idea, giving comments, and expressing a positive opinion, indicate user intention toward a topic or role in a discussion. Human interaction flow in a discussion session is represented as a tree. Tree-based interaction mining algorithms are designed to analyze the structures of the trees and to extract interaction flow patterns. The experimental results show that we can successfully extract several interesting patterns that are useful for the interpretation of human behavior in meeting discussions, such as determining frequent interactions, typical interaction flows, and relationships between different types of interactions.

Assisting culinary activities for inexperienced users has been considered as an important task in most existing works in the field. On the other hand, recipe texts are becoming available on the Internet in increasing numbers. However, they tend to be written simply by mostly non-professional people, and thus are sometimes difficult for an inexperienced person to follow the steps and manage to cook as they are supposed to. In this paper, we propose a method that detects difficult descriptions for an inexperienced user in an existing text recipe, and supplements them with multimedia contents including text information extracted from a large number of recipes, and also images and video clips on certain kinds of cooking operations, to facilitate the understanding of the recipe. Experimental results showed promising ability of the proposed method to assist inexperienced users understand the descriptions in a recipe.

This paper introduces a novel concept of “Memory Ubiquitous” which provides smart memory functions for appliances, facilities, and equipments used in everyday life. At the first step to realize it, we first propose three fundamental functions. These are capturing and recording the surrounding scene including human behaviors, annotating and editing the records to construct smart memories, and presenting the integrated memories. These functions can support human activities by aiding recall, giving instructions, and providing asynchronous communications. We chose a kitchen as an actual test bed and built a prototype cooking support system with the above smart memory functions. We implemented a context sensitive record presentation based on state recognition and simultaneous display of multiple records that is adaptive to users' intentions. By evaluating the system's performance, we obtained several results that show how smart memory functions can assist human activities and how they can be extended for more useful services.

Human interaction is one of the most important characteristics of group social dynamics in meetings. In this paper, we propose an approach for capture, recognition, and visualization of human interactions. Unlike physical interactions (e.g., turn-taking and addressing), the human interactions considered here are incorporated with semantics, i.e., user intention or attitude toward a topic. We adopt a collaborative approach for capturing interactions by employing multiple sensors, such as video cameras, microphones, and motion sensors. A multimodal method is proposed for interaction recognition based on a variety of contexts, including head gestures, attention from others, speech tone, speaking time, interaction occasion (spontaneous or reactive), and information about the previous interaction. A support vector machines (SVM) classifier is used to classify human interaction based on these features. A graphical user interface called MMBrowser is presented for interaction visualization. Experimental results have shown the effectiveness of our approach.

User-defined preferences in a natural style is useful in the pervasive computing environment, but bring a great challenge to understand. People often express conditional as well as independent preferences. We propose an ontology-based quantitative model for conditional preferences that aims to enhance the inference capacity of conditional preference statements and thus reduce users' workload. Different interpretations of the statements of our model are depicted and compared, including the inheritance property of the concept hierarchy in ontology, the connotation of sufficient and necessary conditions, and the bipolar property of preferences in human thinking. An experiment in the trip domain is conducted and shows the feasibility of our conditional preference model.

In this paper, we propose an approach to handle conditional preferences in recommender systems. A quantitative conditional preference model based on domain knowledge is introduced. The inheritance property in concept trees and bipolar property in preference statements are adopted when interpreting conditional preference rules. Group preferences are merged from personal preferences with consideration of manipulability. A graphical user interface is developed for visualization of domain knowledge, conditional preference rules, personal and group preferences.

Pervasive learning promises an exciting learning environment such that users can access content and study them at anytime, anywhere, through any devices. Besides delivering the right content to the learner, it is necessary to provide acceptable Quality-of-Service (QoS) guarantees in terms of presenting the content. In this paper, we propose a recommendation approach based on fuzzy logic theory towards QoS-aware pervasive learning. It determines appropriate presentation form of the learning content according to user's QoS requirements and device/network capability. We also propose an adaptive QoS mapping strategy, which dynamically sets quality parameters at running time according to the capabilities of client devices. The experimental results show the proposed approach is feasible and acceptable to enable QoS-aware pervasive learning.

In this paper, we describe the enabling technologies to develop a smart meeting system based on a three layered generic model. From physical level to semantic level, it consists of meeting capturing, meeting recognition, and semantic processing. Based on the overview of underlying technologies and existing work, we propose a novel real-world smart meeting application, called MeetingAssistant. It is distinctive from previous systems in two aspects. First it provides the real-time browsing that allows a participant to instantly view the status of the current meeting. This feature is helpful in activating discussion and facilitating human communication during a meeting. Second, the context-aware browsing adaptively selects and displays meeting information according to user's situational context, e.g., user purpose, which makes meeting viewing more efficient.

In this paper, we discuss a user interface issue with regard to automatic video editing based on the speaker's intentions. In our experiments, the subjects used previously developed video capturing system to employ 4 types of editing methods by making 3 types of presentations. Subjective evaluation revealed that the editing method that used behaviors-for-attention obtained a good score for a presentation in which the subject was provided with specific instructions regarding the tasks to be performed. In the case of a presentation without a scenario, an editing method using a footswitch and a posture obtained a higher score. It can be concluded that a combination of both behavior-based and footswitch-based editing would provide a good environment for content acquisition

This paper introduces video quality analysis for automated video capture and editing. Previously, we proposed an automated video capture and editing system for conversation scenes. In the capture phase, our system not only produces concurrent video streams with multiple pan-tilt-zoom cameras but also recognizes "conversation states" i.e., who is speaking, when someone is nodding, etc. As it is necessary to know the conversation states for the automated editing phase, it is important to clarify how the recognition rate of the conversation attributes affects our editing system with regard to the quality of the resultant videos. In the present study, we analyzed the relationship between the recognition rate of conversation states and the quality of resultant videos through subjective evaluation experiments. The quality scores of the resultant videos were almost the same as the best case in which recognition was done manually, and the recognition rate of our capture system was therefore sufficient.

In desktop manipulations such as cooking or assembly instruction videos, a performer often draws the viewers' attention to significant portions of the video's content by means of a specific set of physical behaviors. We verified the efficacy of videos edited based on those behaviors by observing television programs and simulating them with the use of our editing scheme. We first discuss the typical behaviors commonly shown in TV programs, and statistically describe the relationships between certain types of shot changes. We then propose an editing method using those behaviors, and demonstrate that our method generates videos whose quality is close to that of some of television programs. Moreover, we demonstrate the usability of our online editing system in experiments involving actual scenes.

We often need too much time for reading documents, since it is often difficult to efficiently grasp its outline. For this purpose, we propose our diagram generation scheme for presenting the structures of a text. The semantic structure of a tagged text is effectively translated to diagrams, and they are linked to the text. In this paper, first, we describe how semantic structures can be expressed by a diagram. Then, we propose our framework for automatic diagram generation from tagged texts to diagrams.

In this paper, we introduce an intelligent system for video recording. First, we categorized targets and purposes of shooting, and discuss the cameraworks appropriate for them. Then, we propose camera control algorithms to realize such cameraworks. Based on this idea, we built a prototype pan-tilt camera control system, in which multiple cameras with different purposes automatically track and shoot the targets. We evaluated our system through recording of some presentations on desktop manipulation. The effectiveness of our algorithm was verified through some experiments.

Introduces a method for analyzing video records which contain personal activities captured by a head mounted camera. This aims to support the user in retrieving the most important or relevant portions from the videos. For this purpose, we use the user's behaviors which appear when he/she pays attention to something. We define two types of those behaviors, one of which is "gaze at something in a short period" and the other is "staying and continuously see something". These behaviors and the focused object can be detected by estimating camera and object motion. We describe the details of the method and experiments in which the method was applied to ordinary events.

We propose a method for classifying image edges caused by different physical phenomena, i.e. reflectance change, shadow, occlusion, etc., by using color information around the edge. We assumed several simple models for object spatial arrangements. For each of them, typical locus of RGB values along the normal direction of each edge segment is modeled. Each locus is parametrized by several features. In the classification of actual edges, the most plausible phenomenon is selected by checking the consistency between the parameters from an actual edge profile and those from each model. For the improvement of accuracy, Dempster-Shafer probability model is employed to deal with the above parameters that are often weak and uncertain. Experiments showed good performances.

Spotting by Associationmethod for video analysis is a novel method to detect video segments with typical semantics.Video data contains various kinds of information through continuous images, natural language, and sound.For videos to be stored and retrieved in a Digital Libmry, it is essential to segment the video data into meaningful pieces.To detect meaningful segments, we need to identify the segment in each modality (video, language, and sound) that corresponds to the same story.For this purpose, we propose a new method for making correspondences between image clues detected by image analysis and language clues detected by natural language analysis.As a result, relevant video segments with sufficient information from every modality are obtained.We applied our method to closed-captioned CNN Headline News.Video segments with important events, such as a public speech, meeting, or visit, are detected fairly well.'For example, a close-up of a policeman's face and the name of the criminal.

In stereo algorithms with more than two cameras, the improvement of accuracy is often reported since they are robust against noise. However, another important aspect of the polynocular stereo, that is the ability of occlusion detection, has been paid less attention. We intensively analyzed the occlusion in the camera matrix stereo (SEA) and developed a simple but effective method to detect the presence of occlusion and to eliminate its effect in the correspondence search. By considering several statistics on the occlusion and the accuracy in the SEA, we derived a few base masks which represent occlusion patterns and are effective for the detection of occlusion. Several experiments using typical indoor scenes showed quite good performance to obtain dense and accurate depth maps even at the occluding boundaries of objects.

We propose a method to analyze detailed patterns on contour shapes. In this method, we express detailed patterns using wavelet local extrema which are obtained through wavelet transforms. Based on this description, two features of detailed patterns, the properties of small fractions (or corners) constituting the detailed patterns and the arrangement of these corners are extracted. Using these two important features, the similarities between two detailed patterns can be examined. The proposed method is applied to detailed patterns of leaf contours and some hand-drawn contours to show its feasibility.

The basic idea of the method for shape learning is presented. Each shape is hierarchically characterized with singularities (local extrema) of wavelet transform of its tangential orientation function. By taking the common structure of the singularities of two or more shapes, the common characteristics of the input shapes are obtained. Through this operation, not only the rough outline of a shape but also several important characteristics of shapes are acquired.

Diagram understanding and its cooperative use with other media are important subjects in both pattern understanding and communication. However, it is quite difficult to understand diagrams without supplementary explanation by other media. For this purpose, the authors propose a new framework for semantic understanding of a diagram by utilizing textual information. In this framework, the elements in a diagram are tightly linked to words or other structure in a natural language text, and the semantic structure of a diagram is interpreted clearly by using natural language information. The obtained result contributes to the construction of hypertext and multimedia.>

Introduces a system (PAFE) which provides a flexible framework for feature extraction. This system is constructed based on the multi agents model, in which many agents work co-operatively and concurrently. The objective of this system is to achieve flexible control and coarse-grained parallelism of feature extraction by organizing feature extraction as parallel graph search.>