Knowledge graphs (KGs) are useful information sources to make machine learning efficient with human knowledge. Since KGs are often incomplete, KG completion has become an important problem to complete missing facts in KGs. Whereas most of the KG completion methods are conducted on a single KG, multiple KGs can be effective to enrich embedding space for KG completion. However, most of the recent studies have concentrated on entity alignment prediction and ignored KG-invariant semantics in multiple KGs that can improve the completion performance. In this paper, we propose a new multiple KG embedding method composed of intra-KG and inter-KG regularization to introduce KG-invariant semantics into KG embedding space using aligned entities between related KGs. The intra-KG regularization adjusts local distance between aligned and not-aligned entities using contrastive loss, while the inter-KG regularization globally correlates aligned entity embeddings between KGs using multi-view loss. Our experimental results demonstrate that our proposed method combining both regularization terms largely outperforms existing baselines in the KG completion task.

Gaze prediction is a significant approach for processing a large amount of incoming visual information of videos. Recent gaze prediction algorithms often employ sparse models with the assumption that every superpixel in the video frames can be represented as linear combinations of a few salient superpixels. However, they are not actuated enough because of the insufficient knowledge that video signals contain a non-negative request. Hence, we develop a novel gaze prediction based on an inverse sparse coding framework with a determinant sparse measure. By introducing this sparse measure, the solutions are non-negative and sparser than conventional sparse constraints. However, the proposed optimization problem becomes nonconvex, which is difficult to solve. To efficiently address the corresponding nonconvex optimization problem, we propose a novel algorithm based on the difference in convex function programming, which can yield the global solutions. Experimental results indicate the improved accuracy of the proposed approach compared with state-of-the-art algorithms.

To construct subject-specific feature extractors and classifiers for a new subject using pooled datasets, overcoming intersubject variabilities is required. In this study, we investigate the efficiency of the proposed subject transfer framework, which applies a discriminability-based source selection approach and semi-supervised style transfer mapping algorithm, by constructing support vector machine classifiers. We collect a surface electromyogram (sEMG) dataset acquired from 25 subjects using a wearable sEMG sensor. Classifiers are trained with gold-standard time-domain and autoregressive features extracted from eight-channel sEMG data. Compared with conventional subject transfer framework (85.08±1.38%), which applies the covariate shift adaptation algorithm to the linear discriminant analysis classifier and uses all source data, our proposed framework improves pattern recognition accuracy (90.63 ± 1.27%) by selection of discriminative source data and the mapping destination in the Euclidean space.

This paper addresses two issues toward practical use of wearable electroencephalogram (EEG) measurement devices. Ocular (eye movement and blink) artifacts often contaminate EEGs and deteriorate the performance of EEG-based brain–computer interfaces (BCIs). Although wearable consumer-grade EEG devices with single electrode allow users to operate BCIs conveniently in daily lives, it remains a challenging issue to attenuate ocular artifacts from single-channel measurements without spatial information. Existing ocular artifact reduction methods are, however, not simple enough for single-channel EEG data in the sense that they require an additional reference channel and/or pre-processing for artifact segment detection. Another issue is how to assess the performance of artifact reduction; the existing studies have used their own datasets that are not accessible from other researchers. Then, this paper makes two major contributions. (1) This paper proposes a novel ocular artifact reduction method, multi-scale dictionary learning (MSDL), which operates under single-channel measurements and without artifact segment detection. (2) We also develop a semi-simulation setting for quantitative evaluation with a publicly available EEG dataset. In particular, we employed BCI Competition IV Dataset 2a, on which the proposed method was compared with state-of-art methods. The proposed technique showed the best performance for recovering artifact-reduced waveforms from single-channel data compared to the other artifact reduction methods. The Matlab scripts for semi-simulation data generation and single-channel artifact reduction are available on GitHub.

To efficiently process and understand a large amount of incoming visual information from first-person perspective (i.e. egocentric vision), predicting human gaze is important. However, even though people continuously gaze in noisy environments, most existing gaze prediction methods mainly use image saliency, which is sensitive to noise in the real-world. To address this issue, we propose a sparse coding-based saliency detection method for gaze prediction. Our model uses a cost function with the 10 norm as a sparse constraint that can control the area of visual saliency in response to the contents of egocentric vision in intuitive and consistent ways. Moreover, we use canonical correlation analysis (CCA) to combine different types of features for reducing noise and the computational complexity. We also utilize the temporal continuity of image frames when defining our saliency. Experiments using a real-world gaze dataset show that our proposed approach outperforms the state-of-the-art algorithms on gaze prediction in egocentric videos.

Quantitative modeling of human brain activity based on language representations has been actively studied in systems neuroscience. However, previous studies examined word-level representation, and little is known about whether we could recover structured sentences from brain activity. This study attempts to generate natural language descriptions of semantic contents from human brain activity evoked by visual stimuli. To effectively use a small amount of available brain activity data, our proposed method employs a pre-trained image-captioning network model using a deep learning framework. To apply brain activity to the image-captioning network, we train regression models that learn the relationship between brain activity and deep-layer image features. The results demonstrate that the proposed model can decode brain activity and generate descriptions using natural language sentences. We also conducted several experiments with data from different subsets of brain regions known to process visual stimuli. The results suggest that semantic information for sentence generations is widespread across the entire cortex.

The goal of fairness-aware classification is to categorize data while taking into account potential issues of fairness, discrimination, neutrality, and/or independence. For example, when applying data mining technologies to university admissions, admission criteria must be non-discriminatory and fair with regard to sensitive features, such as gender or race. In this context, such fairness can be formalized as statistical independence between classification results and sensitive features. The main purpose of this paper is to analyze this formal fairness in order to achieve better trade-offs between fairness and prediction accuracy, which is important for applying fairness-aware classifiers in practical use. We focus on a fairness-aware classifier, Calders and Verwer’s two-naive-Bayes ( CV2NB ) method, which has been shown to be superior to other classifiers in terms of fairness. We hypothesize that this superiority is due to the difference in types of independence. That is, because CV2NB achieves actual independence, rather than satisfying model-based independence like the other classifiers, it can account for model bias and a deterministic decision rule. We empirically validate this hypothesis by modifying two fairness-aware classifiers, a prejudice remover method and a reject option-based classification ( ROC ) method, so as to satisfy actual independence. The fairness of these two modified methods was drastically improved, showing the importance of maintaining actual independence, rather than model-based independence. We additionally extend an approach adopted in the ROC method so as to make it applicable to classifiers other than those with generative models, such as SVMs.

Selecting authentic scenes about activities of daily living (ADL) is useful to support our memory of everyday life. Key-frame extraction for first-person vision (FPV) videos is a core technology to realize such memory assistant. However, most existing key-frame extraction methods have mainly focused on stable scenes not related to ADL and only used visual signals of the image sequence even though the activities usually associate with our visual experience. To deal with dynamically changing scenes of FPV about daily activities, integrating motion and visual signals are essential. In this paper, we present a novel key-frame extraction method for ADL, which integrates multi-modal sensor signals to temper noise and detect salient activities. Our proposed method projects motion and visual features to a shared space by a probabilistic canonical correlation analysis and selects key frames there. The experimental results using ADL datasets collected in a house suggest that our key-frame extraction technique running in the shared space improves the precision of extracted key frames and the coverage of the entire video.

First-person videos (FPVs) in daily living help us to memorize our life experience and information systems to process daily activities. Summarizing FPVs into key frames that represent the entire data would allow us to remember our memory in the past and computers to efficiently process the data. However, most video summarization approaches only use visual information, even though our daily activities consist of multiple modalities such as movements and sounds. FPVs are not as stable as movies or sport scenes since the camera attached to the head shakes frequently, and key frame extraction methods rely only on video frames do not always produce satisfactory results. In this paper, we introduce a novel key frame extraction method for FPVs using multiple wearable sensors. To efficiently integrate multimodal sensor signals, our formulation uses sparse dictionary selection, which minimizes a reconstruction error with a subset (key frames) of the original data. We present experimental results with multimodal datasets captured by wearable sensors in a natural environment. The results suggest multi-sensor information improves the precision of extracted key frames as well as the coverage of an entire video sequence.

In this study, we investigate whether sparse coding helps explain the semantic representation in human cerebral cortex. We show this by using sparse coding to model semantic representation in the cerebral cortex. We propose three methods for estimating semantic representation from brain activity data. For estimating a new semantic representation, in the first method, we use only a semantic representation dictionary obtained via sparse coding. The semantic representation estimated using this method is more similar to the actual semantic representation of the cerebral cortex than that estimated without sparse coding. In the second method, we use only a brain activity dictionary obtained via sparse coding. The semantic representation estimated using this method is also better than that estimated without sparse coding. In addition, in the third method, we estimate semantic representation by applying sparse coding to both semantic representation and brain activity data. The semantic representation estimated by using this third method is better than that estimated by the first or second methods. Through the above three experiments, we have confirmed that sparse coding helps explain the semantic representation in human cerebral cortex.

Muhammad Attamimi, Yuji Ando, Tomoaki Nakamura, Takayuki Nagai, Daichi Mochihashi, Ichiro Kobayashi, Hideki Asoh. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 2015.

The performance of sound source localization is often reduced by the presence of colored noise in the environment, such as room reverberation. In this study, a method for estimating the noise spatial covariance using a hierarchical model is proposed and its performance is evaluated. By employing the hierarchical model in joint Bayesian estimation, robust estimation of the covariance is expected with a relatively small amount of data. Moreover, a method of jointly estimating the number of sources is introduced so that it can be used for cases in which the number of active sources dynamically changes, for example, speech signals. The results of the experiments performed using actual room reverberation show the effectiveness of the proposed method.

In this paper, source localization in spatially colored noise is addressed. The covariance of colored noise is estimated using a hierarchical model in the joint Bayesian estimation. The results of the experiment show that the spatial resolution was improved compared with the approach without hierarchical modeling.

A particle filtering algorithm using the parameters in the EM (expectation-maximization) algorithm is proposed for tracking multiple sound sources. Differently from the conventional EM based algorithms, the proposed algorithm can track multiple sound sources without knowing their starting points. Moreover, an idea of the group tracking is applied to the particle filtering algorithm so that better tracking performances can be obtained. Experimental results show the validity of the proposed algorithm.

A method of detecting and tracking moving human speech events by integrating audio and video signals using a particle filter is proposed and evaluated. Using the particle filter, location, on/off status, and human/non-human determination can be estimated simultaneously for multiple sound sources. Experiments demonstrate that the proposed method performs well for the data obtained in an ordinary meeting room using a microphone array and a monocular camera.

For cooperative work of robots and humans in the real world, a communicative function based on speech is indispensable for robots. To realize such a function in a noisy real environment, it is essential that robots be able to extract target speech spoken by humans from a mixture of sounds by their own resources. We have developed a method of detecting and extracting speech events based on the fusion of audio and video information. In this method, audio information (sound localization using a microphone array) and video information (human tracking using a camera) are fused by a Bayesian network to enable the detection of speech events. The information of detected speech events is then utilized in sound separation using adaptive beam forming. In this paper, some basic investigations for applying the above system to the humanoid robot HRP-2 are reported. Input devices, namely a microphone array and a camera, were mounted on the head of HRP-2, and acoustic characteristics for sound localization/separation performance were investigated. Also, the human tracking system was improved so that it can be used in a dynamic situation. Finally, overall performance of the system was tested via off-line experiments.

Two array signal processing techniques are combined with independent component analysis (ICA) to enhance the performance of blind separation of acoustic signals in a reflective environment. The first technique is the subspace method which reduces the effect of room reflection when the system is used in a room. Room reflection is one of the biggest problems in blind source separation (BSS) in acoustic environments. The second technique is a method of solving permutation. For employing the subspace method, ICA must be used in the frequency domain, and precise permutation is necessary for all frequencies. In this method, a physical property of the mixing matrix, i.e., the coherency in adjacent frequencies, is utilized to solve the permutation. The experiments in a meeting room showed that the subspace method improved the rate of automatic speech recognition from 50% to 68% and that the method of solving permutation achieves performance that closely approaches that of the correct permutation, differing by only 4% in recognition rate.

Two array signal processing techniques are combined with independent component analysis to enhance the performance of blind separation of acoustic signals in a reflective environment such as rooms. The first technique is the subspace method which reduces the effect of room reflection. The second technique is a method of solving the permutation, in which the coherency of the mixing matrix in adjacent frequencies is utilized.

We propose a method for extracting low-dimensional features from omnidirectional images to be used for robot localization and navigation. Edge detection is combined with thresholding to locate sharp edge pixels, the coordinates of which are fed into a Parzen density estimator (1962) to compute the edge spatial density. The use of the fast Fourier transform makes this density estimate feasible in real-time, while principal component analysis further drops the dimensionality of the resulting feature vector to a manageable number. We show experimental results from a Nomad XR4000 robot in an office environment.

A real-time sound localization/separation system for near-field sound sources was constructed and evaluated in a real office environment. As for the sound localization, the experimental results showed that the direction of the two sources was estimated with high accuracy while the range of the sources was estimated with moderate accuracy. As for the sound separation, a recognition rate of 70 % for an on-line recognizer on a network and of 90% for an off-line recognizer were achieved, respectively.

In order to determine both location and orientation of a mobile robot, we use omni-directional view images from a camera mounted on top of the robot. Images are sampled circularly on rings at various radii. Correlation values between rings of different images are regarded as the similarity of the image, therefore the similarity of locations. Using real images taken at different times in navigation experiments, we computed correlation values normalized by brightness, and confirmed the robot can be localized in 50 cm to 100 cm range with 3-10 degree orientation accuracy. The idea of using aggregated sum of circular samples for representing landmarks at multiple levels is also presented.

We have been developing an autonomous robotic agent that helps people in a real world environment, such as in an office. When a robotic agent works by cooperating with a person in a real world environment, it must manage a lot of information and deal with the knowledge and languages that people usually use. Therefore it is important for the agent to recognize what people request as soon as possible. To realize common communication with people, the agent should provide robust communicative functions to obtain information from people. We discuss communicative functions of our robotic agent called Jijo-2. Especially we focus on the problem of detecting human faces, and discuss how a method of detecting a human face can be robustly archived.

This paper proposes a multi-agent software architecture for controlling higher level behaviors of a mobile office-conversant robot. In order to keep the whole system comprehensive and extensible, a group structure is introduced into multi-agent architecture. In the architecture, the whole system is divided into several behavior-manager agents and the behavior-manager manages multiple behavior agents, which are dynamically created and destroyed on demands. Several behaviors and behavior-managers are implemented as classes of an object-oriented Lisp. Dialog-based map learning and teleoperation experiments with real mobile robot Jijo-2 are shown for evaluating the architecture.

This paper reports on our work on the spoken dialogue interface to Jijo-2, an autonomous mobile office robot. We propose an attentional state model for office robot dialogue based on the centering model [2]. We give details of the implementation of Jijo-2's language components and give our algorithm for reducing the ambiguity of dialogues by identifying pronoun referents.

In order for our office conversant robot to take robust actions according to events rising from different kind of sensors, a layered process network architecture based on an event-driven control model is presented. In this architecture, high level modules are freed from time consuming checking of sensors, and are able to follow multiple scenario-driven consistent behaviors, allowing interruption from human and environment. Programs are written as series of event-action pairs, which are also effective to keep reactiveness and modularity high.

A design of mobile robot for robust life-long navigation in office environment is proposed and evaluated. The key idea is combining probabilistic map and dialog with humans for reducing the location uncertainty. Bayesian inference with the map represented by probabilistic automata is used in order to reduce the number of queries and to evaluate the success rate of planned paths. We experimentally implemented the design using a simple Bayesian network with continuous nodes and demonstrated its effectiveness in a real environment.

An architecture of a four-layer (two hidden layers) neural network is proposed in order to approximate nonlinear discriminant analysis. The architecture is based on a previously observed relationship between multilayer neural networks and back-propagation (least mean squared error) learning and nonlinear data analysis methods. The effectiveness of the architecture has been verified experimentally. It is shown that the networks have a much stronger capability of class separation than the usual linear discriminant analysis method