We introduce a novel perception-action-learning system for mobile social-service robots. The state-of-the-art deep learning techniques were incorporated into each module which significantly improves the performance in solving social service tasks. The system not only demonstrated fast and robust performance in a homelike environment but also achieved the highest score in the RoboCup2017@Home Social Standard Platform League (SSPL) held in Nagoya, Japan.

The capability of following a person is crucial in service-oriented robots for human assistance and cooperation. Though a vast variety of following systems exist, they lack robustness against dynamic changes of the environment and relocating to continue following a lost target. Here we present a robust human following system that has the extendability to commercial service robot platforms having a RGB-D camera. The proposed framework integrates deep learning methods for perception and variational Bayesian techniques for trajectory prediction. Deep learning modules enable robots to accompany a person by detecting the target, learning the target and following while avoiding collision within the dynamic home environment. The variational Bayesian techniques robustly predict the trajectory of the target by empowering the following ability of the robot when target is lost. We experimentally demonstrate the capability of the deep Bayesian trajectory prediction method on real-time usage, following abilities, collision avoidance and trajectory prediction of the system. The proposed system was deployed at the RoboCup@Home 2017 Social Standard Platform League and successfully demonstrated its robust functions and smooth person following capability resulting in winning the 1st place.

Concepts have been widely used for categorizing and representing knowledge in artificial intelligence. Previous researches on concept learning have focused on unimodal data, usually on linguistic domains in a static environment. Concept learning from multimodal stream data, such as videos, remains a challenge due to their dynamic change and high-dimensionality. Here we propose an evolutionary method that simulates the process of human concept learning from multimodal video streams. Two key ideas on evolutionary concept learning are representing concepts in a large collection (population) of hyperedges or a hypergraph and to incrementally learning from video streams based on an evolutionary approach. The hypergraph is learned "evolutionarily" by repeating the generation and selection process of hyperedge concepts from the video data. The advantage of this evolutionary learning process is that the population-based distributed coding allows flexible and robust trace of the change of concept relations as the video story unfolds. We evaluate the proposed method on a suite of children's cartoon videos for 517 minutes of total playing time. Experimental results show that the proposed method effectively represents visual-textual concept relations and our evolutionary concept learning method effectively models the conceptual change as an evolutionary process. We also investigate the structure properties of the constructed concept networks.

Text-to-image retrieval is to retrieve the images associated with the textual queries. A text-to-image retrieval model requires an incremental learning method for its practical use since the multimodal data grow up dramatically. Here we propose an incremental text-to-image retrieval method using a multimodal association model. The association model is based on a hypernetwork (HN) where a vertex corresponds to a textual word or a visual patch and a hyperedge represents a higher-order multimodal association. Using the HN incrementally learned by a sequential Bayesian sampling, in the multimodal hypernetwork-based text-to-image retrieval, a given text query is crossmodally expanded to the visual query and then similar images are retrieved to the expanded visual query. We evaluated the proposed method using 3,000 images with textual description from Flickr.com. The experimental results present that the proposed method achieves very competitive retrieval performances compared to a baseline method. Moreover, we demonstrate that our method provides robust text-to-image retrieval results for the increasing data.