Abstract The ability to understand the surrounding environment and being able to communicate with interacting humans are important functionalities for many automated systems where visual input (e.g., images, video) and natural language input (speech or text) have to be related to each other. Possible applications are automatic image caption generation, interactive surveillance systems, or human robot interaction. In this paper, we propose algorithms for automatic responses to natural language queries about an image. Our approach uses a predefined neural net for detection of bounding boxes and objects in images, spatial relations between bounding boxes are modeled with a neural net, the queries are analyzed with a syntactic parser, and algorithms to map natural language to properties in the images are introduced. The algorithms make use of semantic similarity and antonyms. We evaluate the performance of our approach with test users assessing the quality of our system’s generated answers.

Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed.While the power of visual-semantic embeddings comes from the distillation and enrichment of information through machine learning, their inner workings are poorly understood and there is a shortage of analysis tools.To address this problem, we generalize the notion of probing tasks to the visual-semantic case.To this end, we (i) discuss the formalization of probing tasks for embeddings of image-caption pairs, (ii) define three concrete probing tasks within our general framework, (iii) train classifiers to probe for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks.Our experiments reveal an up to 12% increase in accuracy on visual-semantic embeddings compared to the corresponding unimodal embeddings, which suggest that the text and image dimensions represented in the former do complement each other.

In human communication, backchanneling is an important part of the natural interaction protocol. The purpose is to signify the listener's attention, understanding, agreement, or to indicate that a speaker should go on talking. While the effects of backchanneling robots on humans have been investigated, studies of how and when humans backchannel to talking robots is poorly studied. In this paper we investigate how the robot's behavior as a speaker affects a human listener's backchanneling behavior. This is interesting in Human-Robot Interaction since backchanneling between humans has been shown to support more fluid interactions, and human-robot interaction would therefore benefit from mimicking this human communication feature. The results show that backchanneling increases when the robot exhibits backchannel-inviting cues such as pauses and gestures. Furthermore, clear differences between how a human backchannels to another human and to a robot are shown.

In verbal human-robot interaction natural language utterances have to be grounded in visual scenes by the robot. Visual language grounding is a challenging task that includes identifying a primary object among several objects, together with the object properties and spatial relations among the objects. In this paper we focus on extracting this information from sentences only. We propose two language modelling techniques, one uses regular expressions and the other one utilizes Euclidian distance. We compare these two proposed techniques with two other techniques that utilize tree structures, namely an extended Hobb’s algorithm and an algorithm that utilizes a Stanford parse tree. A comparative analysis between all language modelling techniques shows that our proposed two approaches require less computational time than the tree-based approaches. All approaches perform good identifying the primary object and its property, but for spatial relation extraction the Stanford parse tree algorithm performs better than the other language modelling techniques. Time elapsed for the Stanford parse tree algorithm is higher than for the other techniques.

In many complex robotics systems, interaction takes place in all directions between human, robot, and environment. Performance of such a system depends on this interaction, and a proper evaluation of a system must build on a proper modeling of interaction, a relevant set of performance metrics, and a methodology to combine metrics into a single performance value. In this paper, existing models of human-robot interaction are adapted to fit complex scenarios with one or several humans and robots. The interaction and the evaluation process is formalized, and a general method to fuse performance values over time and for several performance metrics is presented. The resulting value, denoted interaction quality, adds a dimension to ordinary performance metrics by being explicit about the interplay between performance metrics, and thereby provides a formal framework to understand, model, and address complex aspects of evaluation of human-robot interaction.

In this paper we model virtual buying cooperatives (VBC) with grammars of regulated rewriting and show that, if VBC relevant information is distributed over several successive VBC processes and mus ...

Many non-classical models of automata are natural objects of theoretical computer science. They are studied from different points of view in various areas, both as theoretical concepts and as formal models for applications. The Fifth Workshop on Non-Classical Models of Automata and Applications (NCMA 2013) was organized in order to provide an opportunity for researchers who work on different aspects of non-classical models of automata and related subjects to exchange and discuss new ideas and recent developments.