Vision-language models have shown impressive progress in recent years. However, existing models are largely limited to turn-based interactions, where each turn must be stepped (i.e., prompted) by the user. Open-ended, asynchronous interactions, where an AI model may proactively deliver timely responses or feedback based on the unfolding situation in real-time, are an open challenge. In this work, we present the QEVD benchmark and dataset, which explores human-AI interaction in the challenging, yet controlled, real-world domain of fitness coaching – a task which intrinsically requires monitoring live user activity and providing immediate feedback. The benchmark requires vision-language models to recognize complex human actions, identify possible mistakes, and provide appropriate feedback in real-time. Our experiments reveal the limitations of existing state-of-the-art vision-language models for such asynchronous situated interactions. Motivated by this, we propose a simple end-to-end streaming baseline that can respond asynchronously to human actions with appropriate feedback at the appropriate time.

Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the “something-something” database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.

Tagmantic.com is a web based social recommendation and search service which exploits the rich information within folksonomies. By developing multiple layers of organizational structure on top of folksonomies the relations between tags, users and resources can be utilized in order to create advanced recommender engines. The first layer of organizational structure is a tag graph which relates different entities within the folksonomy to each other. Its nodes are the concepts of the folksonomy presented by tags and the edges describe the relationships between these concepts. The second layer of organizational structure are tag clusters, which are build on top of the above mentioned tag graphs and represent the ontology of the underlying folksonomy. These Ontologies as well as the less complex tag graphs are utilized by the recommender algorithms within tagmantic in order to provide a unique and intuitive user experience of exploring and searching content in the internet. The methodology can be applied to different types of folksonomies which implies an enormous potential for future development of new recommender engines.

We apply a hierarchical feed-forward neural architecture to the problem of face recognition. The network is similar to the neocognitron-approach and a two-layer variation of this architecture, which has previously been successfully applied to patch classification tasks. We extend this architecture to a three-layer one, which allows not only identification of image patches, but also detection in larger images. In the research area of face recognition, a lot of expertise has been developed for the problem of either identification or detection, but approaches which deal with both problems simultaneously are rarely to be found. In this work, we apply the hierarchical approach to this problem and evaluate the performance on artificial datasets.

We describe an augmented reality system designed for online acquisition of visual knowledge and retrieval of memorized objects. The system relies on a head mounted camera and display, which allow the user to view the environment together with overlaid augmentations by the system. In this setup, communication by hand gestures and speech is mandatory as common input devices like mouse and keyboard are not available. Using gesture and speech, basically three types of tasks must be handled: (i) Communication with the system about the environment, in particular, directing attention towards objects and commanding the memorization of sample views; (ii) control of system operation, e.g. switching between display modes; and (iii) re-adaptation of the interface itself in case communication becomes unreliable due to changes in external factors, such as illumination conditions. We present an architecture to manage these tasks and describe and evaluate several of its key elements, including modules for pointing gesture recognition, menu control based on gesture and speech, and control strategies to cope with situations when vision becomes unreliable and has to be re-adapted by speech.