Driver monitoring will be required in many countries for all new vehicles with automation functions. While the common approach for this task is face and eye gaze monitoring, cameras with a wider field of view have already been introduced in some cars. However, their mounting position can change between vehicle models. To minimize data collection efforts and to facilitate data reuse it is important for algorithms to be able to deal with a changing environment. We conduct an experiment comparing the performance of video-based models with 3D body pose-based activity recognition methods with regards to sensor and position changes. We introduce a modular activity recognition pipeline which uses a sensor independent representation including the 3D body pose of the driver, 3D dynamic object positions and 3D interior positions. We show that while video-based models offer the best quality overall, when trained and tested on the same camera view, body pose-based methods can be far more robust to positional changes. Moreover, augmentation reduces the performance drop across views to 35% compared to 83% without augmentation and 90% for video-based models.

Although human action anticipation is a task which is inherently multi-modal, state-of-the-art methods on well known action anticipation datasets leverage this data by applying ensemble methods and averaging scores of uni-modal anticipation networks. In this work we introduce transformer based modality fusion techniques, which unify multi-modal data at an early stage. Our Anticipative Feature Fusion Transformer (AFFT) proves to be superior to popular score fusion approaches and presents state-of-the-art results outperforming previous methods on EpicKitchens-100 and EGTEA Gaze+. Our model is easily extensible and allows for adding new modalities without architectural changes. Consequently, we extracted audio features on EpicKitchens-100 which we add to the set of commonly used features in the community.1

The selection operation is a basic input operation when interacting with a computer. Traditional manual selection methods like mouse input are challenging when interacting with a dynamic scene containing moving objects as it occurs in surveillance applications. In this contribution, we give an overview on gaze-based selection as a fast and intuitive alternative method considering typical selection tasks in surveillance applications. In this context, we report the results of an evaluation on initialization of an object-tracking algorithm performed by eighteen expert video analysts. Besides its benefits, gaze-based selection is difficult if selection objects are small and close together. We provide first results of a pilot study evaluating a distance measure, which might have the potential of making gaze-based selection more robust. Finally, to leverage gaze-based selection becoming a common technique, low-cost eye-tracking devices have to be available which achieve the same accuracy as the high-end devices. As such cheap eye-trackers only recently became available, it is so far not evident, which accuracy we can expect. Hence, we report the results of an accuracy evaluation of the Tobii 4C conducted with twelve students performing a calibration-style task.

We introduce the novel domain-specific Drive&Act benchmark for fine-grained categorization of driver behavior. Our dataset features twelve hours and over 9.6 million frames of people engaged in distractive activities during both, manual and automated driving. We capture color, infrared, depth and 3D body pose information from six views and densely label the videos with a hierarchical annotation scheme, resulting in 83 categories. The key challenges of our dataset are: (1) recognition of fine-grained behavior inside the vehicle cabin; (2) multi-modal activity recognition, focusing on diverse data streams; and (3) a cross view recognition benchmark, where a model handles data from an unfamiliar domain, as sensor type and placement in the cabin can change between vehicles. Finally, we provide challenging benchmarks by adopting prominent methods for video- and body pose-based action recognition.

Predicting an observer's tasks from eye movements during several viewing tasks has been investigated by several authors. This contribution adds task prediction from eye movements tasks occurring during motion image analysis: Explore, Observe, Search, and Track. For this purpose, gaze data was recorded from 30 human observers viewing a motion image sequence once under each task. For task decoding, the classification methods Random Forest, LDA, and QDA were used; features were fixation- or saccade-related measures. Best accuracy for prediction of the three tasks Observe, Search, Track from the 4-minute gaze data samples was 83.7% (chance level 33%) using Random Forest. Best accuracy for prediction of all four tasks from the gaze data samples containing the first 30 seconds of viewing was 59.3% (chance level 25%) using LDA. Accuracy decreased significantly for task prediction on small gaze data chunks of 5 and 3 seconds, being 45.3% and 38.0% (chance 25%) for the four tasks, and 52.3% and 47.7% (chance 33%) for the three tasks.

Distraction of the driver by secondary tasks is already dangerous while driving manually but especially in handover situations in an automated mode this can lead to critical situations. Currently, these tasks are not taken into account in most modern cars. We present a system that detects typical distracting secondary tasks in an efficient modular way. We first determine the body pose of the driver and afterwards use recurrent neuronal networks to estimate actions based on sequences of the captured body poses. Our system uses knowledge about the surroundings of the driver that is unique to the car environment. Our evaluation shows that this approach achieves better results than other state of the art systems for action recognition on our dataset.

Within a manufacturing system, a smart human-machine interface reduces the chances of human error and helps users to make informed decisions, especially in critical situations. This paper presents a concept for a flexible context specific assistance system for industrial applications using camera based localization. As a central element, a modular and service oriented context aware system aggregates relevant data from different sources. An object recognition service applies image analysis techniques on a video stream, captured from a top mounted multi-camera system to detect a person's location and associate it with a mobile device. A semantically enriched OPC UA server provides access to process data, and a web-service provides connection to a user interface hosted on a tablet PC. A case study provides an application of the proposed solution for the maintenance support and indoor navigation is implemented as a proof of concept.

The knowledge about the body orientation of humans can improve speed and performance of many service components of a smart-room. Since many of such components run in parallel, an estimator to acquire this knowledge needs a very low computational complexity. In this paper we address these two points with a fast and efficient algorithm using the smart-room's multiple camera output. The estimation is based on silhouette information only and is performed for each camera view separately. The single view results are fused within a Bayesian filter framework. We evaluate our system on a subset of videos from the CLEAR 2007 dataset and achieve an average correct classification rate of 87.8%, while the estimation itself just takes 12 ms when four cameras are used.

To provide intelligent services in a smart environments it is necessary to acquire information about the room, the people in it and their interactions. This includes, for example, the number of people, their identities, locations, postures, body and head orientations, among others. This paper gives an overview of the perceptual technology evaluations that were conducted in the CHIL project, specifically those held in the CLEAR 2006 and 2007 evaluation workshops. We then summarize the main achievements and lessons learnt in the project in the areas of person tracking, person identification and head pose estimation, all of which are critical perception components in order to build perceptive smart environments.

This paper presents our work on recognizing the visual focus of attention during dynamic meeting scenarios. We collected a new dataset of meetings, in which acting participants were to follow a predefined script of events, to enforce focus shifts of the remaining, unaware meeting members. Including the whole room, all in all, a total of 35 potential focus targets were annotated, of which some were moved or introduced spontaneously during the meeting. On this dynamic dataset, we present a new approach to deduce the visual focus by means of head orientation as a first clue and show, that our system recognizes the correct visual target in over 57% of all frames, compared to 47% when mapping head pose to the first-best intersecting focus target directly.

In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.

In this work we present our recent approach on estimating head orientations and foci of attention of multiple people in a smart room, which is equipped with several cameras to monitor the room. In our approach, we estimate each person's head orientation with respect to the room coordinate system by using all camera views. We implemented a Neural Network to estimate head pose on every single camera view, a Bayes filter is then applied to integrate every estimate into one final, joint hypothesis. Using this scheme, we can track peoples' horizontal head orientations in a full 360° range at almost all positions within the room. The tracked head orientations are then used to determine who is looking at whom, i.e. people's focus of attention. We report experimental results on one meeting video, that was recorded in the smart room.