Detecting the road terrain ahead of the ego-vehicle is an important issue for modern driver assistance systems. In particular, vehicle motion planning in inner city environment requires the detection of road terrain up to 3 seconds in advance. State-of-the-art visual road terrain detection systems have a hard time fulfilling this task, due to their limited range and the presence of occlusions (other vehicles, buildings, etc.), which are expected to occur often in complex scenarios. However, those systems provide significant information where the conditions are favorable (proximity to the ego-vehicle, no occlusions). Therefore, a complementary approach is needed to enhance already existing and established detection systems. In this paper we propose a probabilistic grid-based approach based on the observation and interpretation of other vehicles' behavior in the scene. It exploits their movements in order to infer the presence and location of occluded road surface. We will show that this approach presents various advantages over current visual road terrain detection systems, especially in those situations that are the most challenging for them. We will illustrate how our approach is designed to work in concert also with other available resources, e.g. offline road maps. Qualitative results on real-world scenes taken from the KITTI benchmark[11] demonstrate that the fusion of this method with visual road terrain detection can potentially extend our time horizon well over the 3 seconds mentioned above. Finally, we will show how our approach is planned to develop into a semantically enriched representation of the road, including road properties such as availability, lanes and directions.

The robust detection of road boundaries is a prerequisite for Advanced Driver Assistant Systems (ADAS), such as Lane Departure Warning and Lane Keeping Assistant Systems. State-of-the-art ADAS rely on lane markings to draw inference about the extent of lanes or the road area. However, on many rural or urban roads markings are worn out or simply not existing. Therefore, this publication proposes a system for vision-based road boundary detection without requiring road markings at the outer side of the road. The fundamental approach is a boundary vicinity classification based on SPatial RAY (SPRAY) features which combines visual and spatial context information going beyond classical image patch analysis. More specifically, the inner road boundary vicinity (IBV), the outer road boundary vicinity (OBV), and the remaining part of the road area (RA) are detected. Because these classes occur in a defined sequence, i.e., a road boundary exhibits a transition from RA to IBV to OBV, this approach extracts the horizontal boundary transition pattern to make inference about possible locations of road boundaries and their direction (left or right road side). The implemented system was evaluated on unmarked urban and rural roads. The results show that the system effectively detects road boundaries such as curbstones and soft shoulders under challenging conditions.

For future driver assistance systems and autonomous vehicles, the road course, i.e., the width and shape of the driving path, is an important source of information. In this paper, we introduce a new hierarchical two-stage approach for learning the spatial layout of road scenes. In the first stage, base classifiers analyze the local visual properties of patches extracted from monocular camera images and provide metric confidence maps. We use classifiers for road appearance, boundary appearance, and lane-marking appearance. The core of the proposed approach is the computation of SPatial RAY (SPRAY) features from each metric confidence map in the second stage. A boosting classifier selecting discriminative SPRAY features can be trained for different types of road terrain and allows capturing the local visual properties together with their spatial layout in the scene. In this paper, the extraction of road area and ego-lane on inner-city video streams is demonstrated. In particular, the detection of the ego-lane is a challenging semantic segmentation task showing the power of SPRAY features, because on a local appearance level, the ego-lane is not distinguishable from other lanes. We have evaluated our approach operating at 20 Hz on a graphics processing unit on a publicly available data set, demonstrating the performance on a variety of road types and weather conditions.

Assigning the ego-vehicle to a lane is not only beneficial for navigation but will be an essential element in future Advanced Driver Assistance Systems. This paper describes an approach for ego-lane index estimation using only a monocular camera and no additional sensing equipment like, e.g., the typically employed GPS and Inertial Measurement Unit. Key aspect of the approach are SPatial RAY (SPRAY) features which represent the spatial layout of the road in the visual scene. The proposed method perceives a variety of local visual properties of the scene by means of base classifiers operating on patches extracted from camera images. The spatial arrangement of these local visual properties are captured using SPRAY features. With a boosting classifier trained on these features the ego-lane index is obtained. The system is evaluated on low traffic density and complementary to an object-based approach suitable for heavy traffic. In the conducted experiments, the proposed approach reaches recognition rates of 93% to 97% on individual highway images without applying any kind of temporal filtering.

Advanced Driver Assistance Systems (ADAS) are becoming more and more popular. Many of these systems though are limited to specific scenes and often detect risky situations very late so they can only mitigate accidents. These effects are mainly caused by the use of simple physical prediction methods, e.g. to estimate the time-to-contact with another vehicle. In this paper we show an ADAS that extends the functionality of physical collision warnings by additionally estimating potential risks based on implicit predictions. As an example we demonstrate the use of vehicle orientation information for classifying situations. Through this, we can in particular assess the risk of static cars, for which physical prediction does not apply, but which can nevertheless easily cause an accident if they start moving into our driving corridor. The proposed system is evaluated online in a test car and is shown to reliably detect classical risky situations as well as those involving static cars.

Many automotive systems use linear approaches to track and predict other traffic participants. While this may be appropriate on highways, linear predictions do not work properly on curved roads or lane crossings. This contribution introduces a generic way for including environmental knowledge - such as the lane trajectory ahead - to anticipate yaw rate and acceleration of other traffic participants. The anticipatory knowledge is used to improve prediction in filtering tasks. It is embedded in a Bayesian framework by introducing attractors, which modify the probabilistic propagation of state estimations. The attractors model how traffic participants typically behave, given environmental knowledge such as lane information, traffic lights, or indicator lights. We demonstrate the potential of this approach by modeling the fact that vehicles usually stay in their lane. We show that given correct context information and nonlinear traffic situations, the tracking error is considerably lower compared to conventional tracking methods. In addition, we also show that the intentions of other traffic participants may be inferred by comparing actual sensory data with anticipated probability distributions, which were generated dependent on alternative attractors.

Neuroscientific research suggests that the human brain encodes spatial information in a Bayesian-optimal way by means of distributed, neural population codes. In this paper we apply this concept to Advanced Driver Assistance Systems, introducing a grid-based population code for tracking and predicting the behavior of individual vehicles. The representation encodes a spatially distributed hidden Markov model of current and future vehicle locations and velocities. Predictive information and additional sensory information are integrated over time by means of Bayesian filters. Performance of the system is compared with a Kalman Filter in an overtaking maneuver in a simulated environment. It is shown that the grid-based approach excels Kalman-Filtering performance in several situations, where the Gaussian distribution and linear system assumptions of the Kalman filter are strongly violated. Moreover, the grid-based approach allows the flexible incorporation of additional behavioral assumptions. When the approach assumes that the tracked vehicle will stay in its lane, the probability distribution can be even more favorably focused and unexpected lane changes can be detected.

The detection of objects in the surrounding environment is a key functionality for every Advanced Driver Assistance System (ADAS). Otherwise, it is impossible to realize any kind of advanced assistance functionality as e.g. a Forward Collision Warning. Besides static obstacles that limit the drivable area, dynamic objects can even be more dangerous due to their ego-movement. Therefore, a detection and also motion estimation of moving objects in the surrounding is essential for every ADAS. In this contribution we present a novel way to detect moving objects only with 3D-measurements of the current surrounding environment that we called 3D-Warping. But we are not only able to detect but also to provide the direction and speed of moving objects in world coordinates. The method is generic and can be applied on any kind of depth sensor, as e.g. a stereo-camera, time-of-flight, and 3D-laser scanner.

In this paper a novel approach for road detection with a monocular camera is introduced. We propose a two step approach, combining a patch-based segmentation with additional boundary detection. We use Slow Feature Analysis (SFA) which leads to improved appearance descriptors for road and non-road parts on patch level. From the slow features a low order feature set is formed which is used together with color and Walsh Hadamard texture features to train a patch-based GentleBoost classifier. This allows extracting areas from the image that correspond to the road with a certain confidence. Typically the border regions between road and non-road have the highest classification error rates, because the appearance is hard to distinguish on the patch level. Therefore we propose a post-processing step with a specialized classifier applied to the boundary region of the image to improve the segmentation results. In order to evaluate the quality of road segmentation we propose an application-based quality measurement applying an inverse perspective mapping on the image to obtain a Birds Eye View (BEV). The advantage of this approach is that the important distant parts and boundaries of the road in the real world, which are only a low fraction in the perspective image, can be assessed in this metric measure significantly better than on the pixel level. In addition, we estimate the driving corridor width and boundary error, because for Advanced Driver Assistant Systems (ADAS) metric information is needed. For all evaluations in different road and weather conditions, our system shows an improved performance of the two step approach compared to the basic segmentation.

We present a flexible and scalable architecture that can learn to predict the future behavior of a vehicle in inner-city traffic. While behavior prediction studies have mainly been focusing on lane change events on highways, we apply our approach to a simple inner-city scenario: approaching a traffic light. Our system employs dynamic information about the current ego-vehicle state as well as static information about the scene, in this case position and state of nearby traffic lights.

We propose a method for vision-based scene understanding in urban traffic environments that predicts the appropriate behavior of a human driver in a given visual scene. The method relies on a decomposition of the visual scene into its constituent objects by image segmentation and uses segmentation-based features that represent both their identity and spatial properties. We show how the behavior prediction can be naturally formulated as scene categorization problem and how ground truth behavior data for learning a classifier can be automatically generated from any monocular video sequence recorded from a moving vehicle, using structure from motion techniques. We evaluate our method both quantitatively and qualitatively on the recently proposed CamVid dataset, predicting the appropriate velocity and yaw rate of the car as well as their appropriate change for both day and dusk sequences. In particular, we investigate the impact of the underlying segmentation and the number of behavior classes on the quality of these predictions.

Currently available traffic sign recognition systems typically focus on a single class of traffic sign and therefore, the algorithms are optimized to find only this specific class. To this end, a number of approaches for real time capable classification of mostly circular signs exist. Nevertheless, to simultaneously recognize a number of classes a different way has to be taken. This paper presents a real-time capable approach, which uses a two-tiered process independent of the diameter of the sign to cope with all distances. The first stage is our attention system, parameterized to find a number of different types of sign classes. The output of our attention system is a region of interest with a potential traffic sign candidate. The second stage is an array of weak classifiers similar to the idea of Viola and Jones, computing a probability value for each of the sign classes. As application area we focus on inner city and therefore, evaluate on the most important traffic sign classes of Stop and Give Way. Nevertheless, the approach can also detect Warning signs and is easily extensible to additional sign classes. The evaluation results show the reliability and mark it as first step towards an overall traffic sign recognition.

State-of-the-art advanced driver assistance systems (ADAS) typically focus on single tasks and therefore, have clearly defined functionalities. Although said ADAS functions (e.g. lane departure warning) show good performance, they lack the general ability to extract spatial relations of the environment. These spatial relations are required for scene analysis on a higher layer of abstraction, providing a new quality of scene understanding, e.g. for inner-city crash prevention when trying to detect a Stop sign violation in a complex situation. Otherwise, it will be difficult for an ADAS to deal with complex scenes and situations in a generic way. This contribution presents the novel task dependent generation of spatial representations, allowing task specific extraction of knowledge from the environment based on our biologically motivated ADAS. Additionally, the hierarchy of the approach provides advantages when dealing with heterogeneous processing modules, a large number of tasks and additional new input cues. First results show the reliability of the approach.

Newly emerging, highly complex Advanced Driver Assistance Systems (ADAS) fuse the output of various system modules (e.g., lane detection, object classification). Such knowledge fusion is realized in order to gain additional information of the environment allowing for complex system tasks as path planning, the active search for specific objects and task-specific analysis of the environment. As part of our previous work, we realized a highly generic type of such ADAS using biological principles. The present contribution offers a novel approach for the detection of curbstones and elevated roadsides in inner-city that relies on biological principles taking inspiration from the human neural signal processing. The gathered results can be fused to an ADAS in order to improve the quality of various other system percepts and allow additional system tasks.

Abstract — In this paper we propose a system architecture that extends the current state-of-the-art in computational visual attention by incorporating the biological concept of ventral attention. According to recent findings regarding the neurobiological foundations of attention, there exist two separate but interacting attention systems in the human brain: the dorsal attention system and the ventral attention system. As opposed to the well-known computational concepts of bottomup and top-down saliency, which both correspond to the dorsal attention system, the ventral attention system is sensitive to behavior-relevant stimuli that are unexpected (i.e. not top-down salient), independent of their perceptual saliency (bottom-up saliency). This results in a dynamic interplay between topdown saliency, bottom-up saliency and ventral attention in the proposed system architecture, enabling the system to redirect its focus of attention to important stimuli while being absorbed in a task, even if their perceptual saliency is low. Our technical system instance implementing the proposed architecture integrates several state-of-the-art methods in a coherent system and concentrates on unexpected motion as a first technical account of ventral attention. In our experiments, we demonstrate that the ventral attention enables our system to detect and reorient to important situations in real-world traffic environments that are relevant for the behavior of driving. I.

Creating robots able to interact and cooperate with humans in household environments and everyday life is an emerging topic. Our goal is to facilitate a human-like and intuitive interaction with such robots. Besides verbal interaction, gestures are a fundamental aspect in human-human interaction. One typical usage of interactive gestures is referencing of objects. This paper describes a novel integrated vision system combining different algorithms for pose tracking, gesture detection, and object attention in order to enable a mobile robot to resolve gesture-based object references. Results from the evaluation of the individual algorithms as well as the overall system are presented. A total of 20 minutes of video data collected from four subjects performing almost 500 gestures are evaluated to demonstrate the current performance of the approach as well as the overall success rate of gestural object references. This demonstrates that our integrated vision system can serve as the gestural front end that enables an interactive mobile robot to engage in multimodal human-robot interaction.

This paper puts forward an approach for a mobile robot to recognize the human's manipulative actions from different single camera views. While most of the related work in action recognition assume a fixed static camera view that is the same for training and testing, such kind of constraints do not apply for mobile robot companions. We propose a recognition scheme that is able to generalize an action model, that has been learned from a very few data items observed from a single camera view, to variant view points and different settings. We tackle the problem of compensating the view dependence of 2D motion models on three different levels. Firstly, we pre-segment the trajectories based on an object vicinity that depends on the camera tilt and object detections. Secondly, an interactive feature vector is designed that represents the relative movements between the human hand and the objects. Thirdly, we propose an adaptive HMM-based matching process that is based on a particle filter and includes a dynamically adjusted scaling parameter that models the systematic error of the view dependency. Finally, we use a two-layered approach for task recognition which decouples the task knowledge from the view dependent primitive recognition. The results of experiments in an office environment show the applicability of this approach.

A very important aspect in developing robots capable of human-robot interaction (HRI) is natural, human-like communication. Besides a flexible dialog system and speech understanding an anthropomorphic appearance has many advantages for intuitive usage and understanding of a robot. As a consequence of our effort in creating an anthropomorphic appearance and to come as close as possible to a human-human interaction, we decided to use human-like sensors, i.e., two cameras and two microphones only, not using a laser range finder or omnidirectional camera for tracking persons. Despite the challenge of a limited field of perception, a robust attention system for tracking and interacting with multiple persons simultaneously in real-time was created. Our approach is sufficiently generic to work on robots with varying hardware, as long as stereo audio data and images of a video camera are available. Since the architecture is designed modular with a XML based data exchange we are able to extend the robot's abilities easily

With the increasing interest in computer vision for interactive systems, the challenges of the development process involving many researchers are becoming more prominent. Issues like reuse of algorithms, modularity, and distributed processing are getting more important in the endeavor of building complex vision systems. We present a framework that allows independent development of enclosed components and supports interactive optimization of algorithmic parameters in an online fashion. The communication between components is performed nearly without any slow down compared to a monolithic system. Through the modular concept, all components can be flexibly distributed and reused in other application domains. The suitability of the approach is demonstrated with an example system.

An ambitious goal in modern robotic science is to build mobile robots that are able to interact as companions in real world environments. Especially for caretaking of elderly people a system robustly working at private homes is essential, requiring a very natural and human oriented way of communication. Since home environments are usually very individual a first task for a newly acquired robot is to get familiar with its new environment. This paper gives a short overview on how we integrated a vision based localization using the advantages of a very modular architecture and extending a spoken dialog system for online labeling and interaction about different locations. We present results from the integrated system working in a real, fully furnished home environment where it was able to learn the names of different rooms. This system enables us to perform real user studies in future without the need to fall back to Wizard-of-Oz experiments. Ongoing work aims at enabling the robot to take initiative by asking for unknown locations. A future extension is the ability to generalize over features of known rooms to make predictions when encountering unknown rooms

One ambitious goal in current robotics research is to build robots that can interact with humans in an intuitive way and can do so outside the lab in real world situations and environments such as private homes or public places. Toy robots like, e.g., Sony's AIBO are already being sold successfully for entertainment purposes, but they usually lack sophisticated human-like interaction capabilities preventing non-expert users to instruct them for useful tasks. We have developed a robot that is capable of processing multi-modal instructions and can, therefore, be instructed interactively in a social situation. This paper gives an overview of the components of the system and their integration. The system performance is described in detail based on observations from human-robot interactions and processing times to identify critical system components and further research directions. Finally, we report on first human-robot interactions with our robot BIRON (Bielefeld Robot companiON) being situated in a real home environment. These interactions demonstrate that our robot is able to interact with users in a real home environment and can thus serve as a basis for comprehensive user studies focussing on embodied interaction for social learning

Creating robots with extendable social skills and interaction capabilities that suffice their operation in the real world with naive users is a very challenging task. In this paper we present a new approach using topic tracking on multi-modal dialogue to provide a mobile robot with a higher level situation awareness in human-robot interaction. The robot is no longer operating in laboratory surroundings, but in its own real world flat. We describe how our topic tracking approach is implemented in this integrated system, operating on verbal speech input. Different modalities like data from video cameras and laser scans are used as additional cues to a semantic understanding and grouping of user utterances into different topics. Both the amount of topics and the according topic names are created dynamically. Evaluating an offline speech corpus demonstrates the suitability of our approach. It is now possible to ask "BIRON, what's the topic?", making the interaction more social

In research on human-robot interaction the interest is currently shifting from uni-modal dialog systems to multi-modal interaction schemes. We present a system for human-style interaction with a robot that is integrated on our mobile robot BIRON. To model the dialog we adopt an extended grounding concept with a mechanism to handle multi-modal in- and output where object references are resolved by the interaction with an object attention system (OAS). The OAS integrates multiple input from, e.g., the object and gesture recognition systems and provides the information for a common representation. This representation can be accessed by both modules and combines symbolic verbal attributes with sensor-based features. We argue that such a representation is necessary to achieve a robust and efficient information processing.

The development of robot companions with nat ural human-robot interaction (HRI) capabilities is a challeng ing task as it requires incorporating various functionalities. Consequently, a flexible infrastructure for controlling module operation and data exchange between modules is proposed, taking into account insights from software system integration. This is achieved by combining a three-layer control architec ture containing a flexible control component with a powerful communication framework. The use of XML throughout the whole infrastructure facilitates ongoing evolutionary develop ment of the robot companion's capabilities.

The communication with robotic assistants or companions is a challenging new domain for the use of dialog systems. In contrast to classical spoken language interfaces users interact with mobile robots mostly in a multi-modal way. In this paper we will present the integration of several modalities in the dialog system of BIRON — the Bielefeld Robot Companion. Besides speech as the main modality the system integrates deictic gestures and visual scene information in order to resolve object references in a task oriented dialog. We will present example interactions with BIRON and first qualitative results from the "home-tour" scenario defined within the COGNIRON project.

Robot companions are intended for operation in private homes with naive users. For this purpose, they need to be endowed with natural interaction capabilities. Additionally, such robots will need to be taught unknown objects that are present in private homes. We present a multi-modal object attention system that is able to identify objects referenced by the user with gestures and verbal instructions. The proposed system can detect known and unknown objects and stores newly acquired object information in a scene model for later retrieval. This way, the growing knowledge base of the robot companion improves the interaction quality as the robot can more easily focus its attention on objects it has been taught previously.

This paper presents the humanoid robot BARTHOC who has been developed to study human-robot interaction (HRI). The main focus of BARTHOC's design was to realize the expression and behavior of the robot to be as human-like as possible. This allows to apply the platform to manifold research and demonstration areas. With his human-like look and mimic possibilities, he differs from other platforms like ASIMO or QRIO, and enables experiments even close to Mori's 'uncanny valley'. The paper describes details of the mechanical and electrical design of BARTHOC together with its PC control interface. Through its humanoid appearance, it can imitate human behavior with its soft- and hardware. Currently, several components for HRI on a mobile robot platform are being ported to BARTHOC. Starting with these components, the robot's human-like appearance enables us to study embodied interaction and to explore theories of human intelligence.

Natural human-robot interaction (HRI) is a key feature of mobile robot companions collaborating with humans. To achieve natural HRI, multiple communication modalities like vision, speech, and gestures have to be utilized. Besides, capabilities to emulate cognitive processes, e.g., object learning and object recognition, are essential. In this work we present a new approach to interactive object learning enabling multi-view object representation. To overcome a robot's limitation of having only one view point, we make use of an iconic memory consisting of previously acquired images. As the relevant scene area is unknown during construction of the iconic memory, a representation in the form of mosaic images is applied. The relevant image patches describing an object referenced by the user are selected through an object attention mechanism. The resulting multi-view object representations improve the flexibility of our interactive approach for object learning.

Building a mobile service robot for home and office environments that incorporates skilled interaction capabilities is a challenging task. The control system has to consider various demands: first, it has to manage unstructured and dynamic environments. Second, as humans are around, aspects of safety are of particular importance. This implies that the system has to be highly reactive. Third, the robot also has to be capable of carrying out dialogs to be taught or instructed. Altogether, this requires a highly integrated control framework. In this paper we present an agent-based architecture for our mobile robot BIRON in order to realize sophisticated human-robot interaction. The architecture is built in a modular fashion and controlled by a central execution supervisor using an event queue to handle asynchronous events. This execution supervisor contains an augmented finite state machine which is specified in XML and thus is highly generic. Similarly, the communication between all modules is based on XML. The overall system, therefore, is easily maintainable and extensible, as the architecture's design allows us to add new modules to the system without requiring major modifications on existing components.

In order to enable the widespread use of robots in home and office environments, systems with natural interaction capabilities have to be developed. A prerequisite for natural interaction is the robot's ability to automatically recognize when and how long a person's attention is directed towards it for communication. As in open environments several persons can be present simultaneously, the detection of the communication partner is of particular importance. In this paper we present an attention system for a mobile robot which enables the robot to shift its attention to the person of interest and to maintain attention during interaction. Our approach is based on a method for multi-modal person tracking which uses a pan-tilt camera for face recognition, two microphones for sound source localization, and a laser range finder for leg detection. Shifting of attention is realized by turning the camera into the direction of the person which is currently speaking. From the orientation of the head it is decided whether the speaker addresses the robot. The performance of the proposed approach is demonstrated with an evaluation. In addition, qualitative results from the performance of the robot at the exhibition part of the ICVS'03 are provided.

The capability to coordinate and interrelate speech and vision is a virtual prerequisite for adaptive, cooperative, and flexible interaction among people. It is therefore fair to assume that human-machine interaction, too, would benefit from intelligent interfaces for integrated speech and image processing. We first sketch an interactive system that integrates automatic speech processing with image understanding. Then, we concentrate on performance assessment which we believe is an emerging key issue in multimodal interaction. We explain the benefit of time scale analysis and usability studies and evaluate our system accordingly.

A major challenge for the realization of intelligent robots is to supply them with cognitive abilities in order to allow ordinary users to program them easily and intuitively. One approach to such programming is teaching work tasks by interactive demonstration. To make this effective and convenient for the user, the machine must be capable of establishing a common focus of attention and be able to use and integrate spoken instructions, visual perception, and non-verbal clues like gestural commands. We report progress in building a hybrid architecture that combines statistical methods, neural networks, and finite state machines into an integrated system for instructing grasping tasks by man-machine interaction. The system combines the GRAVIS-robot for visual attention and gestural instruction with an intelligent interface for speech recognition and linguistic interpretation, and a modality fusion module to allow multi-modal task-oriented man-machine communication with respect to dextrous robot manipulation of objects.

We present a fast and reliable method to analyse an image sequence where two human hands perform assembly actions. Our classifier to detect the initial skin coloured regions is based on a polynomial classifier of sixth degree. For each region a judgement is calculated using two cues. The first cue is motion information obtained from a difference image. The second cue is obtained from confidence mapping performed on the output of the polynomial classifier. All resulting regions with a high judgement indicating movement are tracked using Kalman filters. Based on the trajectories of the Kalman filters action hypotheses are generated. The hypotheses are verified through a detailed analysis using a Fourier transformation on the derivatives of the trajectories.