This paper investigates the development of self-aware mechanisms for automated vehicles, introducing the notion of an automation state estimation system. This system is capable to understand its capabilities in a given context, and can leverage that knowledge to estimate the current and near-future automation performance based on internal metrics, as well as external, static (e.g. lane geometry) and dynamic environmental elements (e.g. traffic and weather information). From an application perspective, we consider automation state estimation in the scope of automation mediation, as part of a broader and holistic mediation system, with the goal to tackle challenging aspects related to transitions of control, mode confusion, and driver engagement. We used real-world data for system design, and implemented the proposed automation estimation system in a prototype vehicle. Based on 70 hours of real-world driving, we also validated the performance of the automation state estimation for automation mediation purposes.

Driver fatigue is a contributing factor in about 20% of all fatal road crashes worldwide. Countermeasures are urgently needed and one of the most promising and currently available approaches for that are in-vehicle systems for driver fatigue detection. The main objective of this paper is to present a video-based driver sleepiness detection system set up as a two-stage model with (1) a generic deep feature extraction module combined with (2) a personalised sleepiness detection module. The approach was designed and evaluated using data from 13 drivers, collected during naturalistic driving conditions on a motorway in Sweden. Each driver performed one 90-minute driving session during daytime (low sleepiness condition) and one session during night-time (high sleepiness condition). The sleepiness detection model outputs a continuous output representing the Karolinska Sleepiness Scale (KSS) scale from 1–9 or a binary decision as alert (defined as KSS 1–6) or sleepy (defined as KSS 7–9). Continuous output modelling resulted in a mean absolute error (MAE) of 0.54 KSS units. Binary classification of alert or sleepy showed an accuracy of 92% (sensitivity = 91.7%, specificity = 92.3%, F1 score = 90.4%). Without personalisation, the corresponding accuracy was 72%, while a standard fatigue detection PERCLOS-based baseline method reached an accuracy of 68% on the same dataset. The developed real-time sleepiness detection model can be used in the management of sleepiness/fatigue by detecting precursors of severe fatigue, and ultimately reduce sleepiness-related road crashes by alerting drivers before high levels of fatigue are reached.

This paper describes quasi-online reinforcement learning: while a robot is exploring its environment, in the background a probabilistic model of the environment is built on the fly as new experiences arrive; the policy is trained concurrently based on this model using an anytime algorithm. Prioritized sweeping, directed exploration, and transformed reward functions provide additional speed-ups. The robot quickly learns goal-directed policies from scratch, requiring few interactions with the environment and making efficient use of available computation time. From an outside perspective it learns the behavior online and in real time. We describe comparisons with standard methods and show the individual utility of each of the proposed techniques

Mobile robot localization and navigation requires a map - the robot's internal representation of the environment. A common problem is that path planning becomes very inefficient for large maps. In this paper we address the problem of segmenting a base-level map in order to construct a higher-level representation of the space which can be used for more efficient planning. We represent the base-level map as a graph for both geometric and appearance based space representations. Then we use a graph partitioning method to cluster nodes of the base-level map and in this way construct a high-level map, which is also a graph. We apply a hierarchical path planning method for stochastic tasks based on Markov decision processes (MDPs) and investigate the effect of choosing different numbers of clusters

This paper addresses the question how robot planning (e.g. for navigation) can be done with hierarchical maps. We present an algorithm for hierarchical path planning for stochastic tasks, based on Markov decision processes (MDPs) and dynamic programming. It is more efficient than standard dynamic programming for "flat" MDPs, because it reduces the state space for all levels in its hierarchy and it allows reuse of previously computed partial policies. This computational advantage comes at the cost of some extra memory and overhead to represent and coordinate the hierarchical system, and in some cases somewhat longer paths to target locations. We demonstrate the method on artificially generated MDP data, and on real robot data from our vision-controlled robot navigating in an office environment.

This paper addresses the problem of automatic construction of a hierarchical map from images. Our approach departs from a large collection of omnidirectional images taken at many locations in a building. First, a low-level map is built that consists of a graph in which relations between images are represented. For this, we use a metric based on visual landmarks (SIFT features) and geometrical constraints. Then, we use a graph partitioning method to cluster nodes and in this way construct the high-level map. Experiments on real data show that meaningful higher and lower level maps are obtained, which can be used for accurate localization and planning.

It is difficult to apply traditional reinforcement learning algorithms to robots, due to problems with large and continuous domains, partial observability, and limited numbers of learning experiences. This paper deals with these problems by combining: (1) reinforcement learning with memory, implemented using an LSTM recurrent neural network whose inputs are discrete events extracted from raw inputs; (2) online exploration and offline policy learning. An experiment with a real robot demonstrates the methodology's feasibility.

This paper describes how learning tasks in partially observable mobile robot domains can be solved by combining reinforcement learning with an unsupervised learning "event extraction" mechanism, called ARAVQ. ARAVQ transforms the robot's continuous, noisy, high-dimensional sensory input stream into a compact sequence of high-level events. The resulting hierarchical control system uses an LSTM recurrent neural network as the reinforcement learning component, which learns high-level actions in response to the history of high-level events. The high-level actions select low-level behaviors which take care of the real-time motor control. Illustrative experiments based on the Khepera mobile robot simulator are presented.

This paper presents reinforcement learning with a Long Short(cid:173) Term Memory recurrent neural network: RL-LSTM. Model-free RL-LSTM using Advantage(,x) learning and directed exploration can solve non-Markovian tasks with long-term dependencies be(cid:173) tween relevant events. This is demonstrated in a T-maze task, as well as in a difficult variation of the pole balancing task.

A Markovian reinforcement learning task can be dealt with by learning a direct mapping from states to actions or values, or from state-action pairs to values. However, this may involve a difficult pattern recognition problem when the state space is large. This paper shows that using internal state, called "supportive state", may alleviate this problem presenting an argument against the tendency to almost automatically use a direct mapping when the task is Markovian. This point is demonstrated in simulation experiments of an agent controlled by a neural network capable of learning the strategy of direct mapping as well as internal state, combining Q(/spl lambda/) learning and recurrent neural networks in a new way. The trade-off between the two strategies is investigated in more detail, focusing particularly on border cases.