We propose a novel reinforcement learning methodology where the system performance is evaluated by a Markov coherent dynamic risk measure with the use of linear value function approximations. We construct projected risk-averse dynamic programming equations and study their properties. We propose new risk-averse counterparts of the basic and multi-step methods of temporal differences and we prove their convergence with probability one. We also perform an empirical study on a complex transportation problem.

The problem of adaptive routing in a network with failures is considered. The network may be in one of finitely many states characterized by different travel times along the arcs, and transitions between the states occur according to a continuous-time Markov chain. The objective was to develop a routing strategy that minimizes the total expected travel time. Dynamic programming models and flow-oriented models were developed and analyzed in the uncapacitated and the capacitated case. It is shown that the robust plan can be found from a special two-stage stochastic programming problem in which the second-stage models the rerouting problem after the state transition in the network. The models are illustrated on an example of the Sioux Falls transportation network. The computational results reveal striking properties of different routing policies and show that substantial improvements in both duration and size of jams can be achieved by employing robust strategies. © 2000 John Wiley & Sons, Inc.