This article addresses the two key challenges in computer-assisted percutaneous tumor ablation: planning multiple overlapping ablations for large tumors while avoiding critical structures, and executing the prescribed plan. Toward semiautomatic treatment planning for image-guided surgical interventions, we develop a systematic approach to the needle-based ablation placement task, ranging from preoperative planning algorithms to an intraoperative execution platform. The planning system incorporates clinical constraints on ablations and trajectories using a multiple objective optimization formulation, which consists of optimal path selection and ablation coverage optimization based on integer programming. The system implementation is presented and validated in both phantom and animal studies. The presented system can potentially be further extended for other ablation techniques such as cryotherapy.

We introduce a set of algorithms for pricing calls on a multiclass loss network with unknown demand elasticity. The algorithms are designed to observe the network and use real-time pricing to estimate demand elasticity and other unknown system parameters, and modify per-class prices in order to improve the long-run average revenue. The algorithms can be implemented online, have small memory and computational requirements, and are robust to parametric uncertainty. We provide sufficient conditions for the convergence of the algorithms to a local optimum, and illustrate their performance through numerous numerical examples. The paper also discusses how these algorithms can be distributed to multiple agents on a per-class basis, and provide bounds to error estimates introduced by our decoupling approach.

We consider simulation-based gradient-estimation and its use in Markov controlled processes with unknown parameters. We consider a Markov reward process controlled by both a set of tunable parameters, and a set of fixed but unknown. We analyze the use a recursive identification procedure, and their application to existing gradient-based algorithms based on simulation. We show that simple modifications of available gradient estimation algorithms, namely assuming parameter certainty, can accommodate system parameter identification, without sacrificing the convergence of these to local optima by following a two-time-scale recursive identification/optimization procedure. This approach is illustrated through an application to the algorithm proposed in (Marbach and Tsitsiklis, 2001). We illustrate our results with a small numerical example, which we further use to test the ability of the proposed scheme to track slow changing system parameters. INTRODUCTION Many stochastic resource allocation and control problems can be naturally modeled using dynamic programming. Unfortunately, the dynamic programming algorithm suffers from the ‘curse of dimensionality’. In order to address these performance problems, researchers have recurred to the use of parameterized control policies, as in (Marbach and Tsitsiklis, 2001), reducing the problem to one of search in policy space. Several simulationbased techniques exist (Marbach and Tsitsiklis, 2001, 2003; Cao and Chen, 1997; Cao and Wan, 1998; Fang and Cao, 2004; Baxter and Bartlett, 2001) that use realizations of the stochastic processes to obtain gradient estimates that can be used in a gradient-based search method in policy space. Many of the algorithms mentioned above are ‘modelfree’ and, hence, implicitly adaptive, e.g., general stochastic approximations (Kiefer and Wolfowitz, 1952; Kushner and Yin, 1997), but they exhibit slow convergence due to biased gradient estimates, or large variances. This problem can be alleviated by incorporating knowledge of the system model, for example by exploiting regenerative structure to eliminate bias, but by requiring knowledge of all system parameters, the usefulness of such techniques is limited. Although the problem of adaptive control of Markov chains has been studied before, most of authors address the case of a finite number of possible models, as in (Mandl, 1974; Borkar and Varaiya, 1979; Kumar and Becker, 1982; Doshi, 1980; El-Fattah, 1981). Other authors, overcome this restriction, but assume that the optimal policies for each model can be easily computed, such as in (Borkar and Varaiya, 1982). Others, embed the estimation process in a value iteration procedure (Hernandez-Lerma, 1989), which is not efficient computationally when the state spaces are large. The work in (Ren and Krogh, 2001; Santharam and Sastry, 1997) estimates Q-factors, a technique that is adaptive since no system model is in principle needed, but requires the action set to be finite in order to build a Q-factor for each state-action pair, and quickly becomes impractical as the state space increases. In some specific cases, the Q-factors can be approximated via a neural network or other architectures (Bertsekas and Tsitsiklis, 1996), with limited success. In this paper we consider the problem of adaptive optimization of parameterized Markov reward processes. We first study the problem of online estimation and optimization of the average reward criterion in Markov reward processes for which the transition probabilities, as well as the expected reward per stage, are functions of two sets of parameters: 1) a set of tunable controls, and 2) a fixed but unknown parameter. We study estimation procedures that can be updated after each transition in the chain, generalizing some of the results of (Campos-Nanez and Patek, 2005), apply them to existing simulation-based algorithms, such as the algorithm of (Marbach and Tsitsiklis, 2001), providing sufficient conditions for the convergence of the resulting method to a local optimum. The paper is organized as follows. We first present the problem of adaptive Markov control, and address the problem of estimation of an unknown parameter. We follow this with a discussion about the use of estimators in Proceedings 23rd European Conference on Modelling and Simulation ©ECMS Javier Otamendi, Andrzej Bargiela, Jose Luis Montes, Luis Miguel Doncel Pedrera (Editors) ISBN: 978-0-9553018-8-9 / ISBN: 978-0-9553018-9-6 (CD) optimization algorithms, and illustrate our approach using an algorithm developed in (Marbach and Tsitsiklis, 2001). We provide sufficient conditions for the convergence of the adaptive scheme to local optima, while identifying the true system parameter values. The resulting adaptive algorithm has small memory and computational requirements, and can hence be implemented in an fashion. Its performance is discussed in the last section of the paper, where its usability under slow changing parameters is explored. MARKOV REWARD PROCESSES WITH UNKNOWN PARAMETERS Let {in}n be a discrete-time Markov chain with finite state space S = {1,2, . . . ,N}. There is a set of ‘tunable’ parameters u ∈ R , and a fixed but unknown parameter θ ∗ ∈Θ, a convex, compact subset of R. These two sets of parameters determine the dynamics of the Markov chain in the sense that the transition probabilities pi j(u,θ ∗) = P(in+1 = j | in = i,u,θ ∗), are functions of the control vector u, and the parameter θ ∗. We define P(u,θ ∗) to be the transition probability matrix with entries pi j(u,θ ∗). The reward observed by this system is also a function of parameters u, and θ ∗, i.e., the expected reward per stage when the system is at state i, and controls u are applied is a function of u, θ ∗, denoted by gi(u,θ ∗). We focus our attention to the average reward criterion, which can be defined as λ (u,θ ∗) = liminf T→∞ 1 T E [ T ∑ n=0 gin(u,θ ∗) ∣

In this paper, we consider a market in which a finite number of firms compete in prices for the incoming demand for service. Upon every customer arrival, an independent auctioneer gathers bids from each one of the competing queuing systems and assigns the incoming customer to the system that submitted the lowest bid. We provide a simple characterization of Markov Perfect equilibrium in terms of ldquoindifference prices,rdquo i.e., price levels at which players are indifferent between committing available capacity or withholding it. We identify sufficient conditions for socially efficient performance in equilibrium.

In this paper, we outline the basis for a new distributed Bayesian search scheme in which all Bayesian decision makers recognize the same performance objective but do not possess the ability to communicate with each other. Coordination among the players is achieved indirectly by tracking search performance. This scheme is ideal for unmanned vehicles searching in extreme environments where extensive bilateral communication between actuators is not feasible. Preliminary results suggest our new search scheme is scalable and can easily be adapted for tracking moving targets.

Recent investigations into the pricing of multiclass loss networks have shown that static prices are optimal in the asymptotic regime of many small sources. These results suggest that nearly optimal prices for highly aggregated systems can be computed from the solution to a limiting deterministic optimization model. When the assumption of many small sources does not hold, static prices are still preferable (for practical reasons), but we are left with the difficult issue of computing an optimal solution when the stochastic nature of the process cannot be ignored. In this paper, we develop a computational procedure for optimizing static prices that operates by adjusting prices in response to actual customer arrivals and departures and is robust to parametric uncertainty about the underlying system. We provide initial arguments for the convergence properties of our optimization algorithm, and we illustrate its application in several numerical examples.