Using stochastic gradient search and the optimal filter derivative, it is possible to perform recursive maximum likelihood estimation in a non-linear state-space model. As the optimal filter and its derivative are analytically intractable for such a model, they need to be approximated numerically. In Poyiadjis et al. (G. Poyiadjis, A. Doucet, and S. S. Singh, Biometrika, vol. 98, no. 1, pp. 65-80, 2011), a recursive maximum likelihood algorithm based on a particle approximation to the optimal filter derivative has been proposed and studied through numerical simulations. This algorithm and its asymptotic behavior are here analyzed theoretically. Under regularity conditions, we show that the algorithm accurately estimates maxima of the underlying log-likelihood rate when the number of particles is sufficiently large. We also provide qualitative upper bounds on the estimation error in terms of the number of particles.

Using stochastic gradient search and the optimal filter derivative, it is possible to perform recursive (i.e., online) maximum likelihood estimation in a non-linear state-space model. As the optimal filter and its derivative are analytically intractable for such a model, they need to be approximated numerically. In [17], a recursive maximum likelihood algorithm based on a particle approximation to the optimal filter derivative has been proposed and studied through numerical simulations. Here, this algorithm and its asymptotic behavior are analyzed theoretically.

The analyticity of the entropy and relative entropy rates of continuous-state hidden Markov models is studied here. Using the analytic continuation principle and the stability properties of the optimal filter, the analyticity of these rates is established for analytically parameterized models. The obtained results hold under relatively mild conditions and cover several useful classes of hidden Markov models. These results are relevant for several theoretically and practically important problems arising in statistical inference, system identification and information theory.

In this paper, the problem of the optimal quantization of a signal generated by a hidden Markov model is considered. For this problem, an efficient algorithm based on Monte Carlo sampling, gradient estimation techniques and stochastic approximation is proposed. The properties of the proposed algorithm are analyzed both theoretically and through simulations.

Particle methods are a set of powerful and versatile simulation-based methods to perform optimal state estimation in nonlinear non-Gaussian state-space models. The ability to compute the optimal filter is central to solving important problems in areas such as change detection, parameter estimation, and control. Much recent work has been done in these areas. The objective of this paper is to provide a detailed overview of them.

The paper addresses the optimisation of particle filtering methods aka sequential Monte Carlo (SMC) methods using stochastic approximation. First, the SMC algorithm is parameterised smoothly by a parameter. Second, optimisation of an average cost function is performed using simultaneous perturbation stochastic approximation (SPSA). Simulations demonstrate the efficiency of our algorithm.

In the discrete-time envelope-constrained filtering problem, the gain of the filter is minimised subject to the constraint that the filter output to a prescribed input fits into a given envelope. A novel adaptive algorithm for solving this problem based on stochastic optimisation is presented. The algorithm is simple to implement on-line and convergence is demonstrated in numerical examples. Under mild regularity assumptions, convergence follows from standard stochastic approximation results.

Classical methods for solving a semi-Markov decision process such as value iteration and policy iteration require precise knowledge of the underlying probabilistic model and are know to suffer from the curse of dimensionality. To overcome both these limitations, this paper presents a reinforcement learning approach where one optimizes directly the performance criterion with respect to a family of parameterised policies. We propose an online algorithm that simultaneously estimates the gradient of the performance criterion and optimises it through stochastic approximation. The gradient estimator is based on the discounted score method as introduced. We demonstrate the utility of our algorithm in a Call Admission Control problem.