We show that several popular discounted reward natural actor-critics, including the popular NAC-LSTD and eNAC algorithms, do not generate unbiased estimates of the natural policy gradient as claimed. We derive the first unbiased discounted reward natural actor-critics using batch and iterative approaches to gradient estimation. We argue that the bias makes the existing algorithms more appropriate for the average reward setting. We also show that, when Sarsa(lambda) is guaranteed to converge to an optimal policy, the objective function used by natural actor-critics is concave, so policy gradient methods are guaranteed to converge to globally optimal policies as well.

Natural gradient ascent (NGA) is a popular optimization method that uses a positive definite metric tensor. In many applications the metric tensor is only guaranteed to be positive semidefinite (e.g., when using the Fisher information matrix as the metric tensor), in which case NGA is not applicable. In our first contribution, we derive generalized natural gradient ascent (GeNGA), a generalization of NGA which allows for positive semidefinite non-smooth metric tensors. In our second contribution we show that, in standard settings, GeNGA and NGA can both be divergent. We then establish sufficient conditions to ensure that both achieve various forms of convergence. In our third contribution we show how several reinforcement learning methods that use NGA without positive definite metric tensors can be adapted to properly use GeNGA.

Quantification of DNA damage, which may be caused by radiation or exposure to chemicals, is very important and can be very time consuming and subject to variability if carried out visually. The quantification of scoring DNA damage includes biomarkers such as micronuclei, nucleoplasmic bridges, and nuclear buds as scored in cytokinesis-blocked binucleated cells. In this paper, we present a new algorithm based on a shortest path technique that enables us to detect the nucleoplasmic bridges joining two nuclei in cell images of binucleated cells. The effectiveness of our algorithm is illustrated using a set of cell images. We believe that this is the first time that a feasible automated nucleoplasmic bridge detection system has been reported.