In this work, we study $\gamma$-discounted infinite-horizon tabular Markov decision processes (MDPs) and introduce a framework called dynamic policy gradient (DynPG). The framework directly integrates dynamic programming with (any) policy gradient method, explicitly leveraging the Markovian property of the environment. DynPG dynamically adjusts the problem horizon during training, decomposing the original infinite-horizon MDP into a sequence of contextual bandit problems. By iteratively solving these contextual bandits, DynPG converges to the stationary optimal policy of the infinite-horizon MDP. To demonstrate the power of DynPG, we establish its non-asymptotic global convergence rate under the tabular softmax parametrization, focusing on the dependencies on salient but essential parameters of the MDP. By combining classical arguments from dynamic programming with more recent convergence arguments of policy gradient schemes, we prove that softmax DynPG scales polynomially in the effective horizon $(1-\gamma)^{-1}$. Our findings contrast recent exponential lower bound examples for vanilla policy gradient.

As an important physical field feature, magnetic anomaly is widely used in the detection of marine ferromagnetic targets. Influenced by the complex ocean measurement environment, the collected magnetic data are usually contaminated with noise, and traditional magnetic anomaly detection methods are less versatile due to the specific conditions required. Faced with the challenging situations in the ocean, the use of deep learning methods can not only automatically extract discriminative features from the magnetic data, but also reduce manual interference in the detection model design. Deep learning requires a large amount of data to train the model, but the real target magnetic anomaly data is expensive to acquire and small in number. In this letter, we propose a multi-task deep transfer learning model for simultaneous denoising and detection of marine target magnetic anomaly data under limited labelled samples. Specifically, a convolutional denoising auto-encoder (CDAE) network is designed for adaptive background noise modeling and discriminative feature learning. Meanwhile, a fully connected classification (FCC) network is cascaded and jointly trained with it for simultaneous noise filtering and anomaly detection. To guarantee sufficient learning effect, both real measured data from sea trials and simulated data from magnetic dipole model are used for training the network in a two-stage manner. During the experiments, the learned multi-task deep model is respectively used to denoise the marine magnetic data and detect target anomaly signal. It achieves both superior noise-removal and anomaly discovery performance than the traditional methods, and meanwhile is much more efficient.

Direct policy search serves as one of the workhorses in modern reinforcement learning (RL), and its applications in continuous control tasks have recently attracted increasing attention. In this work, we investigate the convergence theory of policy gradient (PG) methods for learning the linear risk-sensitive and robust controller. In particular, we develop PG methods that can be implemented in a derivative-free fashion by sampling system trajectories, and establish both global convergence and sample complexity results in the solutions of two fundamental settings in risk-sensitive and robust control: the finite-horizon linear exponential quadratic Gaussian, and the finite-horizon linear-quadratic disturbance attenuation problems. As a by-product, our results also provide the first sample complexity for the global convergence of PG methods on solving zero-sum linear-quadratic dynamic games, a nonconvex-nonconcave minimax optimization problem that serves as a baseline setting in multi-agent reinforcement learning (MARL) with continuous spaces. One feature of our algorithms is that during the learning phase, a certain level of robustness/risk-sensitivity of the controller is preserved, which we termed as the implicit regularization property, and is an essential requirement in safety-critical control systems.

Making decisions in the presence of a strategic opponent requires one to take into account the opponent’s ability to actively mask its intended objective. To describe such strategic situations, we introduce the non-cooperative inverse reinforcement learning (N-CIRL) formalism. The N-CIRL formalism consists of two agents with completely misaligned objectives, where only one of the agents knows the true objective function. Formally, we model the N-CIRL formalism as a zero-sum Markov game with one-sided incomplete information. Through interacting with the more informed player, the less informed player attempts to both infer and optimize the true objective function. As a result of the one-sided incomplete information, the multi-stage game can be decomposed into a sequence of single- stage games expressed by a recursive formula. Solving this recursive formula yields the value of the N-CIRL game and the more informed player’s equilibrium strategy. Another recursive formula, constructed by forming an auxiliary game, termed the dual game, yields the less informed player’s strategy. Building upon these two recursive formulas, we develop a computationally tractable algorithm to approximately solve for the equilibrium strategies. Finally, we demonstrate the benefits of our N-CIRL formalism over the existing multi-agent IRL formalism via extensive numerical simulation in a novel cyber security setting.

In this paper, we confirm the hypothesis that the magnetic field inside a building can vary significantly as a function of height. We collected data with twenty magnetometers spaced evenly from knee height to head height and mounted to a ground robot, which we drove through two different buildings on the campus of the University of Illinois at Urbana-Champaign. We applied Gaussian process regression to build a map of the magnetic field at each height. We compared these maps and saw pairwise differences of more than 1 μT in up to 20% of each test environment, a threshold that we argue would prevent meeting the requirements of common indoor positioning applications. These results call into question the planar assumption that is commonly made when deriving methods of indoor positioning that are based on the use of magnetic fields.