In this paper, we investigate the concentration properties of cumulative reward in Markov Decision Processes (MDPs), focusing on both asymptotic and non-asymptotic settings. We introduce a unified approach to characterize reward concentration in MDPs, covering both infinite-horizon settings (i.e., average and discounted reward frameworks) and finite-horizon setting. Our asymptotic results include the law of large numbers, the central limit theorem, and the law of iterated logarithms, while our non-asymptotic bounds include Azuma-Hoeffding-type inequalities and a non-asymptotic version of the law of iterated logarithms. Additionally, we explore two key implications of our results. First, we analyze the sample path behavior of the difference in rewards between any two stationary policies. Second, we show that two alternative definitions of regret for learning policies proposed in the literature are rate-equivalent. Our proof techniques rely on a martingale decomposition of cumulative reward, properties of the solution to the policy evaluation fixed-point equation, and both asymptotic and non-asymptotic concentration results for martingale difference sequences.

Systems and Control is now a mature subject with a deep mathematical theory and a powerful engineering methodology; its roots lie in antiquity but it emerged in the form of the Watt governor more than two centuries ago as an enabling technology of the Industrial Revolution, and its development since then involves critical contributions in a broad range of domains stretching from the Space Race to the economically vital area of industrial process control. However, the current natural and designed environment presents a vast array of control problems generated by systems of immense complexity which are often populated by subsystems possessing both discrete and continuous states and which, moreover, display so-called intelligent behaviour. A sample would have to include (i) the need to stabilize and control in a distributed manner the huge communication, information, transportation and economic networks of the modern world, (ii) the control of systems such as agricultural, fisheries and ecological systems with the objective of achieving sustainability, (iii) the management of the current integration of distributed and intrinsically stochastic renewable energy sources into deregulated and immensely expanded transmission networks serving users of all scales, and, finally, the most difficult problem area of all, (iv) the effective stabilization of the planet's climate through political and economic intervention based on system theoretic concepts and grounded in the physical sciences. Clearly the problem areas above demand a range of responses from various disciplines, but they raise at least two specific, interlinked challenges of a basic nature for Systems and Control: 1. To create fundamental theory and effective methodology to analyze and control systems with both discrete and continuous states and dynamics. Despite two decades of advances, Hybrid System theory is still in an early stage of development, for instance continuous-to-discrete state aggregation still lacks an adequate theory and so hierarchical control is in its infancy. Furthermore, work taking inspiration from the life sciences, such as gene controlled and self-healing systems, is limited to say the least. 2. To create adequate methods for the analysis and decentralized control of networked systems of huge complexity with social characteristics. While there has been progress in the development of game theory for stochastic dynamical systems, in particular Mean Field Game (MFG) theory, many challenges lie ahead, two examples being (i) relating MFG methods to the currently emerging information based control theory wherein the capacity of each of the feedback channels in a network plays a critical role, and (ii) the development of a dynamical game theory of the formation and stability of coalitions. In addition, the state aggregation problem introduced in the first challenge area also naturally falls under this second heading.

Optimal gear selection problem for energy minimization for two-gear electric vehicles is formulated within the framework of optimal control theory. Methods for restricting the number of switchings are analyzed and a computational example is presented.

This paper proposes and analyzes a new distributed power-control algorithm based on the theory of stochastic approximation. The power-control problem is first converted into a stochastic approximation problem in which the zero point of a specific function is determined. A distributed power-control algorithm is then derived and its convergence properties are analyzed using standard techniques. In the distributed algorithm, each user iteratively updates its power level by using estimates of the inverse of the signal-to-interference ratio (SIR) of its channel. No knowledge of the channel gains or state information of other users is required. Moreover, the algorithm is robust in the sense that it can handle errors in the bit-error rate estimates, and hence, can be used in practical scenarios. Convergence of the algorithm is analyzed in the almost-sure sense

We outline a proof of the strong consistency of the maximum likelihood estimate of the parameters of Gaussian random processes possessing linear autoregressive moving average or state space representations.