A default assumption in reinforcement learning (RL) and optimal control is that observations arrive at discrete time points on a fixed clock cycle. Yet, many applications involve continuous-time systems where the time discretization, in principle, can be managed. The impact of time discretization on RL methods has not been fully characterized in existing theory, but a more detailed analysis of its effect could reveal opportunities for improving data-efficiency. We address this gap by analyzing Monte-Carlo policy evaluation for LQR systems and uncover a fundamental trade-off between approximation and statistical error in value estimation. Importantly, these two errors behave differently to time discretization, leading to an optimal choice of temporal resolution for a given data budget. These findings show that managing the temporal resolution can provably improve policy evaluation efficiency in LQR systems with finite data. Empirically, we demonstrate the trade-off in numerical simulations of LQR instances and standard RL benchmarks for non-linear continuous control.

With technological advances, the number of cores integrated on a chip is increasing. This in turn is leading to thermal constraints and thermal design challenges. Temperature gradients and hotspots not only affect the performance of the system but also lead to unreliable circuit operation and affect the lifetime of the chip. Meeting temperature constraints and reducing hotspots are critical for achieving reliable and efficient operation of complex multi-core systems. In this article, we analyze the use of four of the most promising families of online control techniques for thermal management of multiprocessors system-on-chip (MPSoC). In particular, in our exploration, we aim at achieving an online smooth thermal control action that minimizes the performance loss as well as the computational and hardware overhead of embedding a thermal management system inside the MPSoC. The definition of the optimization problem to tackle in this work considers the thermal profile of the system, its evolution over time, and current time-varying workload requirements. Thus, this problem is formulated as a finite-horizon optimal control problem, and we analyze the control features of different online thermal control approaches. In addition, we implemented the policies on an MPSoC hardware simulation platform and performed experiments on a cycle-accurate model of the eight-core Niagara multi-core architecture using benchmarks ranging from Web-accessing to playing multimedia. Results show different trade-offs among the analyzed techniques regarding the thermal profile, the frequency setting, the power consumption, and the implementation complexity.

Multi-Processor Systems-on-Chip (MPSoCs) are penetrating the electronics market as a powerful, yet commercially viable, solution to answer the strong and steadily growing demand for scalable and high performance systems, at limited design complexity. However, it is critical to develop dedicated system-level design methodologies for multi-core architectures that seamlessly address their thermal modeling, analysis and management. In this work, we first formulate the problem of system-level thermal modeling and link it to produce a global thermal management formulation as a discrete-time optimal control problem, which can be solved using finite-horizon model-predictive control (MPC) techniques, while adapting to the actual time-varying unbalanced MPSoC workload requirements. Finally, we compare the system-level MPC-based thermal modeling and management approaches on an industrial 8-core MPSoC design and show their different trade-offs regarding performance while respecting operating temperature bounds.

Meeting the temperature constraints and reducing the hot-spots are critical for achieving reliable and efficient operation of complex multi-core systems. The goal of thermal management is to meet maximum operating temperature constraints, while tracking timevarying performance requirements. Current approaches avoid thermal violations by forcing abrupt operating points changes, which cause sharp performance degradation. In this paper we aim at achieving an online smooth thermal control action, that minimizes the tracking error. We formulate this problem as a discrete-time optimal control problem, which can be solved via online by using an embedded convex optimization solver using a receding horizon approach. The optimization problem considers the thermal profile of the system, its evolution over time, current and past time-varying workload requirements. We perform experiments on a model of the 8-core Niagara-1 multicore architecture, which show that the proposed method outperforms state-of-the-art thermal management approaches by enabling performance speed-ups of up to 2:5£ and improvements up to 12x and 3.4x in relation to frequency and temperature variations over time, respectively.

Modern high-performance processors employ thermal management systems, which rely on accurate readings of on-die thermal sensors. Systematic tools for analysis and determination of best allocation and placement of thermal sensors is therefore a highly relevant problem. This paper proposes a novel technique for determining the placement of temperature sensors on complex Multi-Processor Systems-on-Chips (MPSoCs) floorplans. The proposed method first analyzes the observability of the system for all the possible sensor placement configurations. Minimum sensors placements ensuring the observability of the portion of the MPSoC system that is relevant to the designer are then compared with simulation-based data coming from a wide set of benchmarks. Pareto points identifying the best configurations are than stored. According to user designer needs the best configuration is selected and a specific location is assigned to each sensor. We compared the proposed method with state-of-the-art approaches. Results show a reduction up to 4.5× in the number of required sensors.

Meeting temperature constraints and reducing the hot-spots are critical for achieving reliable and efficient operation of complex multi-core systems. In this paper we aim at achieving an online smooth thermal control action that minimizes the performance loss as well as the computational and hardware overhead of embedding a thermal management system inside the MPSoC. The optimization problem considers the thermal profile of the system, its evolution over time and current time-varying workload requirements. We formulate this problem as a discrete-time control problem using model predictive control. The solution is computed off-line and partially on-line using an explicit approximate algorithm. This proposed method, compared with the optimum approach provides a significant reduction in hardware requirements and computational cost at the expense of a small loss in accuracy. We perform experiments on a model of the 8-core Niagara-1 multicore architecture using benchmarks ranging from web-accessing to playing multimedia. Results show that the proposed method provides comparable performance(loss up to 2.7%) versus the optimum solution with a reduction up to 72.5× in the the computational complexity.

Thermal balancing and reducing hot-spots are two important challenges facing the MPSoC designers. In this work, we model the thermal behavior of a MPSoC as a control theory problem which enables the design of an optimum frequency controller without depending on the thermal profile of the chip. The optimization performed by the controller is targeted to achieve thermal balancing on the MPSoC thermal profile to avoid hotspots and improve its reliability. The proposed system is able to perform an on-line minimization of chip thermal gradients based on both scheduler requirements and the chip thermal profile. We compare this with state of the art thermal management approaches. Our comparison shows that the proposed system offers a better both thermal profile (temperature differences higher than 4deg C have been reduced from 27.9% to 0.45%) and performance (up to 32% task waiting time reduction).