Direct policy search (DPS) is emerging as one of the most effective and widely applied reinforcement learning (RL) methods to design optimal control policies for multiobjective Markov decision processes (MOMDPs). Traditionally, DPS defines the control policy within a preselected functional class and searches its optimal parameterization with respect to a given set of objectives. The functional class should be tailored to the problem at hand and its selection is crucial, as it determines the search space within which solutions can be found. In MOMDPs problems, a different objective tradeoff determines a different fitness landscape, requiring a tradeoff-dynamic functional class selection. Yet, in state-of-the-art applications, the policy class is generally selected a priori and kept constant across the multidimensional objective space. In this work, we present a novel policy search routine called neuro-evolutionary multiobjective DPS (NEMODPS), which extends the DPS problem formulation to conjunctively search the policy functional class and its parameterization in a hyperspace containing policy architectures and coefficients. NEMODPS begins with a population of minimally structured approximating networks and progressively builds more sophisticated architectures by topological and parametrical mutation and crossover, and selection of the fittest individuals concerning multiple objectives. We tested NEMODPS for the problem of designing the control policy of a multipurpose water system. Numerical results show that the tradeoff-dynamic structural and parametrical policy search of NEMODPS is consistent across multiple runs, and outperforms the solutions designed via traditional DPS with predefined policy topologies.

In most of the transfer learning approaches to reinforcement learning (RL) the distribution over the tasks is assumed to be stationary. Therefore, the target and source tasks are i.i.d. samples of the same distribution. Unfortunately, this assumption rarely holds in real-world conditions, e.g., due to seasonality or periodicity, evolution in the environment or faults in the sensors/actuators. In the context of this work, we consider the problem of transferring value functions through a variational method when the distribution that generates the tasks is time-variant, proposing a solution that leverages this temporal structure inherent in the task generating process. Furthermore, by means of a finite-sample analysis, the previously mentioned solution is theoretically compared to its time-invariant version. Finally, the experimental evaluation of the proposed technique is carried out on the lake Como water system representing a real-world scenario and on three different RL environments with three distinct temporal dynamics.

This paper merges multimedia and environmental research to verify the utility of public web images for improving water management in periods of water scarcity, an increasingly critical event due to climate change. A multimedia processing pipeline fetches mountain images from multiple sources and extracts virtual snow indexes correlated to the amount of water accumulated in the snow pack. Such indexes are used to predict water availability and design the operating policy of Lake Como, Italy. The performance of this informed policy is contrasted, via simulation, with the current operation, which depends only on lake water level and day of the year, and with a policy that exploits official Snow Water Equivalent (SWE) estimated from ground stations data and satellite imagery. Virtual snow indexes allow improving the system performance by 11.6% w.r.t. the baseline operation, and yield further improvement when coupled with official SWE information, showing that the two data sources are complementary. The proposed approach exemplifies the opportunities and challenges of applying multimedia content analysis methods to complex environmental problems.

Climate change and anthropogenic pressures are expected to reduce global freshwater availability and to exacerbate water crises in the near futures. To manage water crisis, the most common path of the past century has been a “hard-path”, based on centralized structural actions, strongly affecting the environment. This work aims at developing novel adaptive water management control strategies, in a multi-stakeholders context, based on soft-path measures. We focus on the direct, data-driven use of exogenous information to improve the systems anticipation capability. Specifically, the value of snow data in informing water resource systems operation is explored using a model free approach. The underline idea is that information on the snow water equivalent (SWE) in the basin may be relevant to manage the reservoir releases. To this aim SWE estimates are retrieved from SAR COSMO-SkyMed X-band images. The approach is demonstrated on snow-rain fed river basin in the Italian Alps: the Lake Como watershed. Preliminary results show the relevance of snow information to the reservoir management as well as the potential for remote sensed products in data-driven optimization.

This paper is about solving multi-objective control problems using a model-free batch-mode reinforcement-learning approach. Although many real-world applications have several conflicting objectives, reinforcement-learning (RL) literature has mainly focused on single-objective control problems. As a consequence, in the presence of multiple objectives, the usual approach is to consider many single-objective control problems (resulting from different combinations of the original problem objectives), each one solved using standard RL techniques. The algorithm proposed in this paper is an extension of Fitted Q-iteration (FQI) that enables to learn the control policies for all the linear combinations of preferences (weights) assigned to the objectives in a single training process. The key idea of multi-objective FQI (MOFQI) is to enlarge the continuous approximation of the action-value function, which is performed by single-objective FQI over the state-action space, also to the weight space. The approach is demonstrated on an interesting real-world application for multi-objective RL algorithms: the optimal operation of a multi-purpose water reservoir.

This paper is about dimensionality reduction by variable selection in high-dimensional real-world control problems, where designing controllers by conventional means is either impractical or results in poor performance.