We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero’s state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.

Credit assignment in reinforcement learning is the problem of measuring an action’s influence on future rewards. In particular, this requires separating skill from luck, i.e. disentangling the effect of an action on rewards from that of external factors and subsequent actions. To achieve this, we adapt the notion of counterfactuals from causality theory to a model-free RL setup. The key idea is to condition value functions on future events, by learning to extract relevant information from a trajectory. We formulate a family of policy gradient algorithms that use these future-conditional value functions as baselines or critics, and show that they are provably low variance. To avoid the potential bias from conditioning on future information, we constrain the hindsight information to not contain information about the agent’s actions. We demonstrate the efficacy and validity of our algorithm on a number of illustrative and challenging problems.

Value estimation is a critical component of the reinforcement learning (RL) paradigm. The question of how to effectively learn value predictors from data is one of the major problems studied by the RL community, and different approaches exploit structure in the problem domain in different ways. Model learning can make use of the rich transition structure present in sequences of observations, but this approach is usually not sensitive to the reward function. In contrast, model-free methods directly leverage the quantity of interest from the future, but receive a potentially weak scalar signal (an estimate of the return). We develop an approach for representation learning in RL that sits in between these two extremes: we propose to learn what to model in a way that can directly help value prediction. To this end, we determine which features of the future trajectory provide useful information to predict the associated return. This provides tractable prediction targets that are directly relevant for a task, and can thus accelerate learning the value function. The idea can be understood as reasoning, in hindsight, about which aspects of the future observations could help past value prediction. We show how this can help dramatically even in simple policy evaluation settings. We then test our approach at scale in challenging domains, including on 57 Atari 2600 games.

The Web of Things (WoT) has recently appeared as the latest evolution of the Internet of Things and, as the name suggests, requires that devices interoperate through the Internet using Web protocols and standards. Currently, only a few theoretical approaches have been presented by researchers and industry, to fight the fragmentation of the IoT world through the adoption of semantics. This further evolution is known as Semantic WoT and relies on a WoT implementation crafted on the technologies proposed by the Semantic Web stack. This article presents a working implementation of the WoT declined in its Semantic flavor through the adoption of a shared ontology for describing devices. In addition to that, the ontology includes patterns for dynamic interactions between devices, and therefore we define it as dynamic ontology. A practical example will give a proof of concept and overall evaluation, showing how the dynamic setup proposed can foster interoperability at information level allowing on the one hand smart discovery, enabling on the other hand orchestration and automatic interaction through the semantic information available.

In model-based reinforcement learning, generative and temporal models of environments can be leveraged to boost agent performance, either by tuning the agent’s representations during training or via use as part of an explicit planning mechanism. However, their application in practice has been limited to simplistic environments, due to the difficulty of training such models in larger, potentially partially-observed and 3D environments. In this work we introduce a novel action-conditioned generative model of such challenging environments. The model features a non-parametric spatial memory system in which we store learned, disentangled representations of the environment. Low-dimensional spatial updates are computed using a state-space model that makes use of knowledge on the prior dynamics of the moving agent, and high-dimensional visual observations are modelled with a Variational Auto-Encoder. The result is a scalable architecture capable of performing coherent predictions over hundreds of time steps across a range of partially observed 2D and 3D environments.

Quasi-Z-source cascaded H-bridge (qZS-CHB) inverters are one promising solution for high power photovoltaic (PV) systems. This type of topologies inherits the advantages of cascaded converters (i.e., multilevel outputs) and impedance-source inverters (i.e., high conversion ratios). In addition, it allows increasing the inverter reliability (with high redundancy). However, the modulation and control of qZS-CHB inverters are challenging to a certain extent. Thus, this paper proposes modified modulation techniques to increase the performances of qZS-CHB converters in terms of voltage gains and stresses. The novelty lies in the use of the switching frequency optimal as reference signals in the modulation techniques. A comparison in terms of harmonics with selected modulation techniques is performed.

Semantic Web technologies act as an interoperability glue among different formats, protocols and platforms, providing a uniform vision of heterogeneous devices and services in the Internet of Things (IoT). Semantic Web technologies can be applied to a broad range of application contexts (i.e., industrial automation, automotive, health care, defense, finance, smart cities) involving heterogeneous actors (i.e., end users, communities, public authorities, enterprises). Smart-M3 is a semantic publish-subscribe software architecture conceived to merge the Semantic Web and the IoT domains. It is based on a core component (SIB, Semantic Information Broker) where data is stored as RDF graphs, and software agents using SPARQL to update, retrieve and subscribe to changes in the data store. This article describes a OSGi SIB implementation extended with a new persistent SPARQL update primitive. The OSGi SIB performance has been evaluated and compared with the reference C implementation. Eventually, a first porting on Android is presented.

The study and engineering of Electro Mobility (EM) involves many industrial and academic players due to its potential significant benefits on society, economy, transportation and eco-sustainability. This article proposes a solution for EM automation, a crucial aspect for the future evolution of the EM market. The solution is based on a service-oriented, IoT and cloud centric ecosystem of charging stations conceived to support EM scenarios. The Eclipse Kura framework ensures the information exchange between the charging stations and the cloud infrastructure, providing remote management and data processing. The cloud platform is based on Eurotech EC and provides an internal service abstraction that offers efficient and secure mechanisms to collect raw data from the field. Kura is used to process data and store them on the cloud as semantically referenced information. Cloud services are accessible through a simple REST interface and published on the Arrowhead Framework, which is responsible for the integration of the multi-domain EM scenario. A real use case about the automation of a rural fast charging infrastructure is described and the benefits of applying the proposed solution to the use case are discussed with the help of the simulation results.

This paper presents an overview and experimental analysis of the MC SPWM techniques for single-phase cascaded H-bridge inverter. The multilevel power converters are an alternative to traditional converters known as “three-level converters”. The voltage waveforms and the related frequency spectra, which have been obtained by simulation analysis in Matlab-Simulink environment, are here reported for all the proposed modulation techniques. The simulation results have been experimentally validated through means of a DC/AC, five-level, single-phase converter prototype with an appropriate test bench.

This paper presents a publish-subscribe architecture designed to support information level interoperability in smart space applications in the Internet of Things (IoT). The architecture is built on top of a generic SPARQL endpoint where publishers and subscribers use standard SPARQL Updates and Queries. Notifications about events [i.e., changes in the resource description framework (RDF) knowledge base] are expressed in terms of added and removed SPARQL binding results since the previous notification, limiting the network overhead and facilitating notification processing at subscriber side. A novel event detection algorithm, tailored on the IoT specificities (i.e., heterogeneous events need to be detected and continuous updates of few RDF triples dominate with respect to more complex updates), is presented along with the envisioned application design pattern and performance evaluation model. Eventually, a reference implementation is evaluated against a benchmark inspired by a smart city lighting case. The performance evaluation results show the capability to process up to 68k subscriptions/s triggered by simple single-lamp updates and up to 3.8k subscriptions/s triggered by more complex updates (i.e., 10 to 100 lamps).

Rotoscoping (cutting out different characters/objects/layers in raw video footage) is a ubiquitous task in modern post-production and represents a significant investment in person-hours. In this work, we study the particular task of professional rotoscoping for high-end, live action movies and propose a new framework that works with roto-artists to accelerate the workflow and improve their productivity. Working with the existing keyframing paradigm, our first contribution is the development of a shape model that is updated as artists add successive keyframes. This model is used to improve the output of traditional interpolation and tracking techniques, reducing the number of keyframes that need to be specified by the artist. Our second contribution is to use the same shape model to provide a new interactive tool that allows an artist to reduce the time spent editing each keyframe. The more keyframes that are edited, the better the interactive tool becomes, accelerating the process and making the artist more efficient without compromising their control. Finally, we also provide a new, professionally rotoscoped dataset that enables truly representative, real-world evaluation of rotoscoping methods. We used this dataset to perform a number of experiments, including an expert study with professional roto-artists, to show, quantitatively, the advantages of our approach.

We present a model for early vision tasks such as denoising, super-resolution, deblurring, and demosaicing. The model provides a resolution-independent representation of discrete images which admits a truly rotationally invariant prior. The model generalizes several existing approaches: variational methods, finite element methods, and discrete random fields. The primary contribution is a novel energy functional which has not previously been written down, which combines the discrete measurements from pixels with a continuous-domain world viewed through continous-domain point-spread functions. The value of the functional is that simple priors (such as total variation and generalizations) on the continous-domain world become realistic priors on the sampled images. We show that despite its apparent complexity, optimization of this model depends on just a few computational primitives, which although tedious to derive, can now be reused in many domains. We define a set of optimization algorithms which greatly overcome the apparent complexity of this model, and make possible its practical application. New experimental results include infinite-resolution upsampling, and a method for obtaining “subpixel superpixels”.

In this paper, a system which constructs a mosaic image of the tunnel surface with little distortion is presented. The tunnel surface is typically composed of a roughly cylindrical surface and protuberant regions containing objects such as pipes, pans and tunnel ridges. Since the true surface is neither planar nor quadric, existing mosaicing methods, which assume either homography or quadratic motion models, suffer from distortion. The proposed system obtains a sparse 3D model of the tunnel by multi-view reconstruction. Then, the Support Vector Machine (SVM) classifier is applied in order to separate image features lying on the cylindrical surface from those of the non-surface. The reconstructed 3D points are reprojected into images to retrieve the priors given by the SVM classifier for accurate cylindrical surface estimation. The final mosaic image is obtained by flattening the estimated textured surface onto a plane. The results suggest that the mosaic quality depends critically on the surface estimation accuracy and the proposed system is able to produce the mosaic image that preserves all physical sense, e.g. line parallelism and straightness, which is important for tunnel inspection.