Communication is crucial in multi-agent reinforcement learning when agents are not able to observe the full state of the environment. The most common approach to allow learned communication between agents is the use of a differentiable communication channel that allows gradients to flow between agents as a form of feedback. However, this is challenging when we want to use discrete messages to reduce the message size, since gradients cannot flow through a discrete communication channel. Previous work proposed methods to deal with this problem. However, these methods are tested in different communication learning architectures and environments, making it hard to compare them. In this paper, we compare several state-of-the-art discretization methods as well as a novel approach. We do this comparison in the context of communication learning using gradients from other agents and perform tests on several environments. In addition, we present COMA-DIAL, a communication learning approach based on DIAL and COMA extended with learning rate scaling and adapted exploration. COMA-DIAL uses COMA to learn the action policy while it uses the mechanism introduced in DIAL to learn the communication policy. Using COMA-DIAL allows us to perform experiments on more complex environments. Our results show that the novel ST-DRU method, proposed in this paper, achieves the best results out of all discretization methods across the different environments. It achieves the best or close to the best performance in each of the experiments and is the only method that does not fail on any of the tested environments.

Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning. The credit assignment problem, the non-stationarity of the communication environment and the problem of encouraging the agents to be influenced by incoming messages are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper introduces the novel multi-agent counterfactual communication learning (MACC) method which adapts counterfactual reasoning in order to overcome the credit assignment problem for communicating agents. Next, the non-stationarity of the communication environment, while learning the communication Q -function, is overcome by creating the communication Q -function using the action policy of the other agents and the Q -function of the action environment. As the exact method to create the communication Q -function can be computationally intensive for a large number of agents, two approximation methods are proposed. Additionally, a social loss function is introduced in order to create influenceable agents, which is required to learn a valid communication protocol. Our experiments show that MACC is able to outperform the state-of-the-art baselines in four different scenarios in the particle environment. Finally, we demonstrate the scalability of MACC in a matrix environment.

The conventional approach for controlling the supply temperature in collective space heating networks relies on a predefined heating curve determined by outdoor temperature and heat emitter type. This prioritises thermal comfort but lacks energetic and financial optimisation. This research proposes an adaptive supply temperature control in well-insulated dwellings, responsive to diverse environmental parameters. The approach considers variable electricity prices and accommodates different indoor temperature set points in dwellings. The study evaluates the effectiveness of two Deep Reinforcement Learning (DRL) algorithms, i.e., Proximal Policy Optimisation (PPO) and Deep Q-Network (DQN), across various scenarios. Results reveal that DQN excels in collective space heating systems with underfloor heating in each dwelling, while PPO proves superior for radiator-based systems. Both outperform the traditional heating curve, achieving up to 13.77% (DQN) and 16.15% (PPO) cost reduction while guaranteeing thermal comfort. Additionally, the research highlights the capability of DRL-based methods to dynamically set the supply temperature based on a cloud of set points, showcasing adaptability to diverse environmental factors and addressing the growing significance of indoor heat gains in well-insulated dwellings. This innovative approach holds promise for more efficient and environmentally conscious heating strategies within collective space heating networks.

As the need for stand-alone energy harvesting devices increases, the alleviation of the ecological and economic impact of their production and maintenance is possible by increasing battery life while reducing needed battery capacity.However, the increasing energy requirements of far-edge Artificial Intelligence and long-range wireless transmissions in the Internet of Things threaten to demand ever-larger battery capacities for such remote devices.Dynamic adaptation of device operation based on harvestable energy -i.e., energy awarenessis a proposed solution and can be implemented using an energy management unit.Standardizing this unit as a separate, active electronic component with standardized drivers can simplify overall system development and benefit existing devices.Hence, we propose a novel interface that allows decoupling this unit from the rest of the system, independent of the power management unit in use.As a first step, we developed a prototype that uses the proposed interface to make existing, solar energy-based, third-party devices energy-aware with provisions to be cross-compatible with differing power management units.The prototype was evaluated using an air quality sensing device and improved the overall device's transmission rate.

Road weather conditions such as ice, snow, or heavy rain can have a significant impact on driver safety. Vehicle safety technologies have a reactive nature to these conditions. In this paper, we discuss the state of our research using a vehicle fleet equipped with external sensors to enhance road weather services. We present the architecture to share data amongst stakeholders. Next, the data are investigated. Significant trends in the data can be found when the rain starts or stops using correlation and relative measurements. This is followed by an investigation of the placed sensors to ensure qualitative measurements. Lastly, we discuss a road weather model that is adapted to make use of these car sensor observations.

Next-generation mobile networks are expected to flaunt highly (if not fully) automated management. To achieve such a vision, Artificial Intelligence (AI) and Machine Learning (ML) techniques will be key enablers to craft the required intelligence for networking, i.e., Network Intelligence (NI), empowering myriad of orchestrators and controllers across network domains. In this paper, we elaborate on the DAEMON architectural model, which proposes introducing a NI Orchestration layer for the effective end-to-end coordination of NI instances deployed across the whole mobile network infrastructure. Specifically, we first outline requirements and specifications for NI design that stem from data management, control timescales, and network technology characteristics. Then, we build on such analysis to derive initial principles for the design of the NI Orchestration layer, focusing on (i) proposals for the interaction loop between NI instances and the NI Orchestrator, and (ii) a unified representation of NI algorithms based on an extended MAPE-K model. Our work contributes to the definition of the interfaces and operation of a NI Orchestration layer that foster a native integration of NI in mobile network architectures.

To enable autonomous vessels to operate on inland waterways, they need to detect, track and localize objects at close range to safely navigate. We deployed current deep learning techniques to detect and track these objects. As there are no large labeled datasets of European inland waterways, we used transfer learning to overcome the lack of data. By using preexisting similar datasets, we were able to significantly decrease the required amount of labeled data from the target distribution. Furthermore, we improved the mean Average Precision from 0.461 to 0.814 by using a limited number of labeled target data samples. We estimated the relative distance of the objects based on the generated bounding boxes. The information from the camera is then combined with LiDar data to generate a top-view map of the environment which is used as input for an object-avoidance control agent. All these methods can run in real-time on the vessel with an fps of 1.83 on a 2.7GHz vCPU.

Indoor temperature modeling has been a vital component to develop accurate digital twins and smart controllers for buildings. Hybrid (also known as gray-box) modeling caught significant attention from the literature for this task. Combining the accumulated physical knowledge we have about thermal behavior with modern data-driven techniques promises more accurate and stable prediction models which can be used in various applications. However, methods such as data-driven parameter optimization and constrained training proposed in the literature show practical limitations such as high computational expense and software incompatibilities. In this paper we propose a transfer learning-based hybrid modeling approach where a CNN-LSTM model is pre-trained with the simulation data and then refined with the real-life data, thus, creating a completely data-driven hybrid model. We compared our approach to the same CNN-LSTM architecture trained only on real-life data. We reported significant accuracy and stability increases with the proposed approach.

Combined heat distribution circuits (CHDCs) are increasingly used in apartment buildings. Here only one supply pipe distributes both space heating (SH) and domestic hot water (DHW). Currently, the supply temperature is set to the highest temperature needed by one of the end-users (i.e. 65ºC for DHW), even if low-temperature emitters are used for SH. However, using decentral storage tanks for DHW enable demand-based temperature controls to reduce unnecessary heat losses and poor efficiencies. This research uses reinforcement learning (RL), a machine learning technique, to develop new control strategies for CHDCs with underfloor heating and DHW storage tanks. The agent controls the supply temperature and the mass flow in the hybrid boiler room. Whether the RL agent is able to find the optimal control strategy depends on the definition of its Markov Decision Process (MDP) model elements, namely the states, the possible control actions and the reward function. The results show that an increasing gamma and decreasing learning rate during training leads to better performance and that the agent with the largest flexibility develops a better control strategy that resulted in up to 23% primary energy savings.

In recent years, interest in autonomous shipping in urban waterways has increased significantly due to the trend of keeping cars and trucks out of city centers. Classical approaches such as Frenet frame based planning and potential field navigation often require tuning of many configuration parameters and sometimes even require a different configuration depending on the situation. In this paper, we propose a novel path planning approach based on reinforcement learning called Model Predictive Reinforcement Learning (MPRL). MPRL calculates a series of waypoints for the vessel to follow. The environment is represented as an occupancy grid map, allowing us to deal with any shape of waterway and any number and shape of obstacles. We demonstrate our approach on two scenarios and compare the resulting path with path planning using a Frenet frame and path planning based on a proximal policy optimization (PPO) agent. Our results show that MPRL outperforms both baselines in both test scenarios. The PPO based approach was not able to reach the goal in either scenario while the Frenet frame approach failed in the scenario consisting of a corner with obstacles. MPRL was able to safely (collision free) navigate to the goal in both of the test scenarios.

The service placement problem considers the placement of multiple connected services across a heterogeneous device network and is one of the core problems of fog computing. We discuss the complexity of this service placement problem, and propose a model for solving it using Multi-Objective Reinforcement Learning (MORL) methodologies. Using a trained neural network greatly reduces the resource consumption of the placement algorithm, making it viable for resource-constrained scenarios. Starting from state-of-the-art techniques, we develop a generic max reward formulation model and apply several MORL methodologies, which solve the placement problem in scenarios where the preference weights change. We compare the results to a baseline methodology and showcase the value of MORL on the placement problem.

5G systems promise to enable autonomous vehicles by empowering road-, water-, and air-vehicles with ultra low latency communications and computing at edge in order to share and process data from multiple sensors. However, in order to realize such fully Connected and Automated Mobility (CAM) for cars, drones and vessels, a crucial intermediary step must be fully achieved: 5G-based tele-operated transport. In order to do so, the European project H2020 5G-Blueprint aims to design, test, and validate in real deployments a 5G-enabled tele-operated transport and its enabling functions in both a relevant and operational environment realised through cross-border trials on the road and on the water along 5G corridors in the Dutch and Belgian border area, resulting in a blueprint for future cooperation on 5G-enabled CAM between public, private and semi-private parties (e.g, ports), gaining new and innovative insights on the stringent particular requirements for safe CAM, on the architecture, on governance and relevant business models.

Convolutional neural networks (CNNs) have accomplished magnificent performance on object classification tasks. This work introduces a novel image classification approach based on feature vector fusion of two CNN architectures using graph attention networks (GAT). In the proposed method we extract feature maps from shallow and deep layers of two CNN architectures. These extracted feature vectors are represented as nodes in a graph, and edges between nodes are constructed based on similarities between the feature vectors. The GAT is used to aggregate and fuse connected nodes based on their importance and relevance to the classification task. We believe that this approach compensates for convolution defects during feature processing when using a single CNN. This paper attempts to show that graph-based deep learning can be used to fuse two CNN architectures and not to push the state-of-the-art of image classification accuracy. Our experimental results prove that GAT can be used to fuse feature vectors resulting from CNN architectures.

Bad weather conditions such as heavy rain, black ice and fog can have a significant impact on road safety. Currently vehicle safety technologies such as the electronic stability program work reactive to hazardous situations. In this paper, we propose the use of crowd-sourced vehicle data to improve road-weather models and provide real-time local warnings for weather-related hazards. We present our initial results from a field test where we used vehicle CAN-bus data and low cost external sensors to observe local weather phenomena. The CAN-bus contains, among others, data on vehicle dynamics such as wheel speeds. Our approach is to isolate anomalies within these signals. Our initial research suggests some anomalies are weather related and can be used to describe local weather phenomena. Furthermore, the externally installed sensors provide more information on which we can build our assumptions. The results show that the gathered measurements are consistent with the reliable observations from road weather stations.

Efficiently using the computational power made available through desktop grids based distributed systems is a complicated and many-sided problem, caused by the intermittent resource availability. In this paper a novel solution is presented for predicting the runtimes of parameter sweep jobs. These jobs are characterized by their lack of inter-dependence and suitability for runtime prediction by modeling. This makes them ideal candidates for deployment on volatile grid configurations using prediction based techniques. The parameter sweep prediction framework used to make the predictions is referred to as GIPSy (grid information prediction system). Previous research involving GIPSy has focused on results obtained during simulation were it is necessary to make some basic assumptions. By combining GIPSy with PGS (prediction based grid scheduling), an actual grid implementation, real results can be obtained. A detailed comparison between the expected results, based on simulation analysis, and the final results is given. Discrepancies are highlighted and possible causes are identified, solutions are proposed and implemented. By comparing the results for different model building configurations an optimal configuration is found that produces reliable result independent of the chosen job type. Results are presented for a quantum physics problem and two simulated workloads represented by sleep jobs.

This paper examines the problem of predicting job runtimes by exploiting the properties of parameter sweeps. A new parameter sweep prediction framework GIPSy (grid information prediction system) is introduced. Predictions are made based on prior runtime information and the parameters used to configure each job. The main objective is providing a tool combining development, simulation and application of prediction models within one framework. The different kinds of available sample selectors and models are discussed in detail. Results are presented for a quantum physics problem. A previously introduced scheduling technique and the implementation called PGS (prediction based grid scheduling) is improved and presented in combination with GIPSy to obtain a real-world grid implementation that optimizes the distribution of parameter sweeps.