Learning to communicate in order to share state information is an active problem in the area of multi-agent reinforcement learning. The credit assignment problem, the non-stationarity of the communication environment and the problem of encouraging the agents to be influenced by incoming messages are major challenges within this research field which need to be overcome in order to learn a valid communication protocol. This paper introduces the novel multi-agent counterfactual communication learning (MACC) method which adapts counterfactual reasoning in order to overcome the credit assignment problem for communicating agents. Next, the non-stationarity of the communication environment, while learning the communication Q -function, is overcome by creating the communication Q -function using the action policy of the other agents and the Q -function of the action environment. As the exact method to create the communication Q -function can be computationally intensive for a large number of agents, two approximation methods are proposed. Additionally, a social loss function is introduced in order to create influenceable agents, which is required to learn a valid communication protocol. Our experiments show that MACC is able to outperform the state-of-the-art baselines in four different scenarios in the particle environment. Finally, we demonstrate the scalability of MACC in a matrix environment.

Maintaining roads is crucial to economic growth and citizen well-being because roads are a vital means of transportation. In various countries, the inspection of road surfaces is still done manually, however, to automate it, research interest is now focused on detecting the road surface defects via the visual data. While, previous research has been focused on deep learning methods which tend to process the entire image and leads to heavy computational cost. In this study, we focus our attention on improving the classification performance while keeping the computational cost of our solution low. Instead of processing the whole image, we introduce a segmentation model to only focus the downstream classification model to the road surface in the image. Furthermore, we employ contrastive learning during model training to improve the road surface condition classification. Our experiments on the public RTK dataset demonstrate a significant improvement in our proposed method when compared to previous works.

In the field of autonomous driving, collaborative perception has emerged as a promising solution for augmenting the capabilities of individual sensors by enabling vehicles to share their sensor information across each other, thereby enhancing their situational awareness. This paper addresses the limitations of classical perception in autonomous vehicles by proposing a novel intermediate collaborative perception methodology employing graph attention network (GAT) to incorporate multiple feature maps and to selectively emphasize important regions within the feature maps. We construct the graph structure as a set of nodes embedding the ego and the neighboring connected vehicles feature maps, as well as establish edge weights between those nodes based on their relationship to each other which is defined by the attention coefficients. The proposed approach leverages both channel and spatial attention-based aggregation and enables the model to determine inter-feature map relationships at a specific channel and spatial regions, while adaptively highlighting the informative regions. This adaptive highlighting mechanism directs the aggregation algorithm towards the most informative areas within the ego and the received feature maps, thereby enhancing the representation power of the ego vehicle’s feature map leading to improved precision in object detection. We quantitatively and qualitatively evaluate the performance of our proposed approach against existing state-of-the-art in collaborative perception. We validate our methodology using V2XSim, a large-scale multi-agent perception dataset. The results demonstrate that our methodology achieves superior performance in enhancing object detection average precision.

To enable autonomous vessels to operate on inland waterways, they need to detect, track and localize objects at close range to safely navigate. We deployed current deep learning techniques to detect and track these objects. As there are no large labeled datasets of European inland waterways, we used transfer learning to overcome the lack of data. By using preexisting similar datasets, we were able to significantly decrease the required amount of labeled data from the target distribution. Furthermore, we improved the mean Average Precision from 0.461 to 0.814 by using a limited number of labeled target data samples. We estimated the relative distance of the objects based on the generated bounding boxes. The information from the camera is then combined with LiDar data to generate a top-view map of the environment which is used as input for an object-avoidance control agent. All these methods can run in real-time on the vessel with an fps of 1.83 on a 2.7GHz vCPU.

Indoor temperature modeling has been a vital component to develop accurate digital twins and smart controllers for buildings. Hybrid (also known as gray-box) modeling caught significant attention from the literature for this task. Combining the accumulated physical knowledge we have about thermal behavior with modern data-driven techniques promises more accurate and stable prediction models which can be used in various applications. However, methods such as data-driven parameter optimization and constrained training proposed in the literature show practical limitations such as high computational expense and software incompatibilities. In this paper we propose a transfer learning-based hybrid modeling approach where a CNN-LSTM model is pre-trained with the simulation data and then refined with the real-life data, thus, creating a completely data-driven hybrid model. We compared our approach to the same CNN-LSTM architecture trained only on real-life data. We reported significant accuracy and stability increases with the proposed approach.

Combined heat distribution circuits (CHDCs) are increasingly used in apartment buildings. Here only one supply pipe distributes both space heating (SH) and domestic hot water (DHW). Currently, the supply temperature is set to the highest temperature needed by one of the end-users (i.e. 65ºC for DHW), even if low-temperature emitters are used for SH. However, using decentral storage tanks for DHW enable demand-based temperature controls to reduce unnecessary heat losses and poor efficiencies. This research uses reinforcement learning (RL), a machine learning technique, to develop new control strategies for CHDCs with underfloor heating and DHW storage tanks. The agent controls the supply temperature and the mass flow in the hybrid boiler room. Whether the RL agent is able to find the optimal control strategy depends on the definition of its Markov Decision Process (MDP) model elements, namely the states, the possible control actions and the reward function. The results show that an increasing gamma and decreasing learning rate during training leads to better performance and that the agent with the largest flexibility develops a better control strategy that resulted in up to 23% primary energy savings.

In recent years, interest in autonomous shipping in urban waterways has increased significantly due to the trend of keeping cars and trucks out of city centers. Classical approaches such as Frenet frame based planning and potential field navigation often require tuning of many configuration parameters and sometimes even require a different configuration depending on the situation. In this paper, we propose a novel path planning approach based on reinforcement learning called Model Predictive Reinforcement Learning (MPRL). MPRL calculates a series of waypoints for the vessel to follow. The environment is represented as an occupancy grid map, allowing us to deal with any shape of waterway and any number and shape of obstacles. We demonstrate our approach on two scenarios and compare the resulting path with path planning using a Frenet frame and path planning based on a proximal policy optimization (PPO) agent. Our results show that MPRL outperforms both baselines in both test scenarios. The PPO based approach was not able to reach the goal in either scenario while the Frenet frame approach failed in the scenario consisting of a corner with obstacles. MPRL was able to safely (collision free) navigate to the goal in both of the test scenarios.

Convolutional neural networks (CNNs) have accomplished magnificent performance on object classification tasks. This work introduces a novel image classification approach based on feature vector fusion of two CNN architectures using graph attention networks (GAT). In the proposed method we extract feature maps from shallow and deep layers of two CNN architectures. These extracted feature vectors are represented as nodes in a graph, and edges between nodes are constructed based on similarities between the feature vectors. The GAT is used to aggregate and fuse connected nodes based on their importance and relevance to the classification task. We believe that this approach compensates for convolution defects during feature processing when using a single CNN. This paper attempts to show that graph-based deep learning can be used to fuse two CNN architectures and not to push the state-of-the-art of image classification accuracy. Our experimental results prove that GAT can be used to fuse feature vectors resulting from CNN architectures.

The autonomous operation of industrial robots with minimal human supervision has always been in high demand. To prepare the autonomous operation of a car part spray painting robot, novel object detection, and pose estimation algorithms have been developed in this paper. The object detection part used principal components analysis (PCA) to reduce the dimension of three-dimensional (3-D) point cloud to 2-D binary image. Distance measure between the auto and cross correlation of the binary features was established to find out the similarity between them. Resultantly, the type of auto part was successfully obtained. Furthermore, iterative closest point (ICP) algorithm was used to estimate the pose difference of the auto part with respect to the camera reference frame, which was mounted on the robot. An issue with ICP's lack of robustness to local minimum was solved by the combination of ICP and genetic algorithm (GA). This allowed the optimization of pose error and addressed the problem of local minimum entrapment in ICP. For experimental validation: the proposed object recognition pipeline was implemented in both serial and parallel programming paradigms. The results were obtained for the acquired point clouds of side body car parts and compared with the major 3-D object detection systems in terms of computational cost. Pose estimation error was calculated with both ICP and the modified point set registration schemes, and it was shown to be decreasing in the case of later. All shown results supported the research claims.

In this paper, problem of identifying and tracking the power port of remote radio unit (RRU) is addressed. The testing of RRU requires the inspection robot to insert the probes into its power and network ports. In order to solve this problem, an experimental setup of visual servoing with 6 degrees of freedom (DoF) manipulator has been established. The initial problem of recognizing and tracking the power port of RRU has been resolved using template matching and camshift tracking algorithms. Furthermore, camshift tracking algorithm has been improved to work more accurately in this application. Modified algorithm addresses the problem of swapping of major and minor axes of camshift and enhances its application to 6 DoF from 4 DoF. Experimental results have been presented to support the research claims, and computational comparison of modified tracking algorithm with camshift has been shown.

The compliant control of the robot is widely used in the Human-Machine Interface (HMI) and robot bionics. Because the system and the environment are mutually constrained, when the external environment changes, the control environment of the system will change with it, which deteriorates the control performance of the system. This article solves the problem of how to realize the compliant control with changing environment and proposes the adaptive impedance control and compensation based on a specific compliance system. We first model our compliance system by focusing on the system's internal cylinder, pressure difference transmitter its other components and then acquire its transfer function. Then we apply the impedance control and adaptive impedance control to the compliance system. Comparing these two methods, we prove that the adaptive impedance control has better tracking performance and robustness in uncertain environment. Furthermore, the stability of the adaptive impedance compliance system is proved by the Lyapunov function. Finally, we verify this algorithm with a flange experimental platform and design an experiment about contact force between the flange and different objects. The stability and practicability of the experimental algorithm are substantiated.