In edge computing environments, efficient computation resource management is crucial for optimizing service allocation to hosts in the form of containers. These environments experience dynamic user demands and high mobility, making traditional static and heuristic-based methods inadequate for handling such complexity and variability. Deep Reinforcement Learning (DRL) offers a more adaptable solution, capable of responding to these dynamic conditions. However, existing DRL methods face challenges such as high reward variability, slow convergence, and difficulties in incorporating user mobility and rapidly changing environmental configurations. To overcome these challenges, we propose a novel DRL framework for computation resource optimization at the edge layer. This framework leverages a customized Markov Decision Process (MDP) and Proximal Policy Optimization (PPO), integrating a Graph Convolutional Transformer (GCT). By combining Graph Convolutional Networks (GCN) with Transformer encoders, the GCT introduces a spatio-temporal reward-shaping mechanism that enhances the agent's ability to select hosts and assign services efficiently in real time while minimizing the overload. Our approach significantly enhances the speed and accuracy of resource allocation, achieving, on average across two datasets, a 30% reduction in convergence time, a 25% increase in total accumulated rewards, and a 35% improvement in service allocation efficiency compared to standard DRL methods and existing reward-shaping techniques. Our method was validated using two real-world datasets, MOBILE DATA CHALLENGE (MDC) and Shanghai Telecom, and was compared against standard DRL models, reward-shaping baselines, and heuristic methods. • Proposing a DRL framework that integrates reward shaping for resource management. • Introducing a novel MDP design that considers the dynamic nature of the users. • Presenting a novel reward shaping mechanism, incorporating GCN and transformers.

Deep Reinforcement Learning (DRL) has emerged as a powerful paradigm for solving complex problems. However, its full potential remains inaccessible to a broader audience due to its complexity, which requires expertise in training and designing DRL solutions, high computational capabilities, and sometimes access to pre-trained models. This necessitates the need for hassle-free services that increase the availability of DRL solutions to a variety of users. To enhance the accessibility to DRL services, this paper proposes a novel blockchain-based crowdsourced DRL as a Service (DRLaaS) framework. The framework provides DRL-related services to users, covering two types of tasks: DRL training and model sharing. Through crowdsourcing, users could benefit from the expertise and computational capabilities of workers to train DRL solutions. Model sharing could help users gain access to pre-trained models, shared by workers in return for incentives, which can help train new DRL solutions using methods in knowledge transfer. The DRLaaS framework is built on top of a Consortium Blockchain to enable traceable and autonomous execution. Smart Contracts are designed to manage worker and model allocation, which are stored using the InterPlanetary File System (IPFS) to ensure tamper-proof data distribution. The framework is tested on several DRL applications, proving its efficacy.

In this paper, we propose Value Iteration Network for Reward Shaping (VIN-RS), a potential-based reward shaping mechanism using Convolutional Neural Network (CNN). The proposed VIN-RS embeds a CNN trained on computed labels using the message passing mechanism of the Hidden Markov Model. The CNN processes images or graphs of the environment to predict the shaping values. Recent work on reward shaping still has limitations towards training on a representation of the Markov Decision Process (MDP) and building an estimate of the transition matrix . The advantage of VIN-RS is to construct an effective potential function from an estimated MDP while automatically inferring the environment transition matrix. The proposed VIN-RS estimates the transition matrix through a self-learned convolution filter while extracting environment details from the input frames or sampled graphs. Due to (1) the previous success of using message passing for reward shaping; and (2) the CNN planning behavior, we use these messages to train the CNN of VIN-RS. Experiments are performed on tabular games, Atari 2600 and MuJoCo, for discrete and continuous action space. Our results illustrate promising improvements in the learning speed and maximum cumulative reward compared to the state-of-the-art. The improvement achieved by VIN-RS can only be observed for some of the games due to the underlying nature of some environments. In terms of the studied MuJoCo games, there is on average an increase of 30% in the maximum reward reached during early stages of learning.