GPU clusters have been widely used to co-locate various deep learning (DL) workloads in a multi-tenant way. Although such resource sharing can significantly reduce training cost, resource contention and interference among co-located workloads make task scheduling very complex and challenging. To simplify the scheduling problem, existing algorithms usually divide the procedure of scheduling into two sub-tasks, i.e., task placement and resource allocation, and allocate resources according to pre-defined and fixed resource demands. However, such a paradigm significantly constrains the selection of potential scheduling solutions. In this article, we present MAIFS, a novel multi-agent reinforcement learning based scheduling algorithm that handles task placement and resource allocation integratedly, and allows fungible resource allocation based on resource sensitivity of DL workloads. The core of MAIFS lies in two mechanisms. The multi-agent attention mechanism is designed to learn and share inter-related resource state features observed from different agents, which enables agents to explore fungible resource allocation solutions. The dynamic coordination graph mechanism is designed for coordinating interactive task placement decisions of agents during integrated scheduling, so as to mitigate potential task conflicts. Simulated experiments using two large scale production DL workload traces and physical deployment experiments based on a Kubernetes based GPU cluster show that MAIFS can outperform state-of-the-art scheduling algorithms by up to 44% in terms of makespan and 46% in terms of job completion time (JCT).

Federated Distillation (FD), a popular variant of Federated Learning (FL), has attracted researchers' attention due to its ability to support heterogeneous model training. Generally, FD allows clients to upload logits associated with public datasets for knowledge transfer, yet logits may pose privacy risks. In this study, we provide the first demonstration of the impact of privacy risks caused by logits. Specifically, we design a data reconstruction attack against logits named L-Attack which can reveal sensitive information about the target client without access to the target model. Via the zeroth-order optimization technique, L-Attack involves training a server-side generator that unveils certain features of private data owned by the target client. To defend against L-Attack, we propose a label aggregation-based FD algorithm called LabelAvg which allows clients to upload predicted hard labels for knowledge transfer instead of logits. Due to the insufficient information in labels for distillation, LabelAvg provides a voting-based label smoothing mechanism that enables the server to construct smooth labels from received labels. The generated smooth labels which stand for the consensus among all clients, indicate the approximate probability distribution. Thus, these smoothed labels bear a striking similarity to logits and can be used for distillation. Analysis and experimental results prove LabelAvg is superior to baselines in terms of accuracy, privacy, and communication data volume.

Workload prediction plays a crucial role in resource management of large scale cloud datacenters. Although quite a number of methods/algorithms have been proposed, long-term changes have not been explicitly identified and considered. Due to shifty user demands, workload re-locations, or other reasons, the “resource usage pattern” of a workload, which is usually quite stable in a short-term view, may change dynamically in a long-term range. Such long-term dynamic changes may cause significant accuracy degradation for prediction algorithms. How to handle such long-term dynamic changes is an open and challenging issue. In this article, we propose Evolution Graph for Workload Prediction (EvoGWP), a novel method that can predict long-term dynamic changes using a delicately designed graph-based evolution learning algorithm. EvoGWP automatically extracts shapelets to explicitly identify resource usage patterns of workloads in a fine-grained level, and predicts workload changes by considering factors in both temporal and spatial dimensions. We design a two-level importance based shapelet extraction mechanism to mine new usage pattern changes in temporal dimension, and design a novel evolution graph model to fuse the interference among resource usage patterns of different workloads in spatial dimension. By combining temporal extraction of shapelets from each single workload and spatial interference of shapelets among different workloads, we then design a spatio-temporal GNN-based encoder-decoder model to predict the long-term dynamic changes of workloads. Experiments using real trace data from Alibaba, Tencent and Google show that EvoGWP improves the prediction accuracy by up to 58.6% over the state-of-the-art prediction methods. Moreover, EvoGWP can outperform the state-of-the-art prediction methods in terms of model convergence. To the best of our knowledge, this is the first work that explicitly identifies fine-grained workload resource usage patterns to accurately predict long-term dynamic changes of workloads.