Dynamic sequences with varying lengths have been widely used in the training of Transformer-based large language models (LLMs). However, current training frameworks adopt a pre-defined static parallel strategy for these sequences, causing neither communication-parallelization cancellation on short sequences nor out-of-memory on long sequences. To mitigate these issues, we propose ParaDySe, a novel adaptive Parallel strategy switching framework for Dynamic Sequences. ParaDySe enables on-the-fly optimal strategy adoption according to the immediate input sequence. It first implements the modular function libraries for parallel strategies with unified tensor layout specifications, and then builds sequence-aware memory and time cost models with hybrid methods. Guided by cost models, ParaDySe selects optimal layer-wise strategies for dynamic sequences via an efficient heuristic algorithm. By integrating these techniques together, ParaDySe achieves seamless hot-switching of optimal strategies through its well-designed function libraries. We compare ParaDySe with baselines on representative LLMs under datasets with sequence lengths up to 624K. Experimental results indicate that ParaDySe addresses OOM and CPC bottlenecks in LLM training by systematically integrating long-sequence optimizations with existing frameworks.

Deep learning (DL) systems are typically used to accelerate training DL jobs. Training DL models requires feeding mass input data. It takes a long time to transfer training data from the storage nodes to the compute nodes. However, the computational resources of the GPUs are idle during the data transmission period, which results in a waste of computing resources. In DL systems, a large number of short-term jobs are queuing longer than their own execution times. Meanwhile, many multi-GPU jobs are suffering a long-queuing time due to not enough free GPU. To the best of our knowledge, no studies try to use the idle computation resources of GPU in the data transmission period.We propose Tereis, a package-based scheduler to make full use of GPU. Tereis predicts a DL job’s execution time and data transmission time, then safely package two jobs on the same GPU. One of the packaged jobs will be completed before the other job ends transferring data, which is what ‘safely’ means. This ‘safe’ Packaging does not cause GPU contention. Tereis also designs multi-level queues to prevent starvation. We implemented Tereis in python on the actual cluster and evaluated its performance. The experiment results show that Tereis decreases the average waiting time by 3.2× to10.5×, and has an improvement of 18% to 42% on makespan compared to other methods. Furthermore, we have made large-scale simulations to explore the sensitivities of Tereis. We find that Tereis performs better in the scenario where the job set has a distribution of large dataset size and jobs are submitted frequently.

Deep learning (DL) jobs normally run on GPU clusters. Some DL jobs need to be scheduled preemptively to avoid long waiting times. However, preempting a DL job is time-consuming, which consists of suspending and resuming. Suspending needs to complete the training process of the current epoch, and resuming needs to reload the model and the training data. The existing schedulers almost do not consider the overhead of preempting jobs; thus, they may preempt jobs with large time loss, increasing the waiting time and the makespan.In this paper, we present PickyMan, a preemptive scheduler to minimize the overhead of preempting jobs to reduce the average waiting time and the makespan. PickyMan has some innovations. (1) Predict execution time using network traffic and database. It predicts the execution time of a DL job by profiling the network traffic from storage nodes to computation nodes and using a database, without the requirement of allocating extra resources from the cluster. It can use profiled information of only four jobs to predict the execution times for other same-model jobs, and most of the predicted errors are less than 10%. (2) Modeling the overhead of preemption. It builds a model to predict the time loss of job suspensions and resumptions with an average error of less than 5%. (3)We abstract the problem of choosing the appropriate jobs for preemption as one of finding an ordered division of the set of running jobs and solve it quickly with a greedy algorithm. By conducting experiments on the small-scale actual cluster and making large-scale simulations, PickyMan reduces the average waiting time by 10%–92% and further reduces the makespan by up to 14%, compared to existing methods.