Machine Learning (ML) has become ubiquitous, fueling data-driven applications across various organizations. Contrary to the traditional perception of ML in research, ML workflows can be complex, resource-intensive, and time-consuming. Expanding an ML workflow to encompass a wider range of data infrastructure and data types may lead to larger workloads and increased deployment costs. Currently, numerous workflow engines are available (with over ten being widely recognized). This variety poses a challenge for end-users in terms of mastering different engine APIs. While efforts have primarily focused on optimizing ML Operations (MLOps) for a specific workflow engine, current methods largely overlook workflow optimization across different engines. In this work, we design and implement Couler, a system designed for unified ML workflow optimization in the cloud. Our main insight lies in the ability to generate an ML workflow using natural language (NL) descriptions. We integrate Large Language Models (LLMs) into workflow generation, and provide a unified programming interface for various workflow engines. This approach alleviates the need to understand various workflow engines' APIs. Moreover, Couler enhances workflow computation efficiency by introducing automated caching at multiple stages, enabling large workflow auto-parallelization and automatic hyperparameters tuning. These enhancements minimize redundant computational costs and improve fault tolerance during deep learning workflow training. Couler is extensively deployed in real-world production scenarios at Ant Group, handling approximately 22k workflows daily, and has successfully improved the CPU/Memory utilization by more than 15% and the workflow completion rate by around 17%.

Deep learning recommendation models (DLRM) rely on large embedding tables to manage categorical sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/CPU/memory usage. Meanwhile, tech companies have built extensive cloud-based services to accelerate training DLRM models at scale. In this paper, we conduct a deep investigation of the DLRM training platforms at AntGroup and reveal two critical challenges: low resource utilization due to suboptimal configurations by users and the tendency to encounter abnormalities due to an unstable cloud environment. To overcome them, we introduce DLRover, an elastic training framework for DLRMs designed to increase resource utilization and handle the instability of a cloud environment. DLRover develops a resource-performance model by considering the unique characteristics of DLRMs and a three-stage heuristic strategy to automatically allocate and dynamically adjust resources for DLRM training jobs for higher resource utilization. Further, DLRover develops multiple mechanisms to ensure efficient and reliable execution of DLRM training jobs. Our extensive evaluation shows that DLRover reduces job completion times by 31%, increases the job completion rate by 6%, enhances CPU usage by 15%, and improves memory utilization by 20%, compared to state-of-the-art resource scheduling frameworks. DLRover has been widely deployed at AntGroup and processes thousands of DLRM training jobs on a daily basis. DLRover is open-sourced and has been adopted by 10+ companies.

In the cloud environment, different kinds of jobs (Flink, PyTorch, TensorFlow, AI-Serving) are running in the same cluster with different service-level agreements (SLA). To manage large amounts of jobs in a cloud environment efficiently, it is critical to build a system to optimize the job performance in consideration of multiple predefined objectives. For example, one kind of optimization target is improving the resource utilization of jobs, other kinds of objectives are to guarantee the system SLA (e.g., system throughput, response time, and so on). Currently, most of the existing frameworks are working on one aspect of optimization, and can not support different kinds of optimization targets via a uniform framework or system. In Antgroup, we have designed and implemented a general framework (named Cougar) to improve jobs and cluster performance to meet such requirements. Cougar provides the ability to support different optimization scenarios like the initial and runtime optimization for one job, and cross-job optimization for multiple jobs. Nowadays, Cougar has widely used in the production environment of Antgroup including 110,000 jobs and 800,000 Pods daily, and has successfully improved the CPU/Memory/GPU utilization by more than 20% and performance (i.e., throughput or completion time or latency) by around 10%. In the end, we also like to share our best practice on how to tune Flink and Deep Learning Job (GPU collocate) in the production environment.