Advancements in edge computing and container technology have made it increasingly popular and convenient to deploy Large Language Models (LLMs) through containers at the edge. However, the limited GPU resources of edge servers make it impractical to retain the model in GPU memory for long periods due to the high memory cost, especially when they remain idle without user requests. Existing work unloads the entire idle models to reduce memory costs on edge servers, but reloading them introduces significant loading delays that affect task Quality of Service (QoS). Therefore, efficient management of idle models is a critical issue that has been largely neglected in existing research and requires urgent attention. To address this gap, this paper studies the problem of idle model management from the perspective of the trade-off between memory cost and loading delay under the QoS constraint. A novel layer-granularity model unloading method is proposed, which leverages the layered characteristics of the model. We formulate an online joint optimization problem to determine which layers to unload and when, and present a layer-granularity unloading strategy inspired by the ski rental problem to solve it. We implement a real system with layer-granularity unloading for LLMs on NVIDIA GPUs and validate the effectiveness of the proposed method. Experimental results show it effectively trades off memory cost and loading delay, improving overall performance by up to 39.6%.

Containers have emerged as a pivotal tool for service deployment in edge computing. Before running the container, an image composed of several layers must exist locally. Recent strategies have utilized layer-sharing in images to reduce deployment delays. However, existing research only focuses on a single aspect of container orchestration, like container placement, neglecting the joint optimization of the entire orchestration process. To fill in such gaps, this article introduces an online strategy that considers layer-aware container orchestration, encompassing request scheduling, container placement, and resource provision. The goal is to reduce costs, adapt to evolving user demands, and adhere to system constraints. We present an online optimization problem that accounts for various real-world factors in orchestration, including container and server expenses. An online algorithm is proposed, integrating a regularization-based approach and stepwise rounding to address this optimization problem efficiently. The regularization approach separates time-dependent container placement and server wake-up costs, requiring only current information and past decisions. The stepwise rounding process generates feasible solutions that meet system constraints, reducing computational costs. Additionally, a competitive ratio proof is provided for the proposed algorithm. Extensive evaluations demonstrate that our approach achieves about 20% performance enhancement compared to baseline algorithms.

Extending serverless computing to the edge has emerged as a promising approach to support service, but startup containerized serverless functions lead to the cold-start delay. Recent research has introduced container caching methods to alleviate the cold-start delay, including cache as the entire container or the Zygote container. However, container caching incurs memory costs. The system must ensure fast function startup and low memory cost of edge servers, which has been overlooked in the literature. This paper aims to jointly optimize startup delay and memory cost. We formulate an online joint optimization problem that encompasses container scheduling decisions, including invocation distribution, container startup, and container caching. To solve the problem, we propose an online algorithm with a competitive ratio and low computational complexity. The proposed algorithm decomposes the problem into two subproblems and solves them sequentially. Each container is assigned a randomized strategy, and these container-level decisions are merged to constitute overall container caching decisions. Furthermore, a greedy-based subroutine is designed to solve the subproblem associated with invocation distribution and container startup decisions. Experiments on the real-world dataset indicate that the algorithm can reduce average startup delay by up to 23% and lower memory costs by up to 15%.