The rapid growth of model parameters presents a significant challenge when deploying large generative models on GPU. Existing LLM runtime memory management solutions tend to maximize batch size to saturate GPU device utilization. Nevertheless, this practice leads to situations where the KV Cache of certain sequences cannot be accommodated on GPUs with limited memory capacity during the model inference, requiring temporary eviction from GPU memory (referred to as KV Cache spillover). However, without careful consideration of the LLM inference's runtime pattern, current LLM inference memory management solutions face issues like one-size-fits-all spillover handling approach for different platforms, under-utilization of GPU in prefill stage, and suboptimal sequence selection due to direct employment of swap or recomputation. In this paper, we introduce FuseSpill, a holistic KV Cache management solution designed to boost LLM inference on memory-constrained GPU by efficiently handling KV Cache spillover. Specifically, FuseSpill consists of a spillover cost model that analyzes the system cost of spillover handling techniques quantitatively, a KV cache swap orchestrator to further refine the basic swap technique to sophisticated disaggregate KV Cache across heterogeneous devices for decoding iterations, a multi-executor scheduler to effectively coordinate task executors across devices, and a response length predictor to exploit the length-aware sequence selection strategy when KV Cache spillover occurs. The experimental results demonstrate that our implementation outperforms existing solutions, delivering a 20% to 40% increase in throughput while simultaneously reducing the inference latency of the spillover sequences.

Large, sparse categorical data is a natural way to represent complex data like sequences, trees, and graphs. Such data is prevalent in many applications, e.g., Criteo released a terabyte size click log data of 4 billion records with millions of dimensions. While most existing clustering algorithms like k-Means work well on dense, numerical data, there exist relatively few algorithms that can cluster sets of sparse categorical features. In this paper, we propose a new method called k-FreqItems that performs scalable clustering over high-dimensional, sparse data. To make clustering results easily interpretable, k-FreqItems is built upon a novel sparse center representation called FreqItem which will choose a set of high-frequency, non-zero dimensions to represent the cluster. Unlike most existing clustering algorithms, which adopt Euclidean distance as the similarity measure, k-FreqItems uses the popular Jaccard distance for comparing sets. Since the efficiency and effectiveness of k-FreqItems are highly dependent on an initial set of representative seeds, we introduce a new randomized initialization method, SILK, to deal with the seeding problem of k-FreqItems. SILK uses locality-sensitive hash (LSH) functions for oversampling and identifies frequently co-occurred data in LSH buckets to determine a set of promising seeds, allowing k-FreqItems to converge swiftly in an iterative process. Experimental results over seven real-world sparse data sets show that the SILK seeding is around 1.1\sim3.2× faster yet more effective than the state-of-the-art seeding methods. Notably, SILK scales up well to a billion data objects on a commodity machine with 4 GPUs. The code is available at https://github.com/HuangQiang/k-FreqItems.

Mobile crowd sensing, also called crowd sensing over smartphones, has been an appealing paradigm for collecting sensory data over a vast urban area, due to advantages of low deployment cost and widely spatial coverage of geographically distributed smartphones or other smart devices. In the paper, we focus on a nontrivial problem of making an agreement between crowdsourcers and smartphone users to find the most efficient assignment of sensing tasks to smartphone users. However, there exist several technical challenges such as the incentive issue to encourage participation of smartphone users, preserving private information and distributed implementation in practice. Existing approaches usually have several limitations, e.g. the absence of proper incentives, the assumption of a centralized auctioneer or platform. To this end, we propose a distributed auction framework that explicitly models the interaction between crowdsourcers and smartphone users, achieving the optimal social profit and providing proper incentives to entities without disclosing their privacy as well. We demonstrate that the proposed distributed auction algorithm satisfies a lot of good properties, including optimality of social profit, computation efficiency, convergence, individual rationality through both solid theoretical analysis and extensive experiments.

Crowdsourcing distributed sensing to smartphones has become an appealing paradigm for harnessing the power of the crowd for collecting and sharing information. Market-driven auctions have been recognized as an effective way to matching sensing service demand and supply. A crowdsourcer acts as an auctioneer and smartphone workers act as bidders who are willing to supply sensing services to crowdsourcers. A number of auction mechanisms have been proposed. Unfortunately, most of them assume there is only one crowdsourcer, which is not true in the real world. In this paper, we consider the design of distributed auctions for a crowdsourced sensing market with multiple crowdsourcers and many smartphone workers. We employ a distributed reverse auction that works in a fully distributed fashion. The mechanism requires no one to release its private information. With extensive simulations, we demonstrate that our distributed mechanism achieves the optimal social welfare, and produces better performance than a competing algorithm.