The drive for privacy-preserving and low-latency artificial intelligence necessitates executing Large Language Models (LLMs) directly on heterogeneous, resource-constrained edge devices. This paradigm presents a challenge: efficiently running large models across multiple end/edge devices under a strict energy budget. Especially, the problem becomes more complicated when it comes to orchestrating multiple complex tasks using large models at the same time. To address this, we introduce FlexiTensor, an offline planning and deployment system for LLM inference, scaling from a single task to multiple concurrent tasks. FlexiTensor first considers LLM inference for a single task under a strict energy budget. We design a heuristic optimization algorithm to minimize the latency under a strict energy budget. Specifically, we select an optimized subset of devices and tensor allocations. Based on the selection, latency can be reduced by using non-uniform tensor parallelism with quantization. FlexiTensor can be extended to multi-task cases. We model the execution structure of multi-agent workflows as pre-defined Directed Acyclic Graphs (DAGs) and reformulate the problem with energy consumption constraints. FlexiTensor introduces a novel hybrid evolutionary algorithm to address joint task scheduling and resource allocation in multi-task cases. This approach navigates the vast search space of task placement, tensor splitting, and thread assignment to minimize the overall workflow makespan. All optimization algorithms run offline before deployment, using pre-profiled device characteristics to compute a static execution plan. Extensive experiments on a physical testbed of heterogeneous edge devices demonstrate that FlexiTensor significantly outperforms baselines. We accelerate single-task inference by up to 50%. For multi-task workflows, we consistently find superior scheduling solutions, achieving speed improvements by an average of 50% against competitive baselines and by up to 100% in certain scenarios, showcasing its effectiveness and adaptability for real-world edge LLM-based applications.

The Large Vision-Language Model (LVLM) integrates computer vision and natural language processing techniques, offering substantial application potential. However, these models demand extensive resources during inference. Adaptive attention techniques can dynamically reduce computational redundancy and thus improve efficiency. Although current adaptive attention methods significantly reduce the memory requirements of Transformer-based language models, they are not tailored for LVLMs. We observe that LVLMs generate responses from both remote image tokens and local text tokens, and different modalities have different attention patterns. This observation inspires us to manage the attention for each modality separately. Specifically, for visual input, we store the cache of potentially useful information but only compute the most critical parts. For language input, we care more about local information. Based on our observation and analysis of vision-language attention patterns, we develop A-VL, a plug-and-play adaptive attention tailored for LVLM inference. Extensive evaluations on three vision-language tasks and five datasets show the effectiveness of our designs. Our approach A-VL outperforms existing adaptive attention methods in reducing memory usage and computational load without compromising performance.

The advent of a large language model (LLM) has revolutionized various domains and services. The inference pipeline system is emerging as an efficient mechanism to deploy LLMs. However, existing works barely study the deployment of LLM inference on heterogeneous GPUs (with different computation and memory capabilities), where inference efficiency can be heavily affected by the imbalanced performance of different pipeline stages. Based on our empirical experience, the unbalanced pipeline stages incur GPU wait time, and average idle time can exceed 50% of the whole LLM inference process. In this paper, we study and optimize the distributed pipeline parallelism system for LLM inference on heterogeneous GPUs. We present a heuristic algorithm and implement a system that automatically deploys an efficient inference pipeline on heterogeneous GPUs. Extensive experiments are evaluated on 26 heterogeneous GPUs. The results demonstrate the superiority of our proposed system, which improves makespan (i.e., the total LLM inference latency) and throughput by an average of 37.1% and a maximum of 83.0% compared to the baselines.

Secure protocol-independent communication is increasingly demanding to support information exchange among neighbor Internet of Things (IoT) devices. For example, recent works utilize ultrasound at the resonant frequency range of a gyroscope to build communication between a speaker and the gyroscope. However, they are vulnerable to eavesdropping attacks and may have limitations in communication delays. In this work, we present WordWhisper, an efficient, word-level, and speaker-to-gyroscope communication system, with which only the target device can receive the correct information. We theoretically analyze Micro-Electro-Mechanical System (MEMS) gyroscope resonance and propose a hardware-dependent mechanism to defend against eavesdropping, making non-target gyroscopes receive ineffective information. Note that WordWhisper is free of costly data collection from gyroscopes, we train and update our decoding model based on the synthesized data (generated from theoretical MEMS resonance analysis) rather than the costly collected data from gyroscopes. Meanwhile, we address the challenge of eavesdropping when it comes to multiple attackers. We evaluate WordWhisper over 50 MEMS gyroscopes and 100 words. Extensive evaluations demonstrate that WordWhisper can achieve word-level communication with 99.33% accuracy while the recognition accuracy drops to a random guess for the non-target. Our decoding delay is lower than 0.63 seconds.