This paper presents TurboInfer, a system that enables power-efficient LLM inference on mobile devices. The core insight behind TurboInfer is that while LLMs are power-intensive due to their heavy computational demands, the model inference experiences unavoidable GPU stalls caused by tensor preparation for subsequent kernel executions at run-time. These GPU stalls arise from the unique host-controlled execution pipeline tailored to mobile phones and the significant DRAM access contention inherent to the shared memory architecture of mobile System-on-chips (SoCs). With LLM inference requiring hundreds to thousands of kernel executions, these short but frequent GPU stalls accumulate, accounting for over 74% of the token generation latency.

Deploying Tiny Deep Learning (TinyDL) on Internet of Things (IoT) devices is gaining popularity. To accommodate the limited memory, recent methods split tensors into fine-grained parts and plan memory offline to minimize its footprint. However, they fail to adapt to dynamic memory, missing the opportunity to utilize temporarily available memory for faster inference. Additionally, existing approaches focus solely on minimizing memory size while neglecting cache usage characteristics, resulting in frequent cache misses and increased latency. In this paper, we propose RAMS, an efficient framework supporting runtime adaptive memory scaling to fully utilize the dynamic memory. We also propose a cache-friendly memory management approach that minimizes cache miss times. RAMS includes an offline planner to minimize the memory footprint essential for inference and an online manager to determine memory sizes and generate layouts for size-controllable tensors based on available memory. RAMS significantly reduces inference latency while maintaining a compact memory footprint. Extensive experiments on commercial devices running RTOS and Android systems demonstrate that, compared to the state-of-the-art methods, RAMS can efficiently reduce latency by up to 1.57× and 1.48× compared to TFLM and TinyTS, respectively using a comparable memory footprint, while reducing power consumption by 67.74% and 15.97%.

Accurate and timely air quality prediction is crucial for cities and individuals to effectively take necessary precautions against potential air pollution. Existing studies typically rely on building prediction models based on large-scale monitoring data, often designed for specific tasks. Recently, pre-trained large language models (LLMs) have achieved significant progress in various time series analysis tasks due to their powerful representation and inference capabilities. However, their application to air quality data with spatio-temporal features remains largely unexplored. In this work, we propose LLMAir, an adaptive reprogramming approach that adapts pre-trained LLMs for air quality prediction. We first construct spatiotemporal tokens based on monitoring stations by integrating value, node, and time embeddings. Next, we design an adaptive semantic-enhanced reprogramming module to compute similarity matching scores between our spatiotemporal tokens and pre-trained word embeddings for alignment. We employ a semantic regulator to generate the optimal length of word prototypes, which serve as prompt prefixes for adaptive reprogramming and guiding the spatiotemporal token embeddings into the frozen LLM. Additionally, we jointly optimize predictive error and alignment loss to train our model. Experimental results demonstrate that LLMAir achieves state-of-the-art performance in air quality prediction and few-shot forecasting across two real-world datasets.

We present nnPerf, a real-time on-device profiler designed to collect and analyze the DNN model run-time inference latency on mobile platforms. nnPerf demystifies the hidden layers and metrics used for pursuing DNN optimizations and adaptations at the granularity of operators and kernels, ensuring every facet contributing to a DNN model's run-time efficiency is easily accessible to mobile developers via well-defined APIs. With nnPerf, the mobile developers can easily identify the bottleneck in model run-time efficiency and optimize the model architecture to meet system-level objectives (SLO). We implement nnPerf on TFLite framework and evaluate its e2e-, operator-, and kernel-latency profiling accuracy across four mobile platforms. The results show that nnPerf achieves consistently high latency profiling accuracy on both CPU (98.12%) and GPU (99.87%). Our benchmark studies demonstrate that running nnPerf on mobile devices introduces the minimum overhead to model inference, with 0.231% and 0.605% extra inference latency and power consumption. We further run a case study to show how we leverage nnPerf to migrate OFA, a SOTA NAS system, to kernel-oriented model optimization on GPUs.