Running LLMs on devices like smartphones has become a key catalyst towards privacy-preserving mobile AI. In state-of-the-art frameworks, the attention operator falls back from the dedicated NPU to the public CPU/GPU due to its sensitivity to quantization. Such a fallback leads to hampered user experience and extra system scheduling complexity. To realize NPU-centric LLM inference, this paper presents shadowNPU, a system-algorithm codesigned sparse attention module with minimal reliance on CPU/GPU by only sparsely computing a very small number of tokens. The key idea is to hide the overhead of estimating the important tokens by offloading it to NPU. On top of this, it further incorporates insightful techniques including NPU compute graph bucketing, head-wise NPU-CPU/GPU pipeline and per-head fine-grained sparsity ratio to achieve high accuracy and efficiency. Compared to design alternatives, shadowNPU achieves the best performance with strictly limited CPU/GPU resource; it requires much less CPU/GPU resource to achieve on-par performance of SoTA frameworks.

Vision Language Models (VLMs) are foundational for low-latency, privacy-preserving on-device AI in real-time applications like UI agents and VQA. The VLM prefilling phase, which processes the entire visual-textual input, faces the critical challenge of a long Time-to-First-Token (TTFT). One promising approach to reduce TTFT is to exploit the temporal locality by reusing block-level computations across consecutive frames. Unfortunately, current Transformer-based VLMs break the spatial invariance of CNNs and invalidate the strict-prefix KV-cache mechanism of decoder-only LLMs; in practice, even a single-pixel mismatch can prevent reuse.

Large language models (LLMs) are renovating the mobile AI, catalyzing novel mobile applications such as UI task automation. A new paradigm of mobile AI ecosystem emerges in the LLM era: LLM as a mobile OS service (LLMaaS), where LLM runs as a system service and exposes its functionality (language understanding and generation) to third-party apps. As a giant step towards on-device LLMaaS, this work presents Libra, a system that tackles with challenge in managing persistent LLM contexts (KV cache) under tight memory constraint. Libra manages the LLM contexts based on the fine-grained, chunk-wise, globally-optimized KV cache compression and swapping. Specifically, it integrates three novel techniques: (1) Tolerance-Aware Compression applies different compression rates to each chunk based on their attention scores. (2) Swapping-Recompute Pipeline simultaneously swaps and recomputes the LLM contexts to improve the hardware resource utilization. (3) Chunk Lifecycle Management judiciously determines when and what to evict to reduce the context switching overhead. Through comprehensive experiments on various edge devices, Libra reduces the context switching latency by up to 20 × and on average 9.7 × compared to competitive baselines.

Federated Learning (FL) facilitates privacy-preserving fine-tuning of Large Language Models (LLMs) for mobile applications, termed FedLLM. A vital challenge of FedLLM is the tension between LLM complexity and resource constraint of mobile devices. In response to this challenge, we first introduceFwdLLM(our conference version), an innovative FL framework designed to enhance the FedLLM efficiency. The key idea ofFwdLLMis to employ backpropagation (BP)-free training methods, requiring devices only to execute memory-efficient “perturbed inference”. Enabled by mobile NPU acceleration and an expanded array of participating devices,FwdLLMdelivers substantially better wall-clock efficiency than BP-based FedLLM. However,FwdLLMbased on vanilla BP-free optimization theoretically requires more optimization steps to converge. In this work, we further enhanceFwdLLMtoFwdLLM+, which incorporates advanced zeroth-order optimization techniques and low-rank perturbation decomposition to reduce convergence steps. Finally, we conduct extensive experiments on 4 models (ranging from 110 M to 7B) and 8 more datasets, demonstrating thatFwdLLM+achieves up to 151× faster training, a 93$\%$memory reduction compared to vanilla BP-based FedLLM, and superior performance compared toFwdLLM, enabling efficient federated fine-tuning of billion-parameter LLMs on commodity mobile devices.

AI is making mobile applications increasingly cooler, but also introduces serious privacy risks due to the extensive user data collection. Federated learning (FL), as a privacy-preserving machine learning paradigm, enables mobile devices to collaboratively learn a shared prediction model while keeping all training data on devices. However, a key obstacle towards practical cross-device FL training is the huge energy consumption, especially for lightweight mobile devices. Prior literature mostly optimizes the convergence speed and network communication cost. In this work, we first perform the experimental analysis of improving FL performance through low-precision training with energy-friendly Digital Signal Processor (DSP) on mobile devices. Then, we demonstrate that directly integrating the state-of-the-art INT8 (8-bit integer) training algorithm and classic FL protocols will significantly degrade the model accuracy. Finally, we propose a novel FL protocol, namelyQ-FedUpdate, incorporates two critical techniques: error-compensated aggregation and pipelined batch quantization. The former can ensure the tiny model updates be accumulated and take effects, and the latter can improve the DSP cache hit rate to reduce the context switching. Extensive experiments show that,Q-FedUpdatecan effectively reduce the on-device energy consumption by 21×, and accelerate the FL convergence by 6.1× with only 2% accuracy loss.

The rapidly growing edge computing platforms, coupled with the still imperfect edge infrastructure, present an excellent opportunity for the emergence of new edge hardware. However, it remains unclear whether alternative architectures built from energy-efficient mobile System-on-Chips (SoCs) can meet the stringent performance, cost, and energy demands of modern edge workloads. In this paper, we propose a new type of edge server composed of 60 Qualcomm Snapdragon 865 mobile SoCs in a 2U rack, referred to as SoC Cluster. We demonstrate its successful deployment on existing edge cloud platforms and its ability to natively serve mobile cloud gaming services. Despite the emergence of new hardware on edge platforms and its successful operation in serving mobile cloud gaming, our trace analysis revealed low hardware utilization and significant dynamic fluctuations in usage. To assess its broader applicability, we conducted the first measurement study of SoC Cluster to reveal its ability to run two popular and modern edge applications: deep learning inference and video transcoding. We developed a cross-platform benchmark suite to evaluate throughput, latency, power consumption, and application-specific metrics like video quality. We then directly compare SoC Cluster with a traditional edge server equipped with Intel CPUs and NVIDIA GPUs in terms of energy efficiency, space efficiency, and monetary cost. Results show that SoC Cluster exhibits up to 6.5? higher energy efficiency and 7.7? higher space efficiency. We also disclose its limitations in serving computation-intensive workloads such as large deep learning models. The outcomes provide insightful implications and offer practical direction for refining SoC Cluster toward broader deployment in edge scenarios.

Linux, developed over the past three decades, has become the cornerstone of modern computing, serving as the foundation for a wide array of applications ranging from large-scale supercomputers to resource-constrained embedded devices. However, this infrastructure has long been plagued by memory and concurrency bugs, primarily due to the permissive nature of memory operations in the C language. The Rust-for-Linux ( RFL ) project offers a promising solution to address these long-standing safety concerns by integrating Rust’s advanced features, such as static ownership and type-checking, into the kernel. This integration could potentially eliminate memory and concurrency bugs while maintaining performance. Although RFL has matured and been merged into the Linux mainline, it remains understudied, and its effectiveness in balancing safety and performance is yet to be fully assessed. In this study, we present the first empirical analysis of RFL to evaluate its current state and benefits, focusing on how Rust integrates with Linux and whether this fusion ensures driver safety without significant overhead. We analyze eight RFL drivers, examining hundreds of issues and pull requests, thousands of GitHub commits, mail exchanges from the Linux mailing list, and over 12,000 discussions on Zulip. Our findings indicate that while Rust mitigates kernel vulnerabilities, it does not completely eliminate them. If not properly managed, Rust’s safety features even introduce notable runtime overhead and increased development complexity.

Small language models (SLMs) have emerged as a promising solution for deploying resource-constrained devices, such as smartphones and Web of Things. This work presents the first comprehensive study of over 60 SLMs such as Microsoft Phi and Google Gemma that are publicly accessible. Our findings show that state-of-the-art SLMs outperform 7B models in general tasks, proving their practical viability. However, SLMs’ in-context learning capabilities remain limited, and their efficiency has significant optimization potential. We identify key SLM optimization opportunities, including dynamic task-specific routing, model-hardware co-design, and vocabulary/KV cache compression. Overall, we expect the work to reveal an all-sided landscape of SLMs, benefiting the research community across algorithm, model, system, and hardware levels.

On-device inference for Large Language Models (LLMs), driven by increasing privacy concerns and advancements of mobile-sized models, has gained significant interest. However, even mobile-sized LLMs (e.g., Gemma-2B) encounter unacceptably high inference latency, often bottlenecked by the prefill stage in tasks like screen UI understanding.

GUI agents hold significant potential to enhance the experience and efficiency of human-device interaction. However, current methods face challenges in generalizing across applications (apps) and tasks, primarily due to two fundamental limitations in existing datasets. First, these datasets overlook developer-induced structural variations among apps, limiting the transferability of knowledge across diverse software environments. Second, many of them focus solely on navigation tasks, which restricts their capacity to represent comprehensive software architectures and complex user interactions. To address these challenges, we introduce GUI-Xplore, a dataset meticulously designed to enhance cross-application and cross-task generalization via an exploration-and-reasoning framework. GUI-Xplore integrates pre-recorded exploration videos providing contextual insights, alongside five hierarchically structured downstream tasks designed to comprehensively evaluate GUI agent capabilities. To fully exploit GUI-Xplore’s unique features, we propose Xplore-Agent, a GUI agent framework that combines Action-aware GUI Modeling with Graph-Guided Environment Reasoning. Further experiments indicate that Xplore-Agent achieves a 10% improvement over existing methods in unfamiliar environments, yet there remains significant potential for further enhancement towards truly generalizable GUI agents.1

Efficient workload forecasting is pivotal for both service orchestration and request dispatching in quality of service (QoS)-oriented multi-tenant edge cloud platforms (MT-ECPs) with a native tiered architecture. However, the spatial-temporal heterogeneity and structural constraints of native tiered architecture present significant challenges for the forecasting in sophisticated MT-ECPs. To tackle these challenges, we propose SCOUT, which is a novel Self-supervised learning-enhanced Cloud-edge collabOrative Unified workload forecasTing framework. First, we design a cross-granularity collaborative mechanism that enables SCOUT to balance accuracy and efficiency in forecasting within the tiered architecture of MT-ECPs. Notably, we employ an auxiliary self-supervised learning method at the cloud that enhances workload pattern representations, making them reflective of both spatial and temporal heterogeneity. Extensive experiments on two real-world workload datasets show that SCOUT outperforms state-of-the-art methods for MT-ECP's workload forecasting, decreases time consumption and reduces communication costs.

Recent advancements in integrating large language models (LLMs) with application programming interfaces (APIs) have gained significant interest in both academia and industry. Recent work demonstrates that these API-based agents exhibit relatively strong autonomy and planning capabilities. However, their ability to handle multi-dimensional difficulty levels, diverse task types, and real-world demands remains unknown. In this paper, we introduce \textsc{ShortcutsBench}, a large-scale benchmark for the comprehensive evaluation of API-based agents in solving real-world complex tasks. \textsc{ShortcutsBench} includes a wealth of real APIs from Apple Inc., refined user queries, human-annotated high-quality action sequences, detailed parameter filling values, and parameters requesting necessary input from the system or user. We revealed how existing benchmarks~/~datasets struggle to accommodate the advanced reasoning capabilities of existing more intelligent LLMs. Moreover, our extensive evaluation of agents built with $5$ leading open-source (size $\geq$ 57B) and $5$ closed-source LLMs (e.g. Gemini-1.5-Pro and GPT-4o-mini) with varying intelligence level reveals significant limitations of existing API-based agents in the whole process of handling complex queries related to API selection, parameter filling, and requesting necessary input from the system and the user. These findings highlight the great challenges that API-based agents face in effectively fulfilling real and complex user queries. All datasets, code, experimental logs, and results are available at https://github.com/EachSheep/ShortcutsBench

Mobile agents powered by large language models (LLMs) have demonstrated remarkable potential in automating operations on mobile devices. Recent studies have demonstrated that incorporating tree search methods and increasing testtime computation can enhance an agent's multi-step reasoning and planning capabilities. However, unlike simulated sandbox environments, Android is a dynamic environment with many irreversible operations, making tree search backtracking less feasible on the Android platform. To address this challenge, we propose AndroidWMSearch, a novel agent tree search framework that leverages a world model to emulate the Android environment. This framework allows the agent to evaluate and rank candidate actions through simulation before actual execution. We systematically explore this paradigm by: (1) Proposing a model-based Android tree search framework, AndroidWMSearch, in which LLMs are utilized both as world models and value functions. (2) Training specialized LLMs to act as world models, utilizing a scalable data synthesis pipeline for the training process. On the AndroidWorld benchmarks, our AndroidWMSearch surpasses the T3A agent by 4.7%, underscoring the effectiveness of our proposed framework. Moreover, utilizing our AndroidWM-7B, which is specifically trained for Android environments, as the world model results in a 3.0% performance gain compared to employing GPT-4o. These findings highlight the importance and efficacy of training a dedicated world model tailored for mobile agents.

On-device Large Language Models (LLMs) are transforming mobile AI, catalyzing applications like UI automation without privacy concerns. Nowadays the common practice is to deploy a single yet powerful LLM as a general task solver for multiple requests. We identify a key system challenge in this paradigm: current LLMs lack the elasticity to serve requests that have diversified Service-Level Objectives (SLOs) on inference latency. To tackle this, we present ElastiLM, an on-device LLM service that elasticizes both the model and the prompt dimension of a full LLM. It incorporates (1) a one-shot neuron-reordering method, which leverages the intrinsic permutation consistency in transformer models to generate high-quality elasticized sub-models with minimal runtime switching overhead; (2) a dual-head tiny language model, which efficiently and effectively refines the prompt and orchestrates the elastification between model and prompt. We implement such an elastic on-device LLM service on multiple COTS smartphones, and evaluate ElastiLM on both standalone NLP/mobile-agent datasets and end-to-end synthesized traces. On diverse SLOs, ElastiLM outperforms 7 strong baselines in (absolute) accuracy by up to 14.83% and 10.45% on average, with <1% TTFT switching overhead, on-par memory consumption and <100 offline GPU hours.

As Large Language Models (LLMs) are frequently updated, LoRA weights trained on earlier versions quickly become obsolete. The conventional practice of retraining LoRA weights from scratch on the latest model is costly, time-consuming, and environmentally detrimental, particularly as the diversity of LLMs and downstream tasks expands. This motivates a critical question: "How can we efficiently leverage existing LoRA weights to adapt to newer model versions?" To address this, we propose LoRASuite, a modular approach tailored specifically to various types of LLM updates. First, we compute a transfer matrix utilizing known parameters from both old and new LLMs. Next, we allocate corresponding layers and attention heads based on centered kernel alignment and cosine similarity metrics, respectively. A subsequent small-scale, skillful fine-tuning step ensures numerical stability. Experimental evaluations demonstrate that LoRASuite consistently surpasses small-scale vanilla LoRA methods. Notably, on backbone LLMs such as MiniCPM and Qwen, LoRASuite even exceeds the performance of full-scale LoRA retraining, with average improvements of +1.4 and +6.6 points on math tasks, respectively. Additionally, LoRASuite significantly reduces memory consumption by 5.5 GB and computational time by 78.23%.

The recent emergence of Low Earth Orbit (LEO) mega-constellations, designed for high-speed broadband connections with low latency, has introduced new deployment challenges. Efficient launch sequence planning is crucial for rapid service rollout, performance enhancement, and service promotion. However, existing research predominantly focuses on the design and performance analysis of fully-deployed constellations and overlooks the evolving process from a partially-deployed constellation to a fully-deployed one. This paper explores the launch sequence optimization problem for mega-constellations, tailored to expedite service delivery and adapt to changing performance demands. To this end, (1) we identify critical network performance metrics for the constellation evolving process and construct a simulation toolchain capable of simulating and evaluating these metrics for any potential partially-deployed constellation. (2) Drawing upon three key observations on network availability, the number of visible satellites, and latency, we propose an algorithm that can construct a launch sequence for an arbitrary mega-constellation topology. Evaluation results show that this algorithm enables the early provision of services and maximizes network performance gains at each launch batch while catering to different user demands. For instance, our algorithm can achieve network performance nearly equivalent to that of Starlink when it initiated its service, without losing redundancy, while using 55% fewer satellites.

Prior studies on traditional centralized clouds independently optimizing static resource utilization and dynamic bandwidth cost are not applicable to edge scenarios, where edge sites are interconnected by wide area networks (WAN) rather than local area networks (LAN) as within clouds. Due to the lack of knowledge about the actual status of public edge platforms and real-world edge datasets, existing influential literature on edge scenarios demonstrates significant disparities in optimization objectives and perspectives. To bridge this gap, we collaborate with a public edge platform and perform a comprehensive measurement, which reveals limitations of the status quo VM scheduling schemes and potential opportunities for improvement. However, resolving VM scheduling considering static resource utilization, dynamic bandwidth cost, and end users’ QoE in a cost-efficient manner faces several challenges, including coupled objectives, exponentially increased complexity, and spatiotemporal dynamics. To address the above challenges, in this work, we propose a holistic online framework that integrates combinatorial bandit-based VM migration and seasonality-aware VM request allocation at two distinct time granularities. Large-scale experiments based on a real-world dataset confirm that our online framework achieves near-offline bandwidth cost and resource utilization while significantly lowering time consumption.

With the rapid development of edge computing in the post-COVID19 pandemic period, precise workload forecasting is considered the basis for making full use of the edge-limited resources, and both edge service providers (ESPs) and edge service consumers (ESCs) can benefit significantly from it. Existing paradigms of workload forecasting (i.e., edge-only or cloud-only) are improper, due to failing to consider the inter-site correlations and might suffer from significant data transmission delays. With the increasing adoption of edge platforms by web services, it is critical to balance both accuracy and efficiency in workload forecasting. In this paper, we propose XELASTIC, which offers three key improvements over the conference version. First, we redesigned the aggregation and disaggregation layers using GCNs to capture more complex relationships among workload series. Second, we introduced a supervised contrastive loss to enhance robustness against outliers, particularly for handling missing or abnormal data in real-world scenarios. Finally, we expanded the evaluation with additional baselines and larger datasets. Extensive experiments on realistic edge workload datasets collected from China’s largest edge service provider (Alibaba ENS) show that XELASTIC outperforms state-of-the-art methods, decreases time consumption, and reduces communication costs.

Generative tasks, such as text generation and question answering, are essential for mobile applications. Given their inherent privacy sensitivity, executing them on devices is demanded. Nowadays, the execution of these generative tasks heavily relies on the Large Language Models (LLMs). However, the scarce device memory severely hinders the scalability of these models. We presentEdgeLLM, an efficient on-device LLM inference system for models whose sizes exceed the device's memory capacity.EdgeLLMis built atop speculative decoding, which delegates most tokens to a smaller, memory-resident (draft) LLM.EdgeLLMintegrates three novel techniques: (1) Instead of generating a fixed width and depth token tree,EdgeLLMproposes compute-efficient branch navigation and verification to pace the progress of different branches according to their accepted probability to prevent the wasteful allocation of computing resources to the wrong branch and to verify them all at once efficiently. (2) It uses a self-adaptive fallback strategy that promptly initiates the verification process when the smaller LLM generates an incorrect token. (3) To not block the generation,EdgeLLMproposes speculatively generating tokens during large LLM verification with the compute-IO pipeline. Through extensive experiments,EdgeLLMexhibits impressive token generation speed which is up to 9.3× faster than existing engines.

Large language models (LLMs) such as GPTs and Mixtral-8x7B have revolutionized machine intelligence due to their exceptional abilities in generic ML tasks. Transiting LLMs from datacenters to edge devices brings benefits like better privacy and availability, but is challenged by their massive parameter size and thus unbearable runtime costs. To this end, we presentEdgeMoE, an on-device inference engine for mixture-of-expert (MoE) LLMs – a popular form of sparse LLM that scales its parameter size with almost constant computing complexity.EdgeMoEachieves both memory- and compute-efficiency by partitioning the model into the storage hierarchy: non-expert weights are held in device memory; while expert weights are held on external storage and fetched to memory only when activated. This design is motivated by a key observation that expert weights are bulky but infrequently used due to sparse activation. To further reduce the expert I/O swapping overhead,EdgeMoEincorporates two novel techniques: (1) expert-wise bitwidth adaptation that reduces the expert sizes with tolerable accuracy loss; (2) expert preloading that predicts the activated experts ahead of time and preloads it with the compute-I/O pipeline. On popular MoE LLMs and edge devices,EdgeMoEshowcase significant memory savings and speedup over competitive baselines.

Intelligent applications rely significantly on the live machine learning pipeline, a couple of deep neural network (DNN) inference services, executed on mobile devices to meet functional requirements while ensuring user data privacy. However, executing these DNN services on resource-constrained mobile devices presents a considerable challenge: low throughput and high energy consumption of inference tasks. To address this issue, we proposeNiagara+, a novel system designed to enhance throughput by jointly scheduling DNN inference services across heterogeneous processors on mobile devices and offloading services to powerful edge servers. To achieve this,Niagara+encounters two critical challenges: unpredictable workload dynamics and high scheduling complexity. To effectively tackle these challenges,Niagara+employs a predictive model to forecast incoming workload patterns and orchestrates service allocation across device heterogeneous processors and edge servers through a combination of two-step offline scheduling optimization and online service dispatching strategies. We implementedNiagara+and conducted comprehensive experiments, demonstrating its superiority over state-of-the-art approaches, reducing DNN service latency by up to 2.6× under high-bandwidth networks and 9.1× under low-bandwidth networks, while consistently meeting stringent inference latency requirements.

The rapid growth of Low Earth Orbit (LEO) satellites, particularly with the increasing deployment of intelligent computing capabilities using commercial off-the-shelf (COTS) hardware, presents significant opportunities to enhance the quality of in-orbit services. However, the current onboard conditions remain insufficient to enhance model accuracy by increasing model size, and inadequate accuracy hampers the effectiveness of in-orbit services. The satellite-ground federated learning (FL) paradigm, leveraging collaborative fine-tuning, offers a promising solution to continuously improve onboard model performance. Prior studies have focused on optimizing fine-tuning under constraints like limited bandwidth and computational resources, they often overlook two critical challenges: the scarcity and skewness of labeled onboard data and the long revisit cycles of satellites. To address these challenges and better support in-orbit services, this paper designs a realistic simulation methodology for the onboard fine-tuning process and conducts a comprehensive measurement study. Based on insights from the measurement results, we propose an efficient satellite-ground federated fine-tuning system,FedCLR+. In this system, we design a FedCLR algorithm to enhance system accuracy through representation optimization. Additionally, we propose a hybrid bias-compensated strategy to further mitigate accuracy loss by enriching the diversity of aggregation information. Experimental results show thatFedCLR+significantly enhances accuracy by up to 21.61×, reduces transmission volume by an average of 7.29%, and maintaining acceptable additional overhead compared to baselines.

SoC-Cluster, a novel server architecture composed of massive mobile system-on-chips (SoCs), is gaining popularity in industrial edge computing due to its energy efficiency and compatibility with existing mobile applications. However, we observe that the deployed SoC-Cluster servers are not fully utilized, because the hosted workloads are mostly user-triggered and have significant tidal phenomena. To harvest the free cycles, we propose to co-locate deep learning tasks on them.

Federated learning (FL) has been widely adopted as a privacy-preserving model training paradigm. However, traditional FL protocol heavily relies on data transmission between clients and servers across the wide-area network (WAN), which is tightly constrained and unreliable, therefore causing expensive communication and slow convergence. To this end, we propose a LAN-aware FL (LanFL) protocol, which can efficiently leverage the network capacity of the local-area network (LAN). By frequent model aggregation among the devices within the same LAN, we can significantly reduce the global aggregation across WAN, thus accelerating the training process. However, due to the unique challenges introduced by LAN, it’s not easy to efficiently utilize LAN resources while preserving the original dignity of FL performance. Therefore, LanFL also incorporates several critical techniques: LAN-aware hierarchical aggregation, intraLAN device topology construction, and inter-LAN heterogeneous bandwidth coordination. Extensive real-world experiments are conducted and the experimental results show that LanFL can significantly accelerate FL training up to $6.0 \times$, while preserving the model accuracy.

With the rapid proliferation of large Low Earth Orbit (LEO) satellite constellations, a huge amount of in-orbit data is generated and needs to be transmitted to the ground for processing. However, traditional LEO satellite constellations, which downlink raw data to the ground, are significantly restricted in transmission capability. Orbital edge computing (OEC), which exploits the computation capacities of LEO satellites and processes the raw data in orbit, is envisioned as a promising solution to relieve the downlink burden. Yet, with OEC, the bottleneck is shifted to the inelastic computation capacities. The computational bottleneck arises from two primary challenges that existing satellite systems have not adequately addressed: the inability to process all captured images and the limited energy supply available for satellite operations. In this work, we seek to fully exploit the scarce satellite computation and communication resources to achieve satellite-ground collaboration and present a satellite-ground collaborative system named TargetFuse for onboard object detection. TargetFuse incorporates a combination of techniques to minimize detection errors under energy and bandwidth constraints. Extensive experiments show that TargetFuse can reduce detection errors by 3.4× on average, compared to onboard computing. TargetFuse achieves a 9.6× improvement in bandwidth efficiency compared to the vanilla baseline under the limited bandwidth budget constraint.

In the wake of the rapid deployment of large-scale low-Earth orbit satellite constellations, exploiting the full computing potential of Commercial Off-The-Shelf (COTS) devices in these environments has become a pressing issue. However, understanding this problem is far from straightforward due to the inherent differences between the terrestrial infrastructure and the satellite platform in space. In this paper, we take an important step towards closing this knowledge gap by presenting the first measurement study on the thermal control, power management, and performance of COTS computing devices on satellites. Our measurements reveal that the satellite platform and COTS computing devices significantly interplay in terms of the temperature and energy, forming the main constraints on satellite computing. Further, we analyze the critical factors that shape the characteristics of onboard COTS computing devices. We provide guidelines for future research on optimizing the use of such devices for computing purposes. Finally, we have released the datasets to facilitate further study in satellite computing.

In the current AI era, mobile devices such as smartphones are tasked with executing a myriad of deep neural networks (DNNs) locally. It presents a complex landscape, as these models are highly fragmented in terms of architecture, operators, and implementations. Such fragmentation poses significant challenges to the co-optimization of hardware, systems, and algorithms for efficient and scalable mobile AI.

With the emergence and continuous prosperity of large language models (LLMs), artificial intelligence (AI) agents have experienced rapid advancements. Most mobile AI agents merely imitate human operations, executing actions based on the human user interface (UI). The restricted input impairs the efficiency and accuracy of mobile tasks. We propose an unexplored approach: learning from the source code. Source code is the plain interaction for mobile applications, which can be used to enhance the UI understanding of mobile agents, improve action execution accuracy, and reduce the average action completion steps. The implementation of the agent prototype is preliminary evaluated on 5 open-source applications and 22 tasks, reducing the average number of task completion steps by 54%.

Speech serves as a ubiquitous input interface for embedded mobile devices. Cloud-based solutions, while offering powerful speech understanding services, raise significant concerns regarding user privacy. To address this, disentanglement-based encoders have been proposed to remove sensitive information from speech signals without compromising the speech understanding functionality. However, these encoders demand high memory usage and computation complexity, making them impractical for resource-constrained wimpy devices. Our solution is based on a key observation that speech understanding hinges on long-term dependency knowledge of the entire utterance, in contrast to privacy-sensitive elements that are short-term dependent. Exploiting this observation, we propose SILENCE, a lightweight system that selectively obscuring short-term details, without damaging the long-term dependent speech understanding performance. The crucial part of SILENCE is a differential mask generator derived from interpretable learning to automatically configure the masking process. We have implemented SILENCE on the STM32H7 microcontroller and evaluate its efficacy under different attacking scenarios. Our results demonstrate that SILENCE offers speech understanding performance and privacy protection capacity comparable to existing encoders, while achieving up to 53.3$\times$ speedup and 134.1$\times$ reduction in memory footprint.

Small satellites (SmallSats) are now widely used in various fields, such as real-time communication and earth observation. These increasingly complex space applications face limited support from conventional radiation-hardened processors onboard. Hence, many SmallSats are designed to utilize high performance commercial off-the-shelf (COTS) computing devices to address this problem but it remains unclear how the unique energy and thermal characteristics of SmallSats impact computing efficiency onboard. This work conducts a systematic and quantitative measurement study of COTS devices’ computing efficiency on two real orbiting SmallSats. The key findings are: 1) inadequate energy management may lead to electricity wastage in sunlit zones and shortages in eclipse zones, impacting onboard computing availability and 2) the weak heat dissipation onboard may compromise COTS computing efficiency by incurring thermal throttling. To address such challenges, we design ProScale, a lightweight application-aware power management and thermal control system to improve computing efficiency under both electrical and thermal energy constraints. Evaluation shows that ProScale can improve the average task completion latency by $2.1 \times$ for computation-intensive applications compared with baselines.

On-device training Neural Networks (NNs) has been a crucial catalyst towards privacy-preserving and personalized mobile intelligence. Recently, a novel training paradigm, namely Parameter-Efficient Training (PET), is attracting attention in both the machine learning and system community. In our preliminary measurements, we find PET well-suited for on-device scenarios; yet, its parameter efficiency does not translate coequal to time efficiency on resource-constrained devices, as the training time is dominated by the frozen layers.

The emergent large language/multimodal models facilitate the evolution of mobile agents, especially in mobile UI task automation. However, existing evaluation approaches, which rely on human validation or established datasets to compare agent-predicted actions with predefined action sequences, are unscalable and unfaithful. To overcome these limitations, this paper presents LlamaTouch, a testbed for on-device mobile UI task execution and faithful, scalable task evaluation. By observing that the task execution process only transfers UI states, LlamaTouch employs a novel evaluation approach that only assesses whether an agent traverses all manually annotated, essential application/system states. LlamaTouch comprises three key techniques: (1) On-device task execution that enables mobile agents to interact with realistic mobile environments for task execution. (2) Fine-grained UI component annotation that merges pixel-level screenshots and textual screen hierarchies to explicitly identify and precisely annotate essential UI components with a rich set of designed annotation primitives. (3) A multi-level application state matching algorithm that utilizes exact and fuzzy matching to accurately detect critical information in each screen, even with unpredictable UI layout/content dynamics. LlamaTouch currently incorporates four mobile agents and 496 tasks, encompassing both tasks in the widely-used datasets and our self-constructed ones to cover more diverse mobile applications. Evaluation results demonstrate LlamaTouch’s high faithfulness of evaluation in real-world mobile environments and its better scalability than human validation. LlamaTouch also enables easy task annotation and integration of new mobile agents. Code and dataset are publicly available at https://github.com/LlamaTouch/LlamaTouch.

AI is making the Web an even cooler place, but also introduces serious privacy risks due to the extensive user data collection. Federated learning (FL), as a privacy-preserving machine learning paradigm, enables mobile devices to collaboratively learn a shared prediction model while keeping all training data on devices. However, a key obstacle towards practical cross-device FL training is huge energy consumption, especially for lightweight mobile devices. In this work, we perform the first-of-its-kind analysis of improving FL performance through low-precision training with an energy-friendly Digital Signal Processor (DSP) on mobile devices. We first demonstrate that directly integrating the state-of-the-art INT8 (8-bit integer) training algorithm and classic FL protocols will significantly degrade the model accuracy. Moreover, we observe that there are still unavoidable frequent quantization operations on devices that cause extreme load stress on DSP-enabled INT8 training. To address the above challenges, we present Q-FedUpdate, an FL framework that efficiently preserves model accuracy with ultra-low energy consumption. It maintains a global full-precision model and allows the tiny model updates to be continuously accumulated, instead of being erased by the quantization. Furthermore, it introduces pipelining technology to parallel CPU-based quantization and DSP-enabled training, which reduces the floating-point computation overhead of frequent data quantization. Extensive experiments show that Q-FedUpdate can effectively reduce the on-device energy consumption by 21×, and accelerate the FL convergence by 6.1× with only 2% accuracy loss.

Privacy, security and data governance constraints rule out a brute force process in the integration of cross-silo data, which inherits the development of the Internet of Things. Federated learning is proposed to ensure that all parties can collaboratively complete the training task while the data is not out of the local. Vertical federated learning is a specialization of federated learning for distributed features. To preserve privacy, homomorphic encryption is applied to enable encrypted operations without decryption. Nevertheless, together with a robust security guarantee, homomorphic encryption brings extra communication and computation overhead. In this paper, we analyze the current bottlenecks of vertical federated learning under homomorphic encryption comprehensively and numerically. We propose a straggler-resilient and computation-efficient accelerating system that reduces the communication overhead in heterogeneous scenarios by 65.26% at most and reduces the computation overhead caused by homomorphic encryption by 40.66% at most. Our system can improve the robustness and efficiency of the current vertical federated learning framework without loss of security.

The emergence of next-generation latency-critical applications places strict requirements on network latency and stability. Edge cloud, an instantiated paradigm for edge computing, is gaining more and more attention due to its benefits of low latency. In this work, we make an in-depth investigation into the network QoS, especially end-to-end latency, at both spatial and temporal dimensions on a nationwide edge computing platform. Through the measurements, we collect a multi-variable large-scale real-world dataset on latency. We then quantify how the spatial-temporal factors affect the end-to-end latency, and verify the predictability of end-to-end latency. The results reveal the limitation of centralized clouds and illustrate how could edge clouds provide low and stable latency. Our results also point out that existing edge clouds merely increase the density of servers and ignore spatial-temporal factors, so they still suffer from high latency and fluctuations. Based on a quantified latency impact factor, we have proposed several optimization strategies for edge cloud latency and validated their effectiveness. We also propose a robust prototype edge cloud model based on lessons we learn from the measurement and evaluate its performance in the production environment. Evaluation result shows that edge clouds achieve 84.1% latency reduction with 0.5 ms latency fluctuation and 73.3% QoS improvement compared with the centralized clouds.

Seamless cross-edge migration for real-time rendering applications is challenging. The strong interactive nature of real-time rendering applications demands a downtime lower than 15ms to achieve an imperceptible migration. Existing methods based on virtual machine migration and container migration suffer from unpleasant downtime brought by dirty page retransmission-induced repeated memory data copy and the shared storage failure-induced extensive disk data copy. In this paper, we propose Cloud-assisted Service Migration (CSM) which leverages cloud-edge collaboration to achieve seamless service migration for real-time rendering applications. CSM improves service migration user experience in three folds: First, it introduces a dual rendering mechanism to bypass the peer-to-peer data copy and compresses the freezing stage. Second, a user equipment-centric session switch mechanism is proposed to save time by well coordinating application session switches and 5G user plane session switches. Third, a smooth switching mechanism is leveraged to prevent unpleasant frame flickers during session switching. We implement CSM in edge-rendering multiplayer games and deploy it on a 5G test bed with a full-stack user plane protocol stack. The evaluation results show that CSM can reduce downtime to < 14ms and the service migration process is user imperceptible.

In the realm of industrial edge computing, a novel server architecture known as SoC-Cluster, characterized by its aggregation of numerous mobile systems-on-chips (SoCs), has emerged as a promising solution owing to its enhanced energy efficiency and seamless integration with prevalent mobile applications. Despite its advantages, the utilization of SoC-Cluster servers remains unsatisfactory, primarily attributed to the tidal patterns of user-initiated workloads. To address such inefficiency, we introduceSoCFlow+, a pioneering framework designed to facilitate the co-location of deep learning training tasks on SoC-Cluster servers, thereby optimizing resource utilization.SoCFlow+incorporates three novel techniques tailored to mitigate the inherent limitations of commercial SoC-Cluster servers. First, it employs group-wise parallelism complemented by delayed aggregation, a strategy engineered to enhance the training efficiency and scalability of deep learning models, effectively circumventing network bottlenecks. Second, it integrates a data-parallel mixed-precision training algorithm, optimized to exploit the heterogeneous processing capabilities inherent to mobile SoCs fully. Third,SoCFlow+employs an underclocking-aware workload re-balanacing mechanism to tackle the training performance degradation caused by the thermal control of mobile SoCs. Through rigorous experimental validation,SoCFlow+achieves a convergence speedup ranging from 1.6× to 740× across 32 SoCs, compared to conventional benchmarks. Furthermore, when juxtaposed with commodity GPU servers (e.g., NVIDIA V100) under identical power constraints,SoCFlow+not only exhibits comparable training speed but also achieves a remarkable reduction in energy consumption by a factor of 2.31× to 10.23×, all while preserving convergence accuracy.

Federated learning (FL) becomes a promising machine learning paradigm. The impact of heterogeneous hardware specifications and dynamic states on the FL process has not yet been studied systematically. This paper presents the first large-scale study of this impact based on real-world data collected from 136k smartphones. We conducted extensive experiments on our proposed heterogeneity-aware FL platform namelyFLASH, to systematically explore the performance of state-of-the-art FL algorithms and key FL configurations in heterogeneity-aware and -unaware settings, finding the following. (1) Heterogeneity causes accuracy to drop by up to 9.2% and convergence time to increase by 2.32×. (2) Heterogeneity negatively impacts popular aggregation algorithms, e.g., the accuracy variance reduction brought byq-FedAvgdrops by 17.5%. (3) Heterogeneity does not worsen the accuracy loss caused by gradient-compression algorithms significantly, but it compromises the convergence time by up to 2.5×. (4) Heterogeneity hinders client-selection algorithms from selecting wanted clients, thus reducing effectiveness. e.g., the accuracy increase brought by the state-of-the-art client-selection algorithm drops by 73.9%. (5) Heterogeneity causes the optimal FL hyper-parameters to drift significantly. More specifically, the heterogeneity-unaware setting favors looser deadline and higher reporting fraction to achieve better training performance. (6) Heterogeneity results in non-trivial failed clients (more than 10%) and leads to participation bias (the top 30% of clients contribute 86% of computations). Our FLASH platform and data have been publicly open sourced.

Large constellations of Low Earth Orbit (LEO) satellites have been launched for Earth observation and satellite-ground communication, which collect massive imagery and sensor data. These data can enhance the AI capabilities of satellites to address global challenges such as real-time disaster navigation and mitigation. Prior studies proposed leveraging federated learning (FL) across satellite-ground to collaboratively train a share machine learning (ML) model in a privacy-preserving mechanism. However, they mostly focus on single unique challenges such as limited ground-to-satellite bandwidth, short connection window, and long connection cycle, while ignoring the completeness of these challenges in deploying efficient FL frameworks in space. In this paper, we propose an efficient satellite-ground FL framework, SatelliteFL, to address these three challenges collectively. Its key idea is to ensure that each satellite must complete per-round training within each connection window. Moreover, we design a progressive block-wise quantization algorithm that determines a unique bitwidth for each block of the ML model to maximize the model utility while not exceeding the connection window. We evaluate SatelliteFL by plugging an implemented FL platform into real-world satellite networks and satellite images. The results show that SatelliteFL highly accelerates the convergence by up to 2.8× and improves the bandwidth utilization ratio by up to 9.3× compared to the state-of-the-art methods.

Deploying deep learning (DL) on mobile devices has been a notable trend in recent years. To support fast inference of on-device DL, DL libraries play a critical role as algorithms and hardware do. Unfortunately, no prior work ever dives deep into the ecosystem of modern DL libraries and provides quantitative results on their performance. In this paper, we first build a comprehensive benchmark that includes 6 representative DL libraries and 15 diversified DL models. Then we perform extensive experiments on 10 mobile devices, and the results reveal the current landscape of mobile DL libraries. For example, we find that the best-performing DL library is severely fragmented across different models and hardware, and the gap between DL libraries can be rather huge. In fact, the impacts of DL libraries can overwhelm the optimizations from algorithms or hardware, e.g., model quantization and GPU/DSP-based heterogeneous computing. Motivated by the fragmented performance of DL libraries across models and hardware, we propose an effective DL Library selection framework to obtain the optimal library on a new dataset that has been created. We evaluate the DL Library selection algorithm, and the results show that the framework at it can improve the prediction accuracy by about 10% than benchmark approaches on average.

Deploying Latency-Critical (LC) services and Best-Effort (BE) services together is expected to improve resource utilization in edge clouds. However, co-locating LC and BE services on edge clouds presents unique challenges. Unlike cloud datacenters, edge clouds are heterogeneous, resource-constrained, and geographically distributed, leading to fiercer competition for resources and greater difficulty in balancing fluctuating co-located workloads. Due to the lack of consideration for the characteristics of edge environments, previous solutions designed for cloud datacenters are no longer applicable. To address these challenges, we introduceTango, a harmonious scheduling framework forKubernetes-based edge cloud systems with mixed services.Tangoincorporates novel components and mechanisms for elastic resource allocation on the edge, as well as two traffic scheduling algorithms that efficiently manage distributed edge resources.Tangofosters harmony not only by supporting compatible mixed services but also by offering collaborative solutions that complement each other. Based on a non-intrusive design forKubernetes,Tangofurther enhances it with automatic scaling and traffic scheduling capabilities. Compared to state-of-the-art approaches, experiments on large-scale hybrid edge clouds, driven by real workload traces, show thatTangoimproves system resource utilization by 36.9%, QoS-guarantee satisfaction rate by 11.3%, and throughput by 47.6%.

The rapid development of Low Earth Orbit (LEO) satellite constellations offers significant potential for in-orbit services, particularly in mitigating the impact of sudden natural disasters. However, the massive data collected by these satellites are often large and severely constrained by limited transmission capabilities when sending data to the ground. Satellite computing, which utilizes onboard computational capacity to process data before transmission, presents a promising solution to alleviate the downlink burden. Nonetheless, this paradigm introduces another bottleneck: limited onboard computing capacity, resulting in slow in-orbit processing and poor results. Current satellite computing systems struggle to efficiently address both data transmission and computing bottlenecks, particularly for urgent disaster services that demand accurate and timely results. Thus, we introduce an efficient satellite computing system designed to jointly mitigate these bottlenecks, thereby providing better service. The core idea is to utilize onboard computing capacity for swift in-orbit annotation of image regions, enabling adaptive compression and download based on annotation confidence and perceived downlink availability. Once the data is downloaded, image restoration and re-inference are performed on the ground to enhance accuracy. Compared to satellite-only inference, our system demonstrates an average improvement in inference accuracy of 3.8%. Furthermore, compared to ground-only inference, with only a 2.8% accuracy loss, our system achieves a 38.4% reduction in response time and saves 71.6% of downlink volume on average.

Co-locating Latency-Critical (LC) and Best-Effort (BE) services in edge-clouds is expected to enhance resource utilization. However, this mixed deployment encounters unique challenges. Edge-clouds are heterogeneous, distributed, and resource-constrained, leading to intense competition for edge resources, making it challenging to balance fluctuating co-located workloads. Previous works in cloud datacenters are no longer applicable since they do not consider the unique nature of edges. Although very few works explicitly provide specific schemes for edge workload co-location, these solutions fail to address the major challenges simultaneously.

Differential privacy (DP) enables model training with a guaranteed bound on privacy leakage, therefore is widely adopted in federated learning (FL) to protect the model update. However, each DP-enhanced FL job accumulates privacy leakage, which necessitates a unified platform to enforce a global privacy budget for each dataset owned by users. In this work, we present a novel DP-enhanced FL platform that treats privacy as a resource and schedules multiple FL jobs across sensitive data. It first introduces a novel notion of device-time blocks for distributed data streams. Such data abstraction enables fine-grained privacy consumption composition across multiple FL jobs. Regarding the non-replenishable nature of the privacy resource (that differs it from traditional hardware resources like CPU and memory), it further employs an allocation-then-recycle scheduling algorithm. Its key idea is to first allocate an estimated upper-bound privacy budget for each arrived FL job, and then progressively recycle the unused budget as training goes on to serve further FL jobs. Extensive experiments show that our platform is able to deliver up to 2.1× as many completed jobs while reducing the violation rate by up to 55.2% under limited privacy budget constraint.

Federated learning (FL) has emerged as a prominent paradigm enabling collaborative model training without transmitting local data, thereby safeguarding data privacy. However, the practical implementation of FL systems on these devices faces a significant challenge: the heterogeneous corruption of data on individual clients, leading to unanticipated accuracy degradation during real-world deployment. In this work, we first introduce a realistic data corruption simulation framework to test the robustness of FL systems. In this framework, an in-depth analysis of potential data corruption patterns occurring on devices is conducted, followed by the construction of individual datasets with varying corruption types and degrees. Such data corruption results in the robustness degradation of conventional FL protocol (FedAVG) significantly higher than centralized learning (CL). Atop this key observation, we propose an adaptive FL protocol that emulates the CL training process. The protocol leverages imbalanced client data sampling to mitigate the negative impact of data corruption. Furthermore, a hybrid aggregation strategy is designed to accelerate model convergence and reduce additional communication overhead. Extensive experiments validate the effectiveness of our approach in enhancing the robustness of FL systems against client data corruption, which achieves up to 12% higher converge accuracy than FedAVG-based systems with acceptable overhead.

Edge clouds have become a de-facto paradigm to deliver low and stable networks to delay-critical applications such as web services and AR/VR. A unique form of edge clouds is those crowdsourced from third parties, e.g., idle PCs or workstations. Such crowdsourced edge platforms can better sink computations closer to users, reduce the purchase cost, and eliminates the carbon generated during manufacturing. Yet, they also face the challenge of out-of-control hardware, e.g., a server dropping in/out anytime. In this paper, we perform the first-of-its-kind measurement of Quality of Service (QoS) for a large-scale crowdsourced edge platform, which covers over 10,000 edge servers, 100,000 users and 10,000,000 user requests. The measurement takes a holistic QoS view: (1) First, we look at how much hardware resources are provided by edge servers, how much time they are available for service deployment, and what are the major abnormal behaviors. (2) Second, we analyze the factors affecting service stability and quantify the resource utilization pattern of containerized services hosted on those edge servers. (3) Third, we investigate the spatial and temporal features of user requests handled by the platform. Many useful and somehow surprising findings are obtained through the above measurements. We also derive insightful implications that could help edge platforms and edge applications to better deliver their services to users.

The emergence of next-generation latency-critical applications places strict requirements on network latency and stability. Edge cloud, an instantiated paradigm for edge computing, is gaining more and more attention due to its benefits of low latency. In this work, we make an in-depth investigation into the network QoS, especially end-to-end latency, at both spatial and temporal dimensions on a nationwide edge computing platform. Through the measurements, we collect a multi-variable large-scale real-world dataset on latency. We then quantify how the spatial-temporal factors affect the end-to-end latency, and verified the predictability of end-to-end latency. The results reveal the limitation of centralized clouds and illustrate how could edge clouds provide low and stable latency. Our results also point out that existing edge clouds merely increase the density of servers and ignore spatial-temporal factors, so they still suffer from high latency and fluctuations. Based on the observations, we propose a robust prototype edge cloud model based on lessons we learn from the measurement and evaluate its performance in the production environment. The further evaluation result shows that edge clouds achieve 84.1% latency reduction with 0.5ms latency fluctuation and 73.3% QoS improvement compared with the centralized clouds.

Transformer-based pre-trained models have revolutionized NLP for superior performance and generality. Fine-tuning pre-trained models for downstream tasks often requires private data, for which federated learning is the de-facto approach (i.e., FedNLP). However, our measurements show that FedNLP is prohibitively slow due to the large model sizes and the resultant high network/computation cost. Towards practical FedNLP, we identify as the key building blocks adapters, small bottleneck modules inserted at a variety of model layers. A key challenge is to properly configure the depth and width of adapters, to which the training speed and efficiency is highly sensitive. No silver-bullet configuration exists: the optimal choice varies across downstream NLP tasks, desired model accuracy, and mobile resources. To automate adapter configuration, we propose AdaFL1, a framework that enhances the existing FedNLP with two novel designs. First, AdaFL progressively upgrades the adapter configuration throughout a training session; the principle is to quickly learn shallow knowledge by only training fewer and smaller adapters at the model's top layers, and incrementally learn deep knowledge by incorporating deeper and larger adapters. Second, AdaFL continuously profiles future adapter configurations by allocating participant devices to trial groups. Extensive experiments show that AdaFL can reduce FedNLP's model convergence delay to no more than several hours, which is up to 155.5× faster compared to vanilla FedNLP and 48× faster compared to strong baselines.

Natural language processing (NLP) sees rich mobile applications. To support various language understanding tasks, a foundation NLP model is often fine-tuned in a federated, privacy-preserving setting (FL). This process currently relies on at least hundreds of thousands of labeled training samples from mobile clients; yet mobile users often lack willingness or knowledge to label their data. Such an inadequacy of data labels is known as a few-shot scenario; it becomes the key blocker for mobile NLP applications.

DNNs are ubiquitous on edge devices nowadays. With its increasing importance and use cases, it's not likely to pack all DNNs into device memory and expect that each inference has been warmed up. Therefore, cold inference, the process to read, initialize, and execute a DNN model, is becoming commonplace and its performance is urgently demanded to be optimized. To this end, we present NNV12, the first on-device inference engine optimizing cold inference. NNV12 is built atop three novel optimization knobs: selecting a proper kernel (i.e., operator implementation) for each DNN operator, bypassing the weights transformation process by caching the post-transformed weights on disk, and pipelined execution of many kernels on asymmetric processors. To tackle with the huge search space, NNV12 employs a heuristic-based scheme to obtain a near-optimal kernel scheduling plan. We fully implement a prototype of NNV12 and evaluate its performance across extensive experiments. It shows that NNV12 achieves up to 15.2× speedup compared to the state-of-the-art DNN engines on edge CPUs and 401.5× speedup on edge GPUs, respectively.

With the rapid development of edge computing in the post-COVID19 pandemic period, precise workload forecasting is considered the basis for making full use of the edge limited resources, and both edge service providers (ESPs) and edge service consumers (ESCs) can benefit significantly from it. Existing paradigms of workload forecasting (i.e., edge-only or cloud-only) are improper, due to failing to consider the inter-site correlations and might suffer from significant data transmission delays. With the increasing adoption of edge platforms by web services, it is critical to balance both accuracy and efficiency in workload forecasting. In this paper, we propose ELASTIC, which is the first study that leverages a cloud-edge collaborative paradigm for edge workload forecasting with multi-view graphs. Specifically, at the global stage, we design a learnable aggregation layer on each edge site to reduce the time consumption while capturing the inter-site correlation. Additionally, at the local stage, we design a disaggregation layer combining both the intra-site correlation and inter-site correlation to improve the prediction accuracy. Extensive experiments on realistic edge workload datasets collected from China’s largest edge service provider show that ELASTIC outperforms state-of-the-art methods, decreases time consumption, and reduces communication cost.

Video streaming applications (VSAs) are increasingly being deployed on large-scale edge platforms, which have the potential to significantly improve the quality of service (QoS) and end-user experience (QoE), ultimately maximizing business outcomes. However, there is currently very little understanding of how QoS, QoE, and the impact of QoS on QoE for VSAs on edge platforms in the wild and at scale. To close the knowledge gap, we collect SNESet, an active measurement dataset comprising QoS and QoE telemetry metrics of 8 VSAs over four months, covering end-users from 798 edge sites,30 cities, and 3 ISPs in one country.We characterize and compare the QoS and QoE metrics in SNESet with existing publicly available datasets, highlighting that SNESet includes a significantly greater number of metrics (horizontal diversity and vertical hierarchy) and provides more comprehensive coverage of specific metrics.Moreover, we qualitatively and quantitatively analyze the impact of QoS on QoE in both domain-general and domain-specific scenarios. Our findings can inform the system design decisions that different entities in the video ecosystem (content providers, video player designers, third-party optimizers, edge vendors) make to maximize end-users experience and ultimately maximize the business outcomes. We hope SNESet can attract more research efforts in the data management community, computer network community, and beyond.

Edge servers are key to the success of edge computing. Compared to cloud servers, edge servers suffer from a more constrained and costly electricity supply due to their dense, near-population deployment. Towards higher energy efficiency, we propose an extreme design of edge servers - SoC-Cluster that consists of massive, inter-connected ARM SoCs. Indeed, such SoC-Clusters have already been adopted to serve the cloud gaming application in the wild. In this paper, we present a concrete implementation of a COTS SoC-Cluster and its hardware specifications. We then discuss the potential killer applications that such SoC-Cluster can well serve and the major challenges to be solved. We also dive deep into two of such applications (live video transcoding and deep learning serving) and carry out a measurement study to demystify the application performance of SoC-Cluster. The results reveal that, compared to traditional servers, SoC-Cluster not only can reduce energy consumption but even deliver higher workload throughput in certain scenarios. Finally, we conclude the paper and discuss the primary research directions that can be explored by our community from applications, software, and hardware aspects.

This paper proposes Mandheling, the first system that enables highly resource-efficient on-device training by orchestrating mixed-precision training with on-chip Digital Signal Processor (DSP) offloading. Mandheling fully explores the advantages of DSP in integer-based numerical calculations using four novel techniques: (1) a CPU-DSP co-scheduling scheme to situationally mitigate the overhead from DSP-unfriendly operators; (2) a self-adaptive rescaling algorithm to reduce the overhead of dynamic rescaling in backward propagation; (3) a batch-splitting algorithm to improve DSP cache efficiency; (4) a DSP compute subgraph-reusing mechanism to eliminate the preparation overhead on DSP. We have fully implemented Mandheling and demonstrated its effectiveness through extensive experiments. The results show that, compared to the state-of-the-art DNN engines from TFLite and MNN, Mandheling reduces per-batch training time by 5.5X and energy consumption by 8.9X on average. In end-to-end training tasks, Mandheling reduces convergence time by up to 10.7X and energy consumption by 13.1X, with only 1.9%--2.7% accuracy loss compared to the FP32 precision setting.

On-device learning is a promising technique for emerging privacy-preserving machine learning paradigms. However, through quantitative experiments, we find that commodity mobile devices cannot well support state-of-the-art DNN training with a large enough batch size, due to the limited local memory capacity. To fill the gap, we propose Melon, a memory-friendly on-device learning framework that enables the training tasks with large batch size beyond the physical memory capacity. Melon judiciously retrofits existing memory saving techniques to fit into resource-constrained mobile devices, i.e., recomputation and micro-batch. Melon further incorporates novel techniques to deal with the high memory fragmentation and memory adaptation. We implement and evaluate Melon with various typical DNN models on commodity mobile devices. The results show that Melon can achieve up to 4.33× larger batch size under the same memory budget. Given the same batch size, Melon achieves 1.89× on average (up to 4.01×) higher training throughput, and saves up to 49.43% energy compared to competitive alternatives. Furthermore, Melon reduces 78.59% computation on average in terms of memory budget adaptation.

Owing to the benefit of light weight, containers have become a promising enabler for cloud native computing. Container images composed of applications and dependencies support flexible service deployment and migration. Rapid adoption and integration of containers generate millions of images to be stored. Additionally, non-local images have to be frequently downloaded from the registry, resulting in huge amounts of traffic. Content Addressable Storage (CAS) has been adopted for saving storage and networking by enabling identical layers sharing across images. However, according to our measurements, the implication of CAS is significantly limited as layers are rarely fully identical in practice. In this paper, we propose to reconstruct the docker images to raise the number of identical layers and thereby reduce storage and network consumption. We explore the layered structure of images and define the commutativity of files to assure image validity. The image reconstruction is formulated as an integer nonlinear programming problem. Inspired by the observed similarity of layers, we design a similarity-aware online image reconstruction algorithm. Extensive evaluations are conducted to verify the performance of the proposed approach.

Deploying deep learning (DL) on mobile devices has been a notable trend in recent years. To support fast inference of on-device DL, DL libraries play a critical role as algorithms and hardware do. Unfortunately, no prior work ever dives deep into the ecosystem of modern DL libs and provides quantitative results on their performance. In this paper, we first build a comprehensive benchmark that includes 6 representative DL libs and 15 diversified DL models. We then perform extensive experiments on 10 mobile devices, which help reveal a complete landscape of the current mobile DL libs ecosystem. For example, we find that the best-performing DL lib is severely fragmented across different models and hardware, and the gap between those DL libs can be rather huge. In fact, the impacts of DL libs can overwhelm the optimizations from algorithms or hardware, e.g., model quantization and GPU/DSP-based heterogeneous computing. Finally, atop the observations, we summarize practical implications to different roles in the DL lib ecosystem.

Public edge platforms have drawn increasing attention from both academia and industry. In this study, we perform a first-of-its-kind measurement study on a leading public edge platform that has been densely deployed in China. Based on this measurement, we quantitatively answer two critical yet unexplored questions. First, from end users' perspective, what is the performance of commodity edge platforms compared to cloud, in terms of the end-to-end network delay, throughput, and the application QoE. Second, from the edge service provider's perspective, how are the edge workloads different from cloud, in terms of their VM subscription, monetary cost, and resource usage. Our study quantitatively reveals the status quo of today's public edge platforms, and provides crucial insights towards developing and operating future edge services.

Human brains are known to be capable of speeding up visual recognition of repeatedly presented objects through faster memory encoding and accessing procedures on activated neurons. For the first time, we borrow and distill such a capability into a semantic memory design, namely SMTM, to improve on-device CNN inference. SMTM employs a hierarchical memory architecture to leverage the long-tail distribution of objects of interest, and further incorporates several novel techniques to put it into effects: (1) it encodes high-dimensional feature maps into low-dimensional, semantic vectors for low-cost yet accurate cache and lookup; (2) it uses a novel metric in determining the exit timing considering different layers' inherent characteristics; (3) it adaptively adjusts the cache size and semantic vectors to fit the scene dynamics. SMTM is prototyped on commodity CNN engine and runs on both mobile CPU and GPU. Extensive experiments on large-scale datasets and models show that SMTM can significantly speed up the model inference over standard approach (up to 2×) and prior cache designs (up to 1.5x), with acceptable accuracy loss.

Big data has become valuable property for enterprises and enabled various intelligent applications. Today, it is common to host data in big data platforms (e.g., Spark), where developers can submit scripts to process the original and intermediate data tables. Meanwhile, it is highly desirable to manage the data to comply with various privacy requirements. To enable flexible and automated privacy policy enforcement, we propose TaintStream, a fine-grained taint tracking framework for Spark-like big data platforms. TaintStream works by automatically injecting taint tracking logic into the data processing scripts, and the injected scripts are dynamically translated to maintain a taint tag for each cell during execution. The dynamic translation rules are carefully designed to guarantee non-interference in the original data operation. By defining different semantics of taint tags, TaintStream can enable various data management applications such as access control, data retention, and user data erasure. Our experiments on a self-crafted benchmarksuite show that TaintStream is able to achieve accurate cell-level taint tracking with a precision of 93.0% and less than 15% overhead. We also demonstrate the usefulness of TaintStream through several real-world use cases of privacy policy enforcement.

Federated learning (FL) is an emerging, privacy-preserving machine learning paradigm, drawing tremendous attention in both academia and industry. A unique characteristic of FL is heterogeneity, which resides in the various hardware specifications and dynamic states across the participating devices. Theoretically, heterogeneity can exert a huge influence on the FL training process, e.g., causing a device unavailable for training or unable to upload its model updates. Unfortunately, these impacts have never been systematically studied and quantified in existing FL literature.

The emerging 6G network will make it possible for cybertwin, which relies deeply on the low latency and powerful computation provided by the edge network. To this end, the convergence of computing and network has been attached great importance. Most existing work study either placing edge servers or deploying user plane functions (UPFs), seldom considers the two processes jointly. In this article, we study how to minimize the latency with cost limitation by means of jointly deploying edge servers and UPFs in 6G scenario. We have shown that the problem is NP-hard. Then, we simplify the problem by analyzing the placement relationship between edge servers and UPFs and prune the solution space of the problem. To solve the problem effectively, a UPF and edge server placement algorithm is proposed. Massive experiments are conducted based on real-world data set and an edge core network emulator. The evaluation results show that our algorithm outperforms the benchmark algorithms.

Low-cost wireless cameras are growing rapidly. With the help of advanced machine learning models (e.g., CNNs), those videos exhibit high business and social values, e.g., for retailing planning [18], wildlife study [21], and traffic monitoring [19, 25]. However, with high compute need, traditional video analytics systems [14, 15, 26, 27] require all videos to be uploaded to a backend server, which stresses the scarce network bandwidth between cameras and servers.

Elf is a runtime for an energy-constrained camera to continuously summarize video scenes as approximate object counts. Elf's novelty centers on planning the camera's count actions under energy constraint. (1) Elf explores the rich action space spanned by the number of sample image frames and the choice of per-frame object counters; it unifies errors from both sources into one single bounded error. (2) To decide count actions at run time, Elf employs a learning-based planner, jointly optimizing for past and future videos without delaying result materialization. Tested with more than 1,000 hours of videos and under realistic energy constraints, Elf continuously generates object counts within only 11% of the true counts on average. Alongside the counts, Elf presents narrow errors shown to be bounded and up to 3.4X smaller than competitive baselines. At a higher level, Elf makes a case for advancing the geographic frontier of video analytics.

Due to their on-body and ubiquitous nature, wearables can generate a wide range of unique sensor data creating countless opportunities for deep learning tasks. We propose DeepWear, a deep learning (DL) framework for wearable devices to improve the performance and reduce the energy footprint. DeepWear strategically offloads DL tasks from a wearable device to its paired handheld device through local network connectivity such as Bluetooth. Compared to the remote-cloud-based offloading, DeepWear requires no Internet connectivity, consumes less energy, and is robust to privacy breach. DeepWear provides various novel techniques such as context-aware offloading, strategic model partition, and pipelining support to efficiently utilize the processing capacity from nearby paired handhelds. Deployed as a user-space library, DeepWear offers developer-friendly APIs that are as simple as those in traditional DL libraries such as TensorFlow. We have implemented DeepWear on the Android OS and evaluated it on COTS smartphones and smartwatches with real DL models. DeepWear brings up to 5.08× and 23.0× execution speedup, as well as 53.5 and 85.5 percent energy saving compared to wearable-only and handheld-only strategies, respectively.

To bridge the knowledge gap between research and practice, we present the first empirical study on 16,500 the most popular Android apps, demystifying how smartphone apps exploit deep learning in the wild. To this end, we build a new static tool that dissects apps and analyzes their deep learning functions. Our study answers threefold questions: what are the early adopter apps of deep learning, what do they use deep learning for, and how do their deep learning models look like. Our study has strong implications for app developers, smartphone vendors, and deep learning R&D. On one hand, our findings paint a promising picture of deep learning for smartphones, showing the prosperity of mobile deep learning frameworks as well as the prosperity of apps building their cores atop deep learning. On the other hand, our findings urge optimizations on deep learning models deployed on smartphones, protection of these models, and validation of research ideas on these models.

In enterprise organizations, the Bring-Your-Own-Device (BYOD) requirement has become prevalent as employees use their own mobile devices to process the workflow-oriented tasks. Consequently, it calls for approaches that can quickly develop and integrate mobile user interactions into existing business processes, and adapt to various contexts. However, designing, developing, and deploying adaptive and mobile-oriented user interfaces for existing process engines are non-trivial, and require significant systematic efforts. To address this issue, we present a novel middleware-based approach, called MUIT, to developing and deploying the Mobility, User Interactions and Tasks into WS-BPEL engines. MUIT provides a Domain-Specific Language (DSL) that provides some intuitive facilities to support the declarative development of adaptive, mobile-oriented, and Web-based user interfaces in WS-BPEL. The DSL can significantly reduce developers' manual efforts of developing user interactions by preventing arbitrarily mixed code, and its runtime supports satisfactory user experiences. Additionally, MUIT can be seamlessly integrated into WS-BPEL without intrusions of existing process instances. We implement a proof-of-concept prototype by integrating MUIT into the commodity WS-BPEL-based Apusic Platform, and evaluate the performance and usability of MUIT platform.

Many apps benefit from knowing their power consumption and adapting their behaviors on the fly. To offer apps power knowledge at run time, an OS often meters system power and divides it among apps. Since the impacts of concurrent apps on system power are entangled, this approach not only makes it difficult to reason about power but also results in power side channels, a serious vulnerability.

We present DeepCache, a principled cache design for deep learning inference in continuous mobile vision. DeepCache benefits model execution efficiency by exploiting temporal locality in input video streams. It addresses a key challenge raised by mobile vision: the cache must operate under video scene variation, while trading off among cacheability, overhead, and loss in model accuracy. At the input of a model, DeepCache discovers video temporal locality by exploiting the video's internal structure, for which it borrows proven heuristics from video compression; into the model, DeepCache propagates regions of reusable results by exploiting the model's internal structure. Notably, DeepCache eschews applying video heuristics to model internals which are not pixels but high-dimensional, difficult-to-interpret data. Our implementation of DeepCache works with unmodified deep learning models, requires zero developer's manual effort, and is therefore immediately deployable on off-the-shelf mobile devices. Our experiments show that DeepCache saves inference execution time by 18% on average and up to 47%. DeepCache reduces system energy consumption by 20% on average.

Background activities on smartphones are essential to today's "always-on" mobile device experience. Yet, there lacks a clear understanding of the cooperative behaviors among background activities as well as a quantification of the consequences. In this paper, we present the first in-depth study of app collusion, in which one app surreptitiously launches others in the background without user's awareness. To enable the study, we develop AppHolmes, a static analysis tool for detecting app collusion by examining the app binaries. By analyzing 10,000 apps from top third-party app markets, we found that i) covert, cooperative behaviors in background app launch are surprisingly pervasive, ii) most collusion is caused by shared services, libraries, or common interest among apps, and iii) collusion has serious impact on performance, efficiency, and security. Overall, our work presents a strong implication on future mobile system design.

Numerous complains have been made by Android users who severely suffer from the sluggish response when interacting with their devices. However, very few studies have been conducted to understand the user-perceived latency or mitigate the UI-lagging problem. In this paper, we conduct the first systematic measurement study to quantify the user-perceived latency using typical interaction-intensive Android apps in running with and without background workloads. We reveal the insufficiency of Android system in ensuring the performance of foreground apps and therefore design a new system to address the insufficiency accordingly. We develop a lightweight tracker to accurately identify all delay-critical threads that contribute to the slow response of user interactions. We then build a resource manager that can efficiently schedule various system resources including CPU, I/O, and GPU, for optimizing the performance of these threads. We implement the proposed system on commercial smartphones and conduct comprehensive experiments to evaluate our implementation. Evaluation results show that our system is able to significantly reduce the user-perceived latency of foreground apps in running with aggressive background workloads, up to 10x, while incurring negligible system overhead of less than 3.1 percent CPU and 7 MB memory.