While on-device small language models (SLMs) enable responsive inference, they often lack the accuracy required for complex reasoning tasks; conversely, server-based large language models (LLMs) deliver high accuracy but incur prohibitive latency for real-time mobile applications. Collaborative inference approaches, such as split computing and speculative decoding, promise to bridge this gap, yet both require repeated network synchronization during generation, accumulating delays that negate their computational benefits. We present SAIL, a collaborative inference framework that achieves both high accuracy and low latency through a strategy we term Prefix Handoff Inference (PHI). In PHI, a server LLM generates the difficult early tokens and hands off completion to the mobile SLM, consolidating communication into a single handoff while providing sufficient context for the SLM to maintain accuracy. SAIL operationalizes PHI through three core modules: the handoff decision module determining the optimal prefix length tailored to each query's difficulty, the branch prediction module speculatively generating candidate continuations on-device while awaiting the server prefix, and the adaptive control module coordinating hand-off and branch prediction under fluctuating network and server conditions to meet service level objectives. Together, these modules enable SAIL to deliver up to 76% accuracy improvement over mobile SLMs with 98.9% server LLM accuracy retention, while strictly meeting latency constraints across translation, math, and expert level QA.

While Vision Transformers (ViTs) have emerged as foundation models for visual recognition, their high computational demands hinder deployment on edge platforms. Temporal redundancy across video frames offers a natural opportunity to reuse prior computations; however, existing methods remain far from ideal, often relying on simple frame-difference signals. To address this, we propose Ouroboros, a framework that encompasses geometric redundancy from spatial displacement of content. We achieve this by aligning invariant content to consistent coordinates across frames, enabled by warping each frame into a global coordinate system via motion vectors from a hardware-accelerated encoder. Yet, this design raises two key challenges: (i) preserving content that drifts out of the limited coordinate system and (ii) maintaining spatial continuity at frame borders. Ouroboros resolves these challenges by introducing a toroidal (i.e., wrap-around) input space and reassigning positional encodings to track displaced content. Leveraging the significant patch reduction via a system-efficient partial computation scheme, our approach accelerates inference by up to 2.61× and reduces energy consumption by 64.5% on NVIDIA Jet-son Orin devices, with <1% accuracy loss on object detection and instance segmentation, outperforming prior methods. Designed to process only non-redundant patches, Ouroboros also excels as an offloading system, yielding higher accuracy at lower bandwidth compared to prior schemes. The source code is available at https://github.com/ckswjd99-lab/Ouroboros.

Modern Deep Neural Network (DNN) frameworks use tensor operators as the main building blocks of DNNs. However, we observe that operator-based construction of DNNs incurs significant drawbacks in parallelism in the form of synchronization barriers. Synchronization barriers of operators confine the scope of parallel computation to each operator and obscure the rich parallel computation opportunities that exist across operators. To this end, we present ASPEN, a novel parallel computation solution for DNNs that achieves fine-grained dynamic execution of DNNs, which (1) removes the operator barriers and expresses DNNs in dataflow graphs of fine-grained tiles to expose the parallel computation opportunities across operators, and (2) exploits these opportunities by dynamically locating and scheduling them in runtime. This novel approach of ASPEN enables opportunistic parallelism, a new class of parallelism for DNNs that is unavailable in the existing operator-based approaches. ASPEN also achieves high resource utilization and memory reuse by letting each resource asynchronously traverse depthwise in the DNN graph to its full computing potential. We provide challenges and solutions to our approach and show that our proof-of-concept implementation of ASPEN on CPU shows exceptional performance, outperforming state-of-the-art inference systems of TorchScript and TVM by up to 3.2$\times$ and 4.3$\times$, respectively.