Adapting the on-device deep learning (DL) models to continual and unpredictable domain shifts is crucial for mobile applications like autonomous driving and augmented reality, which require seamless user experiences in ever-changing environments. Test-time adaptation (TTA) offers a promising solution that fine-tunes models withunlabeledreal-time data just before making predictions. However, TTA introduces a significant challenge, its forward-backward-reforward pipeline increases latency, compromising responsiveness in time-sensitive mobile scenarios. In this paper, we present AdaShadow, a responsive test-time adaptation framework designed for non-stationary mobile data distributions and resource dynamics. AdaShadow focuses on selectively updating only adaptation-critical layers, minimizing the latency impact. While this idea has been explored in general on-device training, the unsupervised and real-time nature of TTA presents unique challenges in assessing layer importance, predicting latency, and planning layer updates efficiently. AdaShadow addresses these challenges through abackpropagation-free importance assessorfor rapid identification of critical layers, a unit-basedruntime latency predictorthat accounts for resource constraints, and anonline layer update schedulerto ensure prompt retraining. Furthermore, a memory I/O-aware computation reuse scheme optimizes the reforward pass to further reduce delays. Our evaluations show that AdaShadow reduces adaptation latency by up to 3.7× (achieving ms-level response time) and improves accuracy by up to 25.4%, all while maintaining low memory and energy costs across CNN, and ViT models, outperforming state-of-the-art methods.

There are many deep learning (DL) powered mobile and wearable applications today continuously and unobtrusively sensing the ambient surroundings to enhance all aspects of human lives. To enable robust and private mobile sensing, DL models are often deployed locally on resource-constrained mobile devices using techniques such as model compression or offloading. However, existing methods, either front-end algorithm level (i.e. DL model compression/partitioning) or back-end scheduling level (i.e. operator/resource scheduling), cannot be locally online because they require offline retraining to ensure accuracy or rely on manually pre-defined strategies, struggle withdynamic adaptability. The primary challenge lies in feeding back runtime performance from theback-endlevel to thefront-endlevel optimization decision. Moreover, the adaptive mobile DL model porting middleware withcross-level co-adaptationis less explored, particularly in mobile environments withdiversityanddynamics. In response, we introduce CrowdHMTware, a dynamic context-adaptive DL model deployment middleware for heterogeneous mobile devices. It establishes anautomated adaptation loopbetween cross-level functional components, i.e. elastic inference, scalable offloading, and model-adaptive engine, enhancing scalability and adaptability. Experiments with four typical tasks across 15 platforms and a real-world case study demonstrate that${\sf CrowdHMTware}$can effectively scale DL model, offloading, and engine actions across diverse platforms and tasks. It hides run-time system issues from developers, reducing the required developer expertise.