On-device tuning of deep neural networks enables long-term adaptation at the edge, while keeping data fully private and secure. However, the high computational demand of back-propagation remains a challenge for ultra-low-power, memory-constrained extreme-edge devices. Attention-based models further exacerbate this challenge, given their complex architecture and scale. We present TrainDeeploy, a novel framework that unifies efficient inference with on-device training on heterogeneous ultra-low-power System-on-Chips (SoCs). TrainDeeploy is the first complete on-device training pipeline for extreme edge SoCs supporting both Convolutional Neural Networks (CNNs) and Transformer models, as well as multiple training techniques, such as selective layer-wise fine-tuning and Low-Rank Adaptation (LoRA). On a RISC-V-based heterogeneous SoC, we demonstrate the first end-to-end fine-tuning of a complete Transformer, Compact Convolutional Transformer (CCT), achieving 11 trained images per second. We show that LoRA on-device leads to a 23% reduction in dynamic memory usage, 15× reduction in trainable parameters and gradients, and 1.6× reduction in memory transfer compared to full backpropagation. TrainDeeploy achieves up to 4.6 FLOP/cycle on CCT (0.28M Param, 71–126M FLOPs) and leading-edge performance up to 13.4 FLOP/cycle on Deep-AE (0.27M Param, 0.8M FLOPs), while simultaneously widening the scope compared to state-of-the-art frameworks to support both CNNs and Transformers with parameter-efficient tuning.

With the rise of embodied foundation models (EFMs), most notably small language models (SLMs), adapting Transformers for the edge applications has become a very active field of research. However, achieving the end-to-end deployment of SLMs on the microcontroller (MCU)-class chips without high-bandwidth off-chip main memory access is still an open challenge. In this article, we demonstrate high efficiency end-to-end SLM deployment on a multicore RISC-V (RV32) MCU augmented with ML instruction extensions and a hardware neural processing unit (NPU). To automate the exploration of the constrained, multidimensional memory versus computation tradeoffs involved in the aggressive SLM deployment on the heterogeneous (multicore+NPU) resources, we introduce Deeploy, a novel deep neural network (DNN) compiler, which generates highly optimized C code requiring minimal runtime support. We demonstrate that Deeploy generates the end-to-end code for executing SLMs, fully exploiting the RV32 cores’ instruction extensions and the NPU. We achieve leading-edge energy and throughput of$490 \; \mu $J per token, at 340 token per second for an SLM trained on the TinyStories dataset, running for the first time on an MCU-class device without the external memory.