We introduce on-modem learning, which enables a commodity communication modem installed on commercial home appliances to perform inference and training of compact-sized DNNs (deep neural networks) to provide custom intelligent services. Given that 80-90% of commercial smart home appliances are never connected to the network, leaving computing resources of the modem idle, the proposed on-modem learning transforms a learning-incapable modem into an independent self-learning module by utilizing the unused or under-utilized modem resources. To this end, we propose an on-modem DNN inference framework consisting of Data Payload Parser and On-Modem DNN engine. We also propose an on-modem DNN training framework, which enables resource-efficient training of DNNs on the modem via Greedy and Diverse Gradient Accumulation (GDGA), Dynamic Programming-based Back-Propagation (DPBP), and On-Modem Reinforcement Learning (OMRL). To the best of our knowledge, the proposed on-modem learning is the first to execute DNN inference and training on the modem of commercial home appliances. To validate the efficacy of on-modem learning, we implement two on-modem learning applications on the real LG Electronics washing machine, i.e., (1) remaining wash time prediction and (2) UE (unbalance error) forecasting. Our experiment shows that on-modem learning successfully conducts DNN inference and training on the modem of commercial home appliances and provides custom intelligent services to end users.

We present Mobile-Oriented Video Diffusion (MOVD) framework, the first diffusion-based text-to-video generation framework designed for efficient on-device execution on smartphone-grade hardware without requiring retraining, compression or pruning of the target denoising model. To address the challenges of diffusion-based text-to-video generation on computation- and memory-limited mobile devices, MOVD applies two novel techniques to pretrained video generative models. First, Linear Proportional Leap (LPL) reduces the excessive denoising steps required in video diffusion through an efficient leap-based approach. Second, Temporal Dimension Token Merging (TDTM) reduces the heavy computational load of attention layers by merging consecutive tokens along the temporal dimension. By integrating these techniques with Concurrent Inference with Dynamic Loading (CI-DL), which splits large models into smaller, executable segments for limited memory environments, MOVD allows a text-to-video diffusion generative model to run on an iPhone 15 Pro. We envision the proposed MOVD as a significant first step toward democratizing state-of-the-art generative technologies, enabling video generation on mobile and embedded devices without resource-intensive optimization procedures. Code implementation accessible at the link1.

Abstract In this paper, we introduce a memory-efficient CNN (convolutional neural network), which enables resource-constrained low-end embedded and IoT devices to perform on-device vision and audio tasks, such as image classification, object detection, and audio classification, using extremely low memory, i.e ., only 63 KB on ImageNet classification. Based on the bottleneck block of MobileNet, we propose three design principles that significantly curtail the peak memory usage of a CNN so that it can fit the limited KB memory of the low-end device. First, ‘input segmentation’ divides an input image into a set of patches, including the central patch overlapped with the others, reducing the size (and memory requirement) of a large input image. Second, ‘patch tunneling’ builds independent tunnel-like paths consisting of multiple bottleneck blocks per patch, penetrating through the entire model from an input patch to the last layer of the network, maintaining lightweight memory usage throughout the whole network. Lastly, ‘bottleneck reordering’ rearranges the execution order of convolution operations inside the bottleneck block such that the memory usage remains constant regardless of the size of the convolution output channels. We also present ‘peak memory aware quantization’, enabling desired peak memory reduction in actual deployment of quantized network. The experiment result shows that the proposed network classifies ImageNet with extremely low memory ( i.e ., 63 KB) while achieving competitive top-1 accuracy ( i.e ., 61.58%). To the best of our knowledge, the memory usage of the proposed network is far smaller than state-of-the-art memory-efficient networks, i.e ., up to 89x and 3.1x smaller than MobileNet ( i.e ., 5.6 MB) and MCUNet ( i.e ., 196 KB), respectively.

We propose SMMF (Square-Matricized Momentum Factorization), a memory-efficient optimizer that reduces the memory requirement of the widely used adaptive learning rate optimizers, such as Adam, by up to 96%. SMMF enables flexible and efficient factorization of an arbitrary rank (shape) of the first and second momentum tensors during optimization, based on the proposed square-matricization and one-time single matrix factorization. From this, it becomes effectively applicable to any rank (shape) of momentum tensors, i.e., bias, matrix, and any rank-d tensors, prevalent in various deep model architectures, such as CNNs (high rank) and Transformers (low rank), in contrast to existing memory-efficient optimizers that applies only to a particular (rank-2) momentum tensor, e.g., linear layers. We conduct a regret bound analysis of SMMF, which shows that it converges similarly to non-memory-efficient adaptive learning rate optimizers, such as AdamNC, providing a theoretical basis for its competitive optimization capability. In our experiment, SMMF takes up to 96% less memory compared to state-of-the-art memoryefficient optimizers, e.g., Adafactor, CAME, and SM3, while achieving comparable model performance on various CNN and Transformer tasks.

Long-term Time Series Forecasting (LTSF) tasks, which leverage the current data sequence as input to predict the future sequence, have become increasingly crucial in real-world applications such as weather forecasting and planning of electricity consumption. However, state-of-the-art LTSF models often fail to achieve prediction output alignment for the same timestamps across lagged input sequences. Instead, these models exhibit low output alignment, resulting in fluctuation in prediction outputs for the same timestamps, undermining the model's reliability. To address this, we propose AliO (Align Outputs), a novel approach designed to improve the output alignment of LTSF models by reducing the discrepancies between prediction outputs for the same timestamps in both the time and frequency domains. To measure output alignment, we introduce a new metric, TAM (Time Alignment Metric), which quantifies the alignment between prediction outputs, whereas existing metrics such as MSE only capture the distance between prediction outputs and ground truths. Experimental results show that AliO effectively improves the output alignment, i.e., up to 58.2\% in TAM, while maintaining or enhancing the forecasting performance (up to 27.5\%). This improved output alignment increases the reliability of the LTSF models, making them more applicable in real-world scenarios. The code implementation is on an anonymous GitHub repository.

We introduce On-Demand Federated Learning (On-Demand FL), which enables on-demand federated learning of a deep model for an arbitrary target data distribution of interest by making the best use of the heterogeneity (non-IID-ness) of local client data, unlike existing approaches trying to circumvent the non-IID nature of federated learning. On-Demand FL composes a dataset of the target distribution, which we call the composite dataset, from a selected subset of local clients whose aggregate distribution is expected to emulate the target distribution as a whole. As the composite dataset consists of a precise yet diverse subset of clients reflecting the target distribution, the on-demand model trained with exactly enough selected clients becomes able to improve the model performance on the target distribution compared when trained with off-target and/or unknown distributions while reducing the number of participating clients and federating rounds. We model the target data distribution in terms of class and estimate the class distribution of each local client from the weight gradient of its local model. Our experiment results show that On-Demand FL achieves up to 5% higher classification accuracy on various target distributions just involving 9${\times}$ fewer clients with FashionMNIST, CIFAR-10, and CIFAR-100.

In multivariate time series (MTS) classification, finding the important features (e.g., sensors) for model performance is crucial yet challenging due to the complex, high-dimensional nature of MTS data, intricate temporal dynamics, and the necessity for domain-specific interpretations. Current explanation methods for MTS mostly focus on time-centric explanations, apt for pinpointing important time periods but less effective in identifying key features. This limitation underscores the pressing need for a feature-centric approach, a vital yet often overlooked perspective that complements time-centric analysis. To bridge this gap, our study introduces a novel feature-centric explanation and evaluation framework for MTS, named CAFO (Channel Attention and Feature Orthgonalization). CAFO employs a convolution-based approach with channel attention mechanisms, incorporating a depth-wise separable channel attention module (DepCA) and a QR decomposition-based loss for promoting feature-wise orthogonality. We demonstrate that this orthogonalization enhances the separability of attention distributions, thereby refining and stabilizing the ranking of feature importance. This improvement in feature-wise ranking enhances our understanding of feature explainability in MTS. Furthermore, we develop metrics to evaluate global and class-specific feature importance. Our framework's efficacy is validated through extensive empirical analyses on two major public benchmarks and real-world datasets, both synthetic and self-collected, specifically designed to highlight class-wise discriminative features. The results confirm CAFO's robustness and informative capacity in assessing feature importance in MTS classification tasks. This study not only advances the understanding of feature-centric explanations in MTS but also sets a foundation for future explorations in feature-centric explanations. The codes are available at https://github.com/eai-lab/CAFO.

This paper introduces MicroDeblur, an on-device image motion deblur solution for resource-constrained microcontroller-based vision systems. Although motion blurs caused by the movement or shake of the device (camera) are pervasive in embedded, IoT, and mobile devices, it has been considered a hard nut to crack for many microcontrollers with extremely-limited resources (e.g., hundreds of KB of RAM). To tackle this problem, we combine the DNN (deep neural network) motion deblur method with the classical motion deblur approach and take the best of both worlds, i.e., 1) powerful pattern recognition ability of DNNs and 2) simplicity and stability of matrix-based classical algorithms. To deblur an image, MicroDeblur takes three steps: 1) blur kernel estimation, 2) blur image transformation, and 3) iterative clear image restoration. We propose 1) depth-independent convolution that efficiently estimates the blur kernel (pattern) and 2) Toeplitz-based motion blur modeling that enhances the time and space complexity of the deblurring process by and , respectively, compared to the existing methods. To the best of our knowledge, MicroDeblur is the first self-sufficient blind deconvolution solution for a stand-alone microcontroller that does not rely on extra hardware or external systems. We implement MicroDeblur on an ARM Cortex-M4F, achieving a competitive quality of deblurred images using 187x and 429x smaller memory and energy, respectively, compared to high-end GPU-based solutions.

In this paper, we propose to approximate the softmax output, which is the key product of the attention mechanism, to reduce its activation memory usage when training attention-based networks (aka Transformers). During the forward pass of the network, the proposed softmax output approximation method stores only a small fraction of the entire softmax output required for back-propagation and evicts the rest of the softmax output from memory. Then, during the backward pass, the evicted softmax activation output is approximated to compose the gradient to perform back-propagation for model training. Considering most attention-based models heavily rely on the softmax-based attention module that usually takes one of the biggest portions of the network, approximating the softmax activation output can be a simple yet effective way to decrease the training memory requirement of many attention-based networks. The experiment with various attention-based models and relevant tasks, i.e., machine translation, text classification, and sentiment analysis, shows that it curtails the activation memory usage of the softmax-based attention module by up to 84% (6.2× less memory) in model training while achieving comparable or better performance, e.g., up to 5.4% higher classification accuracy.

We introduce On-NAS, a memory-efficient on-device neural architecture search (NAS) solution, that enables memory-constrained embedded devices to find the best deep model architecture and train it on the device. Based on the cell-based differentiable NAS, it drastically curtails the massive memory requirement of architecture search, one of the major bottlenecks in realizing NAS on embedded devices. On-NAS first pre-trains a basic architecture block, called meta cell, by combining n cells into a single condensed cell via two-fold meta-learning, which can flexibly evolve to various architectures, saving the device storage space n times. Then, the offline-learned meta cell is loaded onto the device and unfolded to perform online on-device NAS via 1) expectation-based operation and edge pair search, enabling memory-efficient partial architecture search by reducing the required memory up to k and m/4 times, respectively, given k candidate operations and m nodes in a cell, and 2) step-by-step back-propagation that saves the memory usage of the backward pass of the n-cell architecture up to n times. To the best of our knowledge, On-NAS is the first standalone NAS and training solution fully operable on embedded devices with limited memory. Our experiment results show that On-NAS effectively identifies optimal architectures and trains it on the device, on par with GPU-based NAS in both few-shot and full-task learning settings, e.g., even 1.3% higher accuracy on miniImageNet, while reducing the run-time memory and storage usage up to 20x and 4x, respectively.

Healthcare professionals and individual users use wearable devices equipped with various sensors for healthcare management. Recently, the joint usage of artificial intelligence and these wearable sensors has played an essential role in healthcare management by providing a wide range of applications such as fitness tracking, gym activity monitoring, patient rehabilitation monitoring, and disease detection. These tasks eventually aim to enhance personal well-being and better manage the user’s physical health by monitoring different activity types and body weight changes. Here, we present an efficient multi-task learning framework based on commercial smart insoles that can solve three tasks related to physical health management: activity classification, speed estimation, and body weight estimation. Our multi-task framework converts the sensor data from the smart insole to a recurrence plot, which shows significant performance improvement compared to processing the raw time series data. In addition, we utilized a modified MobileNetV2 as our backbone network, which has a total parameter of less than 100K and a computational budget of 0.34G of multiply-accumulate operations. Furthermore, we collected a vast dataset from 72 users carrying out 16 experiments, which contains the largest number of people for multi-task learning purposes using smart insoles. Extensive experiments show that the proposed multi-task learning framework is extremely efficient while outperforming or leading to comparable performance against single-task models.

We propose a new concept called Weight Separation of deep neural networks (DNNs), which enables memory-efficient and accurate deep multitask learning on a memory-constrained embedded system. The goal of weight separation is to achieve extreme packing of multiple heterogeneous DNNs into the limited memory of the system while ensuring the prediction accuracy of the constituent DNNs at the same time. The proposed approach separates the DNN weights into two types of weight-pages consisting of a subset of weight parameters, i.e., shared and exclusive weight-pages. It optimally distributes the weight-pages into two levels of the system memory hierarchy and stores them separately, i.e., the shared weight-pages in primary (level-1) memory (e.g., RAM) and the exclusive weight-pages in secondary (level-2) memory (e.g., flask disk or SSD). First, to reduce the memory usage of multiple DNNs, less critical weight parameters are identified and overlapped onto the shared weight-pages that are deployed in the limited space of the primary (main) memory. Next, to retain the prediction accuracy of multiple DNNs, the essential weight parameters that play a critical role in preserving prediction accuracy are stored intact in the plentiful space of secondary memory storage in the form of exclusive weight-pages without overlapping. We implement two real systems applying the proposed weight separation: 1) a microcontroller-based multitask IoT system that performs multitask learning of 10 scaled-down DNNs by separating the weight parameters into FRAM and flash disk, and 2) an embedded GPU system that performs multitask learning of 10 state-of-the-art DNNs, separating the weight parameters into GPU RAM and eMMC. Our evaluation shows that memory efficiency, prediction accuracy, and execution time of deep multitask learning improve up to 5.9x, 2.0%, and 13.1x, respectively, without any modification of DNN models.

We introduce Deep Functional Network (DFN) that approximates a black-box Deep Neural Network (DNN) to a functional program consisting of a set of well-known functions and data flows among them. A DFN not only provides a semantic interpretation of a DNN but also enables easy deployment and optimization of the translated program according to the requirements and constraints of the target intelligent sensing system. To interpret a DNN, we propose the DFN framework consisting of two steps: 1) function estimation that estimates the distribution of functions likely to be used in the source DNN and 2) network formation that finds a functional network in the form of a directed acyclic graph (DAG) given the estimated function distribution. Our empirical study conducted with 16 state-of-the-art DNNs demonstrates that the generated DFNs provide semantic understandings of the DNNs along with comparable classification accuracy to the source DNNs. We implement two intelligent sensing systems that use the proposed DFN: 1) a mobile robot that avoids obstacles detected by a camera and 2) a smartphone-based human activity recognizer using IMU sensors, where different sizes of DFNs are generated to complete the task under various resource budgets, i.e., execution time and energy consumption dynamically imposed by run-time scenarios. The experiment result demonstrates that a set of DFNs generated from a single DNN enable both systems to achieve the desired performance under various resource constraints based on semantic understanding of the DNNs.

This paper introduces the concept of Neural Weight Virtualization - which enables fast and scalable in-memory multitask deep learning on memory-constrained embedded systems. The goal of neural weight virtualization is two-fold: (1) packing multiple DNNs into a fixed-sized main memory whose combined memory requirement is larger than the main memory, and (2) enabling fast in-memory execution of the DNNs. To this end, we propose a two-phase approach: (1) virtualization of weight parameters for fine-grained parameter sharing at the level of weights that scales up to multiple heterogeneous DNNs of arbitrary network architectures, and (2) in-memory data structure and run-time execution framework for in-memory execution and context-switching of DNN tasks. We implement two multitask learning systems: (1) an embedded GPU-based mobile robot, and (2) a microcontroller-based IoT device. We thoroughly evaluate the proposed algorithms as well as the two systems that involve ten state-of-the-art DNNs. Our evaluation shows that weight virtualization improves memory efficiency, execution time, and energy efficiency of the multitask learning systems by 4.1x, 36.9x, and 4.2x, respectively.

We introduce SubFlow-a dynamic adaptation and execution strategy for a deep neural network (DNN), which enables real-time DNN inference and training. The goal of SubFlow is to complete the execution of a DNN task within a timing constraint that may dynamically change while ensuring comparable performance to executing the full network by executing a subset of the DNN at run-time. To this end, we propose two online algorithms that enable SubFlow: 1) dynamic construction of a sub-network which constructs the best subnetwork of the DNN in terms of size and configuration, and 2) time-bound execution which executes the sub-network within a given time budget either for inference or training. We implement and open-source SubFlow by extending TensorFlow with full compatibility by adding SubFlow operations for convolutional and fully-connected layers of a DNN. We evaluate SubFlow with three popular DNN models (LeNet-5, AlexNet, and KWS), which shows that it provides flexible run-time execution and increases the utility of a DNN under dynamic timing constraints, e.g., lx-6.7x range of dynamic execution speed with average -3% of performance (inference accuracy) difference. We also implement an autonomous robot as an example system that uses SubFlow and demonstrate that its obstacle detection DNN is flexibly executed to meet a range of deadlines that varies depending on its running sped.

In this paper, we argue that the fusion of machine learning (ML) and batteryless computing systems enables true lifelong learning in mobile devices. The lack of learning from experience in current batteryless systems makes them ignorant of changes in their operating environment. Due to high communication cost, latency, privacy, and dependency issues of offloading computation to an edge device, on-device training is a solution for batteryless systems to learn and adapt in dynamically changing environments. Combining batteryless systems and ML is however a challenging task. Sporadic energy supply and limited resources in a batteryless system cause execution-discontinuity and data-constraints in ML processes. To understand these challenges, we identify suitable ML tasks for such systems and study the energy producers, i.e., harvesters, and consumers, i.e., intermittently executable tasks in a ML pipeline. Using a trace-driven simulation, we demonstrate the feasibility of on-device training of a batteryless learner.

We introduce Neuro.ZERO---a co-processor architecture consisting of a main microcontroller (MCU) that executes scaled-down versions of a deep neural network1 (DNN) inference task, and an accelerator microcontroller that is powered by harvested energy and follows the intermittent computing paradigm [76]. The goal of the accelerator is to enhance the inference performance of the DNN that is running on the main microcontroller. Neuro.ZERO opportunistically accelerates the run-time performance of a DNN via one of its four acceleration modes: extended inference, expedited inference, ensemble inference, and latent training. To enable these modes, we propose two sets of algorithms: (1) energy and intermittence-aware DNN inference and training algorithms, and (2) a fast and high-precision adaptive fixed-point arithmetic that beats existing floating-point and fixed-point arithmetic in terms of speed and precision, respectively, and achieves the best of both. To evaluate Neuro.ZERO, we implement low-power image and audio recognition applications and demonstrate that their inference speedup increases by 1.6× and 1.7×, respectively, and the inference accuracy increases by 10% and 16%, respectively, when compared to battery-powered single-MCU systems.

Although many Cyber-Physical Systems (CPS) have similarities among themselves, their control systems are often designed from the scratch. As a result, the knowledge of one expert control system does not come into use in designing and improving other types of control systems. In this paper, we explore the problem of knowledge transfer between two embedded control systems - which enables us to design effective and accurate control systems efficiently and at a large scale. To realize this idea, we formally define the problem of transferring knowledge between two linear time-variant systems. We derive necessary conditions for transferring parameters between two scalar systems as well as two high-order systems. We describe the transfer process which constitutes of a parameter update procedure, a convergence test, and a stability test. We derive a closed-form expression to quantify the performance benefit of the proposed technique in terms of the speed of convergence of system parameter adaptation process. In order to demonstrate the efficacy of the proposed technique, we conduct experiments with a real robotic arm as well as a mobile robot simulator. Our results show that the robotic arm learns its system dynamics 3--5 times faster when it uses transferred knowledge from a well-adapted robotic hand. Similarly, with transferred knowledge, the mobile robot navigates successfully to its target location while making 10 times less learning errors.