Large language models are being deployed for various applications, and they are being scaled to larger parameter sizes. However, this makes it challenging to use these models for embedded applications due to the limited computing memory and processing capabilities available on embedded platforms. Consequently, it limits the usage of these models to invocation through function calls to a remotely hosted platform, which introduces latency and privacy challenges. A smaller parameter-sized language model running locally on embedded platforms could mitigate this challenge. However, such smaller models make trade-offs with dataset diversity and parameters, leading to inaccurate prompt responses and high hallucination rates. In response, we present our ongoing work: OTTER, a framework for pre-training custom language models and fine-tuning language models for embedded applications. We observe that smaller parameter-sized models outperform larger ones when curated with appropriate pre-training and fine-tuning datasets. Their smaller size allows these models to perform inference on constrained embedded platforms. Building on this framework, we train a custom model for breathing detection and observe high accuracy.

Large language models have attracted a significant recent interest, with an overwhelming efforts and emphasis on scaling their parameter size to support general-purpose and emergent capabilities. However, memory and processing requirements for model inference also scales proportionally with the model's parameter size. This makes it challenging for state-of-the-art, larger models to perform inference locally on edge and mobile devices, even though such models could benefit numerous tasks on these systems. Consequently, these devices are often restricted to accessing larger models through network calls. This approach introduces challenges related to increased inference latency due to network delays and provider capacity. It also raises concerns about sharing private information with third-party vendors. To address these issues, we are developing a system called OTTER. This system tackles the particular problem by enabling the training of smaller yet highly capable foundational language models. As a result of the reduced parameter size, these models can run locally even on constrained edge devices, such as mobile phones and wearables, and are able to provide low-latency responses compared with their larger remotely hosted counterparts. We present our ongoing work describing the framework as applied to training a smaller foundational model for embedded sensing application for tracking a person's breathing with high accuracy comparable to models orders of magnitude larger in size. Our results demonstrate that carefully pre-trained and fine-tuned smaller sized models outperform much larger counterparts for some tasks, while inferring locally on the constrained edge and mobile devices.