Deep learning (DL) is quickly becoming a core technology to process sensor data from IoT devices. Nowadays, data is usually sent to remote Cloud services where GPU-based ML platforms process it. In contrast, our vision is to have IoT devices with local, on-device DL. This can increase privacy and accessibility as well as reduce processing costs. To do so, specialized hardware accelerators are needed to perform ML operations efficiently without high performance CPUs. In this paper we introduce our approach for a generator for ML hardware accelerators in the IoT. We focus specifically on the inference of ML algorithms for a typical class of IoT use cases, i.e., processing time-series data under real-time constraints. Additionally our optimization techniques rely on the co-design of hardware and ML algorithms to explore the most specialized and efficient corners of the design space. In this work we analyze the requirements for our generator, discuss optimization stages and techniques and show a case study based on an LSTM model for traffic flow prediction.

Abstract Artificial intelligence (AI) is an important part of today’s pervasive computing systems. Still, there is no end-to-end system platform that allows to deploy, update, manage and execute AI models in pervasive systems. We propose such a system platform in this paper. Most importantly, we reuse concepts and techniques from twenty years of pervasive computing research on how to enable runtime adaptation and apply it to AI. This allows to specify adaptive AI models that are able to react to a multitude of dynamic changes, e.g. with respect to available devices, networking conditions, but also application requirements and sensor data sources. Developers can optimise their applications iteratively, starting with a generic setup and refining it step by step towards their specific pervasive computing scenario. To show the applicability of our platform, we apply it to two pervasive use cases and evaluate them, achieving up to four times faster inference and three times lower energy consumption compared to a classical AI deployment.