In the realm of Internet of Things (IoT) systems, accurately forecasting the runtime of Inferencing models on heterogeneous devices is paramount for optimizing resource allocation, particularly in the contexts of Model Distributed Inferencing (MDI) and Data Distributed Inferencing (DDI). This paper delves into the application of Gradient Boosting Regression (GBR) as a predictive modeling technique for estimating runtime in both MDI and DDI scenarios. GBR presents an equitable trade-off between interpretability, robustness against noise, and suitability for moderately sized datasets. The study reviews previous research on IoT inference optimization. It accentuates the multifaceted intricacies of device diversity and the significance of model interpretability within MDI and DDI setups. The primary contribution of this research is the novel application of the GBR model to predict machine learning inferencing runtime in MDI and DDI contexts. This approach is invaluable when empirical data is limited and characterizing the behavior of newly introduced devices is imperative. The paper elaborates on the GBR algorithm’s utilization, hyperparameters, and custom loss functions tailored explicitly for MDI and DDI. The results section exemplifies GBR’s performance across various computational regimes, including MDI and DDI. It offers insights into the model’s balance between accuracy and complexity. A performance comparison with prior models underscores GBR’s efficacy in predicting runtime in MDI and DDI. This work contributes to the ongoing discourse on IoT optimization and predictive modeling.

Over the past several years, there has been a large increase to the number of edge sensing devices gathering various types of data, from images to audio. One of the issues however is getting this data to a device that can quickly perform analysis. Limited network and compute are just a few of the issues that will be taken into consideration, when it comes to offloading data for inferencing. Existing solutions to the limits of edge computation includes, local computation w/ compression and pruned models, cloud offloading, and several distributed methods, including data distributed and model distributed inferencing. Previous work has explored the performance of these options using standard HTTP communication between the edge and external devices, now new network communication protocols are explored specifically gRPC. The impact of utilizing a more lightweight communication protocol to efficiently transfer data from the edge to other devices is explored while specifically focusing on potential drawbacks that may occur. gRPC while shown to perform more efficiently than standard HTTP, however does come at the cost of increased CPU utilization which may impact performance at the edge in select cases. Experiments will be conducted using a testbed which was created to allow definition of both edge, mid-range, and cloud computation devices to be defined each with corresponding compute and memory allocations. Additional, control of the network will be managed via the software defined network allowing for multiple bandwidth cases to be tested for performance comparison in various network constrained environments. Results highlight the performance benefits gRPC in each of the cases showing inferencing speed improvements of the previous HTTP implementation. However several of the cases show better improvements than expected.