Serverless computing offers attractive scalability, elasticity and cost-effectiveness. However, constraints on memory, CPU and function runtime have hindered its adoption for dataintensive applications and machine learning (ML) workloads. Traditional ‘server-ful’ platforms enable distributed computation via fast networks and well-established inter-process communication (IPC) mechanisms such as MPI and shared memory. In the absence of such solutions in the serverless domain, parallel computation with significant IPC requirements is challenging. We present FSD-Inference, the first fully serverless and highly scalable system for distributed ML inference. We explore poten-tial communication channels, in conjunction with Function-as-a-Service (FaaS) compute, to design a state-of-the-art solution for distributed ML within the context of serverless data-intensive computing. We introduce novel fully serverless communication schemes for ML inference workloads, leveraging both cloud-based publish-subscribe/queueing and object storage offerings. We demonstrate how publish-subscribe/queueing services can be adapted for FaaS IPC with comparable performance to object storage, while offering significantly reduced cost at high parallelism levels. We conduct in-depth experiments on benchmark DNNs of various sizes. The results show that when compared to server-based alternatives, FSD-Inference is significantly more cost-effective and scalable, and can even achieve competitive performance against optimized HPC solutions. Experiments also confirm that our serverless solution can handle large distributed workloads and leverage high degrees of FaaS parallelism.

Abstract Building a real-time spatio-temporal forecasting system is a challenging problem with many practical applications such as traffic and road network management. Most forecasting research focuses on achieving (often marginal) improvements in evaluation metrics such as MAE/MAPE on static benchmark datasets, with less attention paid to building practical pipelines which achieve timely and accurate forecasts when the network is under heavy load. Transport authorities also need to leverage dynamic data sources such as roadworks and vehicle-level flow data, while also supporting ad-hoc inference workloads at low cost. Our cloud-based forecasting solution Foresight, developed in collaboration with Transport for the West Midlands (TfWM), is able to ingest, aggregate and process streamed traffic data, enhanced with dynamic vehicle-level flow and urban event information, to produce regularly scheduled forecasts with high accuracy. In this work, we extend Foresight with several novel enhancements, into a new system which we term Foresight Plus. New features include an efficient method for extending the forecasting scale, enabling predictions further into the future. We also augment the inference architecture with a new, fully serverless design which offers a more cost-effective solution and which seamlessly handles sporadic inference workloads over multiple forecasting scales. We observe that Graph Neural Network (GNN) forecasting models are robust to extensions of the forecasting scale, achieving consistent performance up to 48 hours ahead. This is in contrast to the 1 hour forecasting periods popularly considered in this context. Further, our serverless inference solution is shown to be more cost-effective than provisioned alternatives in corresponding use-cases. We identify the optimal memory configuration of serverless resources to achieve an attractive cost-to-performance ratio.