Efficiently scheduling concurrent deep neural network (DNN) inference on the same GPU can significantly optimize resource utilization. Such scheduling requires accurate prediction of concurrent inference time. Existing approaches primarily rely on model-level features for prediction and scheduling, which necessitates retraining and resampling when encountering unseen models to ensure prediction accuracy. However, in MLOps pipelines, the rapid iteration of models introduces numerous unseen models, making accurate predictions highly challenging and increasing the risk of SLA violations. To address these challenges posed by unseen models, we present SKADI, a scheduling framework based on block-level feature extraction and two-stage greedy scheduling. First, SKADI introduces block-level feature extraction, decomposing DNN models into homogeneous blocks (contiguous operator sequences) to enable zero-shot inference time prediction for unseen models. Second, it proposes a round-based and two-stage greedy scheduling strategy that rapidly selects optimal model pairs and overlaps their critical operators. Experimental results show that for unseen models, SKADI reduces the MAPE of concurrent inference time prediction by 59.25% and 57.55% compared to DeInfer and Abacus. Additionally, SKADI reduces SLA violation rates by 70.6% while increasing throughput by 7.3%.

Improving QoS by simultaneously reducing the latency violation rate and jitter in the presence of multiple deep learning inference (DLI) tasks sharing a single edge computing processor remains a challenge. However, existing DLI systems at the edge, designed to maximize throughput, face performance challenges when confronted with requests with varying QoS.

Public clouds provide a bewildering choice of configurations for Deep Learning (DL) models, and the choice of configuration will significantly impact the performance and budget. However, it is an obvious challenge to recommend a near-optimal configuration for a particular DL model from a wide range of candidates. The huge search overhead of finding such a configuration is the notorious cold start problem in state-of-the-art efforts, and this problem becomes more severe when they are faced with unseen DL models.

The amount of time-series data that is generated has exploded due to the growing popularity of Internet of Things (IoT) devices and applications. These applications require efficient management of the time-series data on both the edge and cloud side that support high throughput ingestion, low latency query and advanced time series analysis. In this demonstration, we present Apache IoTDB managing time-series data to enable new classes of IoT applications. IoTDB has both edge and cloud versions, provides an optimized columnar file format for efficient time-series data storage, and time-series database with high ingestion rate, low latency queries and data analysis support. It is specially optimized for time-series oriented operations like aggregations query, down-sampling and sub-sequence similarity search. An edge-to-cloud time-series data management application is chosen to demonstrate how IoTDB handles time-series data in real-time and supports advanced analytics by integrating with Hadoop and Spark. An end-to-end IoT data management solution is shown by integrating IoTDB with PLC4x, Calcite, and Grafana.