Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of LLMs’ high computational and memory demands. Specialized models optimized for specific tasks can be combined through a routing mechanism to address these challenges, creating a modular inference system. This paper introduces Expert Router, a scalable routing architecture that directs prompts to specialized expert models. We characterize multiple Expert Router configurations, including different LLama 3 models with quantized and non-quantized weights under up to 1,000 concurrent users. Our findings reveal that Expert Router introduces minimal latency overhead, with the configuration of expert models being a dominating factor in performance outcomes. High-parameter expert models deliver stable throughput and latency under moderate concurrency levels. In contrast, smaller expert models maintain competitive performance across a wider range of concurrent users compared to tensor-parallelized baseline models. This highlights the potential of Expert Router for efficient and scalable LLM deployment.

Large Language Model (LLM) routing architectures allow to manage and deploy multiple fine-tuned expert models through a single inference endpoint. However, a key challenge is that individually fine-tuned models may suffer from limited generation diversity and can produce hallucinations when responses extend beyond their specialization domains. To address this, we introduce the Dynamic Expert Router, an architecture that dynamically routes token generation across expert LLMs based on domain classification. The system monitors the generated output from expert LLMs and reroutes to a different model if the response begins to shift away from the relevant domain. We present preliminary experiments, in which we identify domain shifts during generation by analyzing the response sentences of a Llama 3 model. These findings guide our research questions on optimizing embedding techniques, clustering methods, and routing mechanisms.

The Internet-of-Things (IoT) is growing in importance enabling an increasing number of scientific, industrial, and societal applications. At the same time, the computational capabilities of IoT and edge devices are rapidly improving making them viable for machine learning. Thus, the deployment of machine learning models on the edge is becoming a critical capability. However, such deployments are challenging as edge devices are more resource-constrained than clouds and cannot elastically scale on demand. Moving the application to the cloud can provide more computational power, but raises other challenges, such as security, reliability, and bandwidth. The integration of edge and cloud computing resources is often essential for many applications allowing them to choose the best configuration with respect to their requirements and characteristics, such as data rates and computational complexity. To understand the trade-offs between edge and cloud computing the assessment of different cloud and edge configurations is required. In this paper, we present EdgeInsight, a framework for characterizing and modeling of the inference performance of edge and cloud infrastructures. EdgeInsight enables system builders to size and fine-tune infrastructure parameters for different workloads and applications. We use EdgeInsight to qualitatively and quantitatively study edge and cloud deployment configuration for deep learning inference. Our evaluation shows, that edge inference can outperform cloud inference when model architecture and accuracy, inference framework and pre-processing parameters are carefully selected.