Large Language Models (LLMs) have experienced widespread adoption across scientific and industrial domains due to their versatility and utility for diverse tasks. Nevertheless, deploying and serving these models at scale with optimal throughput and latency remains a significant challenge, primarily because of LLMs’ high computational and memory demands. Specialized models optimized for specific tasks can be combined through a routing mechanism to address these challenges, creating a modular inference system. This paper introduces Expert Router, a scalable routing architecture that directs prompts to specialized expert models. We characterize multiple Expert Router configurations, including different LLama 3 models with quantized and non-quantized weights under up to 1,000 concurrent users. Our findings reveal that Expert Router introduces minimal latency overhead, with the configuration of expert models being a dominating factor in performance outcomes. High-parameter expert models deliver stable throughput and latency under moderate concurrency levels. In contrast, smaller expert models maintain competitive performance across a wider range of concurrent users compared to tensor-parallelized baseline models. This highlights the potential of Expert Router for efficient and scalable LLM deployment.

Large Language Model (LLM) routing architectures allow to manage and deploy multiple fine-tuned expert models through a single inference endpoint. However, a key challenge is that individually fine-tuned models may suffer from limited generation diversity and can produce hallucinations when responses extend beyond their specialization domains. To address this, we introduce the Dynamic Expert Router, an architecture that dynamically routes token generation across expert LLMs based on domain classification. The system monitors the generated output from expert LLMs and reroutes to a different model if the response begins to shift away from the relevant domain. We present preliminary experiments, in which we identify domain shifts during generation by analyzing the response sentences of a Llama 3 model. These findings guide our research questions on optimizing embedding techniques, clustering methods, and routing mechanisms.