Mixture-of-Experts (MoE) is widely adopted to deploy Large Language Models (LLMs) on edge devices with limited memory budgets. Although MoE is, in theory, an inborn memory-friendly architecture requiring only a few activated experts to reside in the memory for inference, current MoE architectures cannot effectively fulfill this advantage and will yield intolerable inference latencies of LLMs on memory-constrained devices. Our investigation pinpoints the essential cause as the remarkable temporal inconsistencies of inter-token expert activations, which generate overly frequent expert swapping demands dominating the latencies. To this end, we propose a novel MoE architecture, Oracle-MoE, to fulfill the real on-device potential of MoE-based LLMs. Oracle-MoE route tokens in a highly compact space suggested by attention scores, termed the oracle space, to effectively maintain the semantic locality across consecutive tokens to reduce expert activation variations, eliminating massive swapping demands. Theoretical analysis proves that Oracle-MoE is bound to provide routing decisions with better semantic locality and, therefore, better expert activation consistencies. Experiments on the pretrained GPT-2 architectures of different sizes (200M, 350M, 790M, and 2B) and downstream tasks demonstrate that without compromising task performance, our Oracle-MoE has achieved state-of-the-art inference speeds across varying memory budgets, revealing its substantial potential for LLM deployments in industry.

Customization is a fundamental aspect of embedded system development, requiring developers to acquire extensive domain-specific knowledge from technical documents. However, the sheer volume of these documents, coupled with the intricate relationships between their contents, makes it challenging to efficiently retrieve the necessary information. This challenge highlights the need for a structured approach to organize domain knowledge, with explicit representation of interrelationships. Moreover, while advanced large language models (LLMs) show promise in aiding embedded system development, they often lack the specialized knowledge needed to address domain-specific queries effectively. In this article, we present HSG-RAG, a knowledge base construction and retrieval method tailored for embedded system development that leverages knowledge graphs to represent the hierarchical structure within technical documentation. Unlike prior retrieval-augmented generation (RAG) or GraphRAG-based approaches, which build the index either rely on semantic similarity or keyword co-occurrence, HSG-RAG captures the inherited dependency and hierarchical relationships in the documents, which benefits the retrieval in both performance and efficiency. We also introduce a benchmark for evaluating the effectiveness of RAG systems in solving real-world challenges in embedded systems, particularly for multi-hop question answering. Experimental results show that HSG-RAG outperforms both RAG and GraphRAG, generating more specific and concise responses.