In the evolving landscape of vehicular networks, it is crucial to ensure robust security and efficient data handling. In this work, we introduce a novel federated learning (FL) algorithm integrated within a distributed edge intelligence (DEI) framework, enhanced by a blockchain consensus mechanism, specifically designed for Internet of Vehicles (IoV) to enhance data privacy, efficiency, and system resilience. Motivated by the pressing need for improved data privacy and security in the IoV, our approach can not only prioritize these aspects but also enhance the efficiency and accuracy of distributed machine learning. The proposed consensus mechanism, by integrating Proof-of-Knowledge (PoK) with practical Byzantine fault tolerance (PBFT), is crafted to be lightweight, making it suitable for the dynamic and resource-constrained vehicular environments. Our evaluation findings demonstrate the algorithm’s superior performance and scalability, suggesting its applicability in diverse IoV scenarios and its potential to facilitate secure, robust, and efficient collaborative learning.

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in ShuffleInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that ShuffleInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in terms of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.