Modern large language models rely on attention mechanisms that attend to all tokens in a sequence, resulting in quadratic computational complexity that limits scalability. While sparse attention reduces compute and memory requirements by attending to only important tokens, implementing these techniques presents significant challenges due to the complexity of combining static and dynamic sparse patterns and optimizing key-value (KV) cache management.

Large Language Models (LLMs) are wildly popular today and it is important to serve them efficiently. Existing LLM serving systems are stateless across requests. Consequently, when LLMs are used in the common setting of multi-turn conversations, a growing log of the conversation history must be processed alongside any request by the serving system at each turn, resulting in repeated processing.

While modern wireless devices are capable of using multiple WiFi interfaces, the Multipath TCP (MPTCP) protocol has been employed to make full use of the capacity of many radios by enabling multiple path communication simultaneously. To provide exceptional mobility support in wireless networks, a key question is to determine the best handoff strategy to switch between access points or among WiFi/3G interfaces during roaming. In this paper, we propose SERO, a novel model-driven SEamless ROaming framework to optimize layer-2 handoff and vertical handoff for multihomed devices using MPTCP. The proposed framework adopts a measurement-based method to derive the TCP throughput model for wireless communication during handoff. Based on the throughput model, we propose a hybrid handoff strategy that uses multiple WiFi interfaces for data transmission and employs 3G augmentation to bridge the network interruption caused by handoff and to guarantee the total throughput above a predefined threshold for roaming devices. We implement the SERO framework in a real-deployed wireless mesh network testbed, and evaluate its performance by extensive experiments, which shows that SERO achieves performance gain of 26%-180% compared with several existing handoff strategies.

Performing inference on pre-trained neural network models must meet the requirement of low-latency, which is often at odds with achieving high throughput. Existing deep learning systems use batching to improve throughput, which do not perform well when serving Recurrent Neural Networks with dynamic dataflow graphs. We propose the technique of cellular batching, which improves both the latency and throughput of RNN inference. Unlike existing systems that batch a fixed set of dataflow graphs, cellular batching makes batching decisions at the granularity of an RNN "cell" (a subgraph with shared weights) and dynamically assembles a batched cell for execution as requests join and leave the system. We implemented our approach in a system called BatchMaker. Experiments show that BatchMaker achieves much lower latency and also higher throughput than existing systems.

You put a program on a concurrent server, but you don't trust the server; later, you get a trace of the actual requests that the server received from its clients and the responses that it delivered. You separately get logs from the server; these are untrusted. How can you use the logs to efficiently verify that the responses were derived from running the program on the requests? This is the Efficient Server Audit Problem, which abstracts real-world scenarios, including running a web application on an untrusted provider. We give a solution based on several new techniques, including simultaneous replay and efficient verification of concurrent executions. We implement the solution for PHP web applications. For several applications, our verifier achieves 5.6-10.9x speedup versus simply re-executing, with <10% overhead for the server.

Wireless mesh network (WMN) has been proposed to extend wireless coverage by using multi-hop communication to enable network service for mobile devices via densely deployed mesh APs. While modern wireless devices are capable of using multiple WiFi interfaces, the Multipath TCP (MPTCP) protocol has been employed to make full use of the capacity of multiple radios. A key question of MPTCP communication in WMN is to decide the best access point (AP) association for each wireless interface to achieve maximum network performance. In this paper, we propose an optimal AP association scheme based on goodput measurement. We setup a wireless mesh testbed with densely deployed APs to study the impact of signal strength and multi-hop communication. We fit the measured data by a three-dimensional surface, which shows that network goodput can be estimated by a function of Received Signal Strength Indicator (RSSI) and hops. With the proposed estimation function, we formulate the AP association problem as a mathematical optimization problem and solve it by Integer Programming.We implement the proposed strategy in the real deployed testbed and evaluate its performance using MPTCP, which shows that it improves MPTCP goodput significantly compared to several existing strategies.