The quadratic complexity of attention mechanisms poses a critical bottleneck for large language models processing long contexts. While dynamic sparse attention methods offer input-adaptive efficiency, they face fundamental trade-offs: requiring preprocessing, lacking global evaluation, violating query independence, or incurring high computational overhead. We present RRAttention, a novel dynamic sparse attention method that simultaneously achieves all desirable properties through a head round-robin (RR) sampling strategy. By rotating query sampling positions across attention heads within each stride, RRAttention maintains query independence while enabling efficient global pattern discovery with stride-level aggregation. Our method reduces complexity from O(L^2) to O(L^2/S^2) and employs adaptive Top-\tau selection for optimal sparsity. Extensive experiments on natural language understanding (HELMET) and multimodal video comprehension (Video-MME) demonstrate that RRAttention recovers over 99% of full attention performance while computing only half of the attention blocks, achieving 2.4\times speedup at 128K context length and outperforming existing dynamic sparse attention methods. The code is available at https://github.com/PaddlePaddle/PaddleFleet (see ‘Research/RRAttention‘).

Kernel-based remote memory systems are gaining traction in datacenters due to their significant improvement in memory utilization and their ability to transparently provide applications with unlimited memory capacity. However, the high degree of parallelism in contemporary applications leads to a significant demand for remote access throughput, which mismatches with the state-of-the-art kernel swap path due to its inherently limited parallelism. As a result, it cannot scale up the multi-core applications. We dive into the implementation of the swap path and identify the root cause behind it - significant lock contentions and inefficient swap tasks offloading.

While agile chip development methodologies have accelerated RTL design and simulation, performance evaluation remains constrained by three challenges: (1) inefficient feature prototyping caused by the tight coupling between functional correctness and performance evaluation, particularly for large-scale, error-prone microarchitectures; (2) limited workloads due to incomplete peripheral/software environments or unavailable source code; and (3) time-consuming warm-up phases in sampling-based simulation, required to mitigate cold-start effects. To address these challenges, we propose TRACERTL, an agile, trace-driven performance evaluation methodology that decouples the functional and performance components of CPU RTL designs. It introduces three techniques: (1) a trace-driven performance exploration framework that bypasses full functional correctness while preserving performance accuracy; (2) a trace transformation technique, TraceBridge, that replays traces across different formats and instruction sets; and (3) a fast warm-up strategy, TraceDedup, that eliminates redundant traces and efficiently initializes microarchitectural states. Using TRACERTL, we develop the first trace-driven RTL CPU derived from XiangShan, a high-performance out-of-order RISC-V processor. TRACERTL achieves performance accuracies of 99.87% and 99.86% on SPECint2017 and SPECfp2017, respectively. With TraceBridge, we evaluate x86-based Google workload traces on a RISC-V RTL CPU and reveal distinct memory-bound behavior. TraceDedup further accelerates warm-up phases in sampling-based simulations by$\text{1. 5} \times$to$\text{1 1. 8} \times$.

Verification is a critical process for ensuring the correctness of modern processors. The increasing complexity of processor designs and the emergence of new instruction set architectures (ISAs) like RISC-V have created demands for more agile and efficient verification methodologies, particularly regarding verification efficiency and faster coverage convergence. While simulation-based approaches now attempt to incorporate advanced software testing techniques such as fuzzing to improve coverage, they face significant limitations when applied to processor verification, notably poor performance and inadequate test case quality. Hardware-accelerated solutions using FPGA or ASIC platforms have tried to address these issues, yet they struggle with challenges including host-FPGA communication overhead, inefficient test pattern generation, and suboptimal implementation of the entire multi-step verification process. In this paper, we present TurboFuzz, an end-to-end hardwareaccelerated verification framework that implements the entire Test Generation-Simulation-Coverage Feedback loop on a single FPGA for modern processor verification. TurboFuzz enhances test quality through optimized test case (seed) control flow, efficient inter-seed scheduling, and hybrid fuzzer integration, thereby improving coverage and execution efficiency. Additionally, it employs a feedback-driven generation mechanism to accelerate coverage convergence. Experimental results show that TurboFuzz achieves up to 2.23× more coverage collection than software-based fuzzers within the same time budget, and up to 571× performance speedup when detecting real-world issues, while maintaining full visibility and debugging capabilities with moderate area overhead.

Transformer-based large language model (LLM) inference serving is now the backbone of many cloud services. LLM inference consists of a prefill phase and a decode phase. However, existing LLM deployment practices often overlook the distinct characteristics of these phases, leading to significant interference. To mitigate interference, our insight is to carefully schedule and group inference requests based on their characteristics. We realize this idea in ShuffleInfer through three pillars. First, it partitions prompts into fixed-size chunks so that the accelerator always runs close to its computation-saturated limit. Second, it disaggregates prefill and decode instances so each can run independently. Finally, it uses a smart two-level scheduling algorithm augmented with predicted resource usage to avoid decode scheduling hotspots. Results show that ShuffleInfer improves time-to-first-token (TTFT), job completion time (JCT), and inference efficiency in terms of performance per dollar by a large margin, e.g., it uses 38% less resources all the while lowering average TTFT and average JCT by 97% and 47%, respectively.

Improving resource utilization in datacenters is vital for reducing costs for cloud service providers (CSPs). Increasing resource utilization must be balanced with maintaining quality of service (QoS) for latency-critical applications. In cloud environments, users often request excessive resources for applications to ensure QoS. To address this issue, CSPs use resource overcommitment - offering users resources that exceed the actual capacity of physical infrastructure. However, if not properly managed, such strategies may result in performance degradation or even request failure. Therefore, to achieve optimal resource utilization while maintaining QoS to applications, it is critical to implement a fine-grained overcommitment strategy.We propose HAPPIES, a History-aware management system with a precise prediction for machine resource demand. HAPPIES uses historical usage to extract resource characteristics and build application portraits that describe their resource demands. Compared to the existing strategy, this is a more aggressive overcommitment strategy that achieves higher resource utilization. We simulated experiments on 3,021 nodes and deployed over 14,000 applications on them. Results show that HAPPIES significantly outperforms Kubernetes Least Request and Peak Oracle in load balancing. Not only does it reduce the number of nodes experiencing high utilization, but it also decreases the peak usage of the most heavily utilized nodes. Therefore, HAPPIES scheduling reduces the risk of a machine being used beyond capacity.

Identifying and managing performance interference in clouds has long been a critical and challenging task for cloud providers. They keep seeking useful performance indicators from underlying systems to monitor cloud applications accurately. However, state-of-the-art indicators are either sensitive to limited applications and resource contention or are unrobust to the continually changing production environments. There still lacks a practical and efficient indicator for production environments.

Coverage metrics have been widely adopted to quantify the completeness of hardware verification. Recently, coverage-guided fuzzing has emerged as a popular method for automatically creating test inputs toward higher verification coverage reach. However, we observe that its effectiveness on CPUs is hindered by limited sources of seed corpus and efficiency of mutations. To broaden the fuzzing horizons, this paper proposes the PathFuzz framework incorporating an efficient input format for fuzzing CPUs, the footprint memory, with seed corpus from real-world large-scale programs. Experiments demonstrate that using PathFuzz reaches over 95% verification coverage with four long-standing bugs newly identified in two well-known open-source CPU designs.

Many parallel mechanisms, including data parallelism, tensor parallelism, and pipeline parallelism, have been proposed and combined together to support training increasingly large deep neural networks (DNN) on massive GPU devices. Given a DNN model and GPU cluster, finding the optimal configuration by combining these parallelism mechanisms is an NP-hard problem. Widely adopted mathematical programming approaches have been proposed to search in a configuration subspace, but they are still too costly when scaling to large models over numerous devices.

Intel CAT is a widely available cache partitioning technique in commercial hardware but falls short in partitioning granularity. We propose LazyCAT, a fine-grained, on-demand, and easy-to-use cache partitioning technique, which not only can ensure the QoS of High-Priority (HP) applications but also yield the under-utilized cache sets to other Best-Effort (BE) applications for better resource efficiency. LazyCAT retains the easy-to-use philosophy of CAT and introduces a new soft LLC partitioning boundary, which is lower than the original CAT partitioning boundary (hard boundary). LazyCAT detects and selects under-utilized cache sets in HP applications during profiling, and specifies them to the soft boundary at runtime dynamically, yielding the cache blocks between these two boundaries for other applications. Meanwhile, LazyCAT provides users with a simple software interface to guide the set-level space allocation according to their needs. Experimental results show that LazyCAT exhibits substantial performance improvements (up to 12.2%) for BE applications with less than 3% performance degradation of HP applications.

Privacy preservation is critical for neural network inference, which often involves collaborative execution of different parties to make predictions on sensitive data based on sensitive neural network models. However, the expensive cryptographic operations of privacy preservation also pose performance chal-lenges to neural network inference. We address this performance-security tension by designing PP-Stream, a distributed stream processing system for high-performance privacy-preserving neural network inference. PP-Stream adopts hybrid privacy-preserving mechanisms for linear and non-linear operations of neural network inference. It treats inference data as real-time data streams, and parallelizes the inference operations across multiple pipelined stages that are executed by multiple servers and threads. It also solves the load-balanced resource allocation across servers and threads as an optimization problem. We prototype PP-Stream and show via testbed experiments that it achieves low inference latencies on various neural network models.

Vehicle-Infrastructure Cooperative Autonomous Driving (CAD) is a new paradigm of autonomous driving, which relies on the cooperation between intelligent roads and autonomous vehicles. This paradigm has been shown to be safer and more efficient compared to the on-vehicle-only autonomous driving paradigm. Our real-world deployment data indicate that the effectiveness of Vehicle-Infrastructure CAD is constrained by the reliability and performance of commercial communication networks. This paper targets this exact problem and proposes Tentacles, a middleware to achieve high communication reliability between intelligent roads and autonomous vehicles, in the context of Vehicle-Infrastructure CAD. Specifically, Tentacles dynamically matches Vehicle-Infrastructure CAD applications and the underlying communication technologies based on varying communication performance and quality needs. Evaluation results confirm that Tentacles reduces deadline violations by more than 88%, significantly improving the reliability of Vehicle-Infrastructure CAD systems.

Abstract The accurate representation of objects holds pivotal significance in the realm of panoptic segmentation. Presently, prevalent object representation methodologies, including box-based, keypoint-based and query-based techniques, encounter a challenge known as the ‘representation confusion’ issue in specific scenarios, often resulting in the mislabeling of instances. In response, this paper introduces Convex Object Representation (COR), a straightforward yet highly effective approach to address this problem. COR leverages a CNN-based Euclidean Distance Transform to convert the target instance into a convex heatmap. Simultaneously, it offers a parallel embedding method for encoding the object. Subsequently, COR characterizes objects based on the distinctive embedding vectors of their convex vertices. This paper seamlessly integrates COR into a state-of-the-art query-based panoptic segmentation framework. Experimental findings validate that COR successfully mitigates the representation confusion predicament, enhancing segmentation accuracy. The COR-augmented methods exhibit notable improvements of +1.3 and +0.7 points in PQ on the Cityscapes validation and MS COCO panoptic 2017 validation datasets, respectively.

Robot apps are becoming more automated, complex and diverse. An app usually consists of many functions, interacting with each other and the environment. This allows robots to conduct various tasks. However, it also opens a new door for cyber attacks: adversaries can leverage these interactions to threaten the safety of robot operations. Unfortunately, this issue is rarely explored in past works. We present thefirstsystematic investigation about the function interactions in common robot apps. First, we disclose the potential risks and damages caused by malicious interactions. We introduce a comprehensive graph to model the function interactions in robot apps by analyzing 3,100 packages from the Robot Operating System (ROS) platform. From this graph, we identify and categorize three types of interaction risks. Second, we propose novel methodologies to detect and mitigate these risks and protect the operations of robot apps. We introduce security policies for each type of risks, and design coordination nodes to enforce the policies and regulate the interactions. We conduct extensive experiments on 110 robot apps from the ROS platform and two complex apps (Baidu Apollo and Autoware) widely adopted in industry. Evaluation results showed our methodologies can correctly identify and mitigate all potential risks.

Data-intensive systems are the backbone of today's computing and are responsible for shaping data centers. Over the years, cloud providers have relied on three principles to maintain cost-effective data systems: use disaggregation to decouple scaling, use domain-specific computing to battle waning laws, and use serverless to lower costs. Although they work well individually, they fail to work in harmony: an issue amplified by emerging data system workloads.

While research has shown that the agile chip design methodology is promising to sustain the scaling of computing performance in a more efficient way, it is still of limited usage in actual applications due to two major obstacles: 1) Lack of tool-chain and developing framework supporting agile chip design, especially for large-scale modern processors. 2) The conventional verification methods are less agile and become a major bottleneck of the entire process. To tackle both issues, we propose MINJIE, an open-source platform supporting agile processor development flow. MINJIE integrates a broad set of tools for logic design, functional verification, performance modelling, pre-silicon validation and debugging for better development efficiency of state-of-the-art processor designs. We demonstrate the usage and effectiveness of MINJIE by building two generations of an open-source superscalar out-of-order RISC-V processor code-named XIANGSHAN using agile methodologies. We quantify the performance of XIANGSHAN using SPEC CPU2006 benchmarks and demonstrate that XIANGSHAN achieves industry-competitive performance.

With the deepening of research and increasing size of data sets, deep neural networks have become larger and larger. To reduce the training time of large neural networks, researchers propose to optimize neural networks from different levels. When performing optimizations, prior knowledge about execution time of each part of the network can help avoid repeatedly time-consuming testing and profiling process. However it is quite challenging to build an accurate iteration time prediction model, due to opaque underlying implementation of network operators and complex architecture of accelerators. In this paper, we propose SEER, an iteration time prediction model for CNNs, targeting on GPU platforms. We propose to categorize convolution kernels into three different types: Compute-bound, DRAM-bound and Under-utilized, then we build performance model for each type respectively. We combined analytical models and learning-based models to make the performance model accurate and in line with GPU execution model. Experimental results show that, our model achieves 14.71% prediction error on convolution kernels and up to 1.79% prediction error for the overall computation time in one iteration of common CNNs. Besides, when used for selecting the best convolution algorithm, our model shows 7.14% lower error rate than cuDNN's official algorithm picker.

This paper investigates how to better track and deliver dependency in dependency-based cores to exploit instruction-level parallelism (ILP) as much as possible. To this end, we first propose an analytical performance model for the state-of-art dependency-based core, Forwardflow, and figure out two vital factors affecting its upper bound of performance. Then we propose Omegaflow,a dependency-based architecture adopting three new techniques, which respond to the discovered factors. Experimental results show that Omegaflow improves IPC by 24.6% compared to the state-of-the-art design, approaching the performance of the OoO architecture with an ideal scheduler (94.4%) without increasing the clock cycle and consumes only 8.82% more energy than Forwardflow.

Low-cost Ground Vehicles (LGVs) have been widely adopted to conduct various tasks in our daily life. However, the limited on-board battery capacity and computation resources prevent LGVs from taking more complex and intelligent workloads. A promising approach is to offload the computation from local LGVs to remote servers. However, current cloud-robotic research and platforms are still at a very early stage. Compared to other systems and devices, optimizing LGV workload offloading faces more challenges, such as the uncertainty of environments and the mobility feature of devices.In this paper, we explore the opportunities of optimizing cloud offloading of LGV workloads from the perspectives of performance, energy efficiency and network robustness. We first build an analytical model to reveal the computation role and impact of each function in LGV workloads. Then we propose several optimization strategies (fine-grained migration, cloud acceleration, real-time monitoring and adjustment) to accelerate workload computation, reduce on-board energy consumption, and increase the network robustness. We implement an end-to-end cloud-robotic framework with such strategies to achieve dynamic and adaptive offloading. Evaluations on physical LGVs show that our strategies can significantly reduce the total energy consumption by 2.12× and mission completion time by 2.53×, and maintain strong robust ness under poor network quality.

Computing per-key aggregation is indispensable in streaming data analysis formulated as two phases, an update phase and a recovery phase. As the size and speed of data streams rise, accurate per-key information is useful in many applications like anomaly detection, attack prevention, and online diagnosis. Even though many algorithms have been proposed for per-key aggregation in stream processing, their accuracy guarantees only cover a small portion of keys. In this paper, we aim to achieve nearly full accuracy with limited resource usage. We follow the line of sketch-based techniques. We observe that existing methods suffer from high errors for most keys. The reason is that they track keys by complicated mechanism in the update phase and simply calculate per-key aggregation from some specific counter in the recovery phase. Therefore, we present PR-Sketch, a novel sketching design to address the two limitations. PR-Sketch builds linear equations between counter values and per-key aggregations to improve accuracy, and records keys in the recovery phase to reduce resource usage in the update phase. We also provide an extension called fast PR-Sketch to improve processing rate further. We derive space complexity, time complexity, and guaranteed error probability for both PR-Sketch and fast PR-Sketch. We conduct trace-driven experiments under 100K keys and 1M items to compare our algorithms with multiple state-of-the-art methods. Results demonstrate the resource efficiency and nearly full accuracy of our algorithms.

Simultaneous multithreading (SMT) technology improves CPU throughput, but also causes unpredictable performance fluctuations for co-located workloads. Although recent major SMT processors have adopted some techniques to promote hardware support for quality-of-service (QoS), achieving both precise performance control and high throughput on SMT architectures is still a challenging open problem.

Cloud platform provides great flexibility and cost-efficiency for end-users and cloud operators. However, low resource utilization in modern datacenters brings huge wastes of hardware resources and infrastructure investment. To improve resource utilization, a straightforward way is co-locating different workloads on the same hardware. To figure out the resource efficiency and understand the key characteristics of workloads in co-located cluster, we analyze an 8-day trace from Alibaba's production trace. We reveal three key findings as follows. First, memory becomes the new bottleneck and limits the resource efficiency in Alibaba's datacenter. Second, in order to protect latency-critical applications, batch-processing applications are treated as second-class citizens and restricted to utilize limited resources. Third, more than 90% of latency-critical applications are written in Java applications. Massive self-contained JVMs further complicate resource management and limit the resource efficiency in datacenters.

Big data applications that rely on relational databases gradually expose limitations on scalability and performance. In recent years, Hadoop ecosystem has been widely adopted as an evolving solution. This paper presents the migration of a legacy data analytics application in a provincial data center. The target platform follows "no one size fits all" method. Considering different workloads, data storage is hybrid with distributed file system (HDFS) and distributed NoSQL database. Beyond the architecture re-design, we focus on the problem of data model transformation from relational database to NoSQL database. We propose a query-aware approach to free developers from tedious manual work. The approach generates query-specific views (NoView) for NoSQL and re-structures the views to align with NoSQL's data model. Our results show that the migrated application achieves high scalability and high performance. We believe that our practice provides valuable insights (such as NoSQL data modeling methodology), and the techniques can be easily applied to other similar migrations.

Virtualization greatly improves resource utilization in IaaS platforms, but it also introduces potential interference between virtual machines (VMs). For example, VMs may suffer from performance degradation, when they are located in one host and compete for sharing physical resources. Thus, how to efficiently monitor and quantify the VMs interference becomes a key challenge for IaaS providers. In this paper, we present Vmon, a system to transparently monitor and quantify the interference between VMs with the hardware performance counters (HPCs). By collecting the HPCs of different VMs and exploring the LLC miss rates within HPCs, Vmon analyzes the relationship between the LLC miss rates and VM performance degradation to predict the interference between different resource-intensive VMs, and mitigate the VMs interference. The experimental results show that Vmon predicts the performance degradation in the accuracy of more than 90% with less than 10% performance overhead.

E-commerce systems are typically QoS-sensitive, so QoS-oriented tunings of e-commerce servers are very important for such systems. However, existing e-commerce benchmarks are insufficient for supporting QoS-oriented tunings, because some critical QoS features of e-commerce systems cannot be precisely evaluated by them. One example of these features is the integrality of service, which is usually expressed as a session, provided to customers. This paper presents a QoS-oriented e-commerce benchmark, which is named Bench4Q and is an extension of TPC-W supporting QoS-oriented tuning of e-commerce servers. The main features of Bench4Q include: (1) supporting session-based metrics analysis and (2) simulating QoS-sensitive load for QoS-oriented capacity analysis. We illustrate the promising benefits of these features for QoS-oriented tuning of an e-commerce server by a series of Bench4Q benchmarking on a typical e-commerce server.