Simultaneous Localization and Mapping (SLAM) is a critical task that enables autonomous vehicles to construct maps and localize themselves in unknown environments. Recent breakthroughs combine SLAM with 3D Gaussian Splatting (3DGS) to achieve exceptional reconstruction fidelity. However, existing 3DGS-SLAM systems provide insufficient throughput due to the need for multiple training iterations per frame and the vast number of Gaussians.

Tensor parallelism (TP) in large-scale LLM inference and training introduces frequent collective operations that dominate inter-GPU communication. While in-switch computing, exemplified by NVLink SHARP (NVLS), accelerates collective operations by reducing redundant data transfer, its communication-centric design philosophy introduces the mismatch between its communication mode and the memory semantic requirement of LLM's computation kernel. Such a mismatch isolates the compute and communication phases, resulting in underutilized resources and limited overlap in multi-GPU systems. To address the limitation, we propose CAIS, the first ComputeAware In-Switch computing framework that aligns communication modes with computation's memory semantics requirement. CAIS consists of three integral techniques: (1) compute-aware ISA and microarchitecture extension to enable compute-aware in-switch computing. (2) merging-aware TB (Thread Block) coordination to improve the temporal alignment for efficient request merging. (3) graph-level dataflow optimizer to achieve a tight cross-kernel overlap. Evaluations on LLM workloads show that CAIS achieves$1.38 \times$average end-to-end training speedup over the SOTA NVLS-enabled solution, and$1.61 \times$over T3, the SOTA compute-communicate overlap solutions but do not leverage NVLS, demonstrating its effectiveness in accelerating TP on multi-GPU systems.

3D Gaussian Splatting (3DGS) has emerged as a powerful technique for novel view synthesis, combining high-quality reconstruction with efficient rendering. It has been widely adopted in domains such as AR/VR, robotics, and autonomous driving. However, achieving real-time performance on resource-constrained platforms remains challenging due to strict power and area budgets. Prior accelerators improve hardware performance but still overlook key inefficiencies, including insufficient rasterization efficiency, poor sorting scalability, and pipeline imbalance. This paper presents an architecture-algorithm co-design to address these challenges. First, we propose axis-shared rasterization, which precomputes and reuses common terms along the X- and Y-axes, reducing multiply-and-accumulate (MAC) operations by up to 38% while preserving high parallelism. Second, we develop a novel order-independent transmittance method that removes the need for explicit sorting by leveraging a lightweight multilayer perceptron (MLP) to directly approximate the transmittance of each Gaussian, enabling efficient alpha blending with negligible quality loss. Third, we design a unified reconfigurable PE array that supports both rasterization and MLP inference, sustaining high utilization without costly sorting hardware. Our experiments demonstrate that our design preserves rendering quality while achieving a 1.33 to 1.88x speedup over state-of-the-art 3DGS accelerators. Our code is open source at https://github.com/WangZhican/ISCA26_3DGS_Acc.

Neural Radiance Fields (NeRF) offer significant promise for generating photorealistic images and videos. However, existing mainstream neural rendering models often fall short in meeting the demands for immediacy and power efficiency in practical applications. Specifically, these models frequently exhibit irregular access patterns and substantial computational overhead, leading to undesirable inference latency and high power consumption. Computing-in-memory (CIM), an emerging computational paradigm, has the potential to address these access bottlenecks and reduce the power consumption associated with model execution.

Graph-traversal-based Approximate Nearest Neighbor (GANN) search and construction have become key retrieval techniques in various domains, such as recommendation systems and social networks. However, deploying GANN in real-world scenarios faces significant challenges, as high-dimensional vertices within the graph can lead to intensive memory demands. Although architectures like NDSearch have been proposed to accelerate GANN search, they are hard to deploy for GANN construction, as their pre-processing methods introduce massive overhead in dynamic graphs. In this paper, given the observation that neighboring vertices in a dynamic graph exhibit feature similarity, we propose SAGA, the first accelerator that alleviates memory bound in GANN construction. To capture this similarity, we directly leverage the first step of construction to gather vertices with the same starting point into a cluster to minimize the similarity detection overhead. Next, we decompose vertices into key and non-key ones, where their deltas fall in a narrow range, which is suitable to be quantized to lower bit widths. Building upon this approach, we design a specialized architecture, which efficiently implements the GANN construction by twolevel scheduling and a mixed-precision supported bit-serial unit. Through comprehensive evaluation, we demonstrate that SAGA can achieve an average speedup of $9.30 \times 4.87 \times 4.15 \times$ and $35.46 \times 7.60 \times 5.15 \times$ energy savings over CPU, GPU and NDSearch, respectively, while retaining task accuracy.

Diffusion models have demonstrated superior performance in image generation tasks, thus becoming the mainstream model for generative visual tasks. Diffusion models need to execute multiple timesteps sequentially, resulting in a dramatic increase in workload. Existing accelerators leverage the data similarity between adjacent timesteps and perform mixed-precision differential quantization to accelerate diffusion models. However, merging differential values with raw inputs in each layer of each timestep to ensure computational correctness requires significant memory access for loading raw inputs, which creates a heavy memory burden. Moreover, mixed-precision computations may lead to low hardware utilization if not well designed. Unlike these works, we propose MHDiff, a tailored framework that identifies the focal pixels at the first layer and finetunes them to fit all layers, then represents focal pixels with high-precision while using low-precision for others, thereby accelerating diffusion models while minimizing memory burden. To improve hardware utilization, MHDiff employs a packing module that merges low-precision values into high-precision values to create full high-precision matrices and designs a processing element (PE) array to efficiently process the packed matrices. Extensive experiment results demonstrate that MHDiff can achieve satisfactory performance with negligible quality loss.

Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks by effectively extracting global features. However, their self-attention mechanism suffers from quadratic time and memory complexity as image resolution or video duration increases, leading to inefficiency on GPUs. To accelerate ViTs, existing works mainly focus on pruning tokens based on value-level sparsity. However, they miss the chance to achieve peak performance as they overlook the bit-level sparsity. Instead, we propose Inter-token Bit-sparsity Awareness (IBA) algorithm to accelerate ViTs by exploring bit-sparsity from similar tokens. Next, we implement IBA on GPUs that synergize CUDA and Tensor Cores by addressing two issues: firstly, the bandwidth congestion of the Register File hinders the parallel ability of CUDA and Tensor Cores. Secondly, due to the varying exponent of floating-point vectors, it is hard to accelerate bitsparse matrix multiplication and accumulation (MMA) in Tensor Core through fixed-point-based bit-level circuits. Therefore, we present SynGPU, an algorithm-hardware co-design framework, to accelerate ViTs. SynGPU enhances data reuse by a novel data mapping to enable full parallelism of CUDA and Tensor Cores. Moreover, it introduces Bit-Serial Tensor Core (BSTC) that supports fixed- and floating-point MMA by combining the fixedpoint Bit-Serial Dot Product (BSDP) and exponent alignment techniques. Extensive experiments show that SynGPU achieves an average of $2.15 \times \sim 3.95 \times$ speedup and $2.49 \times \sim 3.81 \times$ compute density over A100 GPU.

Spiking neural networks (SNNs) are an alternative computational paradigm to artificial neural networks (ANNs) that have attracted attention due to their event-driven execution mechanisms, enabling extremely low energy consumption. However, the existing SNN execution model, based on software simulation or synchronized hardware circuitry, is incompatible with the event-driven nature, thus resulting in poor performance and energy efficiency. The challenge arises from the fact that neuron computations across multiple time steps result in increased latency and energy consumption. To overcome this bottleneck and leverage the full potential of SNNs, we propose TAIL, a pioneering temporal asynchronous execution mechanism for SNNs driven by a comprehensive analysis of SNN computations. Additionally, we propose an efficient dataflow design to support SNN inference, enabling concurrent computation of various time steps across multiple layers for optimal Processing Element (PE) utilization. Our evaluations show that TAIL greatly improves the performance of SNN inference, achieving a 6.94× speedup and a 6.97× increase in energy efficiency on current SNN computing platforms.

3D Gaussian Splatting (3DGS) introduces a novel methodology for representing scenes with anisotropic 3D Gaussian primitives, achieving exceptional quality and rendering speed in neural scene representation (NSR). However, the insufficient training speed of 3DGS limits its applicability in tasks that require online learning to perceive dynamic environments, such as autonomous driving and embodied intelligence. Although recent work, GSCore, has introduced a specialized accelerator for the rendering process of 3DGS, it overlooks the time-consuming backward propagation during 3DGS training.In this paper, we propose GSArch, a hardware architecture designed to overcome memory barriers and boost the efficiency of 3DGS training. Through a thorough characterization of 3DGS training, we identify three root causes of inefficiency: redundant data loading from off-chip memory, time-consuming atomic write operations, and severe bank conflicts during on-chip buffer reading. To address these challenges, GSArch introduces three architectural innovations. First, acknowledging that Gaussians vary in shape and often span multiple pixels, with larger Gaussians causing more repetitive data loading, GSArch employs hybrid memory management. This approach categorizes Gaussians into ‘hot’ and ‘cold’ ones, storing hot Gaussians in a fast but small on-chip buffer to reduce redundant loading while minimizing hardware costs. Second, GSArch leverages the informativeness variability of Gaussians’ gradients to filter out low-contribution gradients, significantly reducing atomic operations. Lastly, a rearrangement unit is designed to pack conflicting memory read requests into non-conflicting bundles. Our evaluation results demonstrate that GSArch achieves up to $6.49 \times$ and $15.42 \times$ speedups compared to Nvidia A100 and Jetson AGX Xavier, respectively, with substantially lower energy consumption and negligible image quality loss.

Graph Convolutional Networks (GCNs) are gaining attraction in AI research due to their ability to learn from graph data effectively. However, deploying GCNs on CPUs presents substantial challenges, as large-scale graph structures can lead to intensive computational and memory demands. While acceleration methods dedicatedly designed for GCNs on CPU platforms have emerged, they may not be the optimal solution as they tend to overlook the opportunity gained from the sparsity in the feature and adjacency matrices, leading to unsatisfactory memory and computation savings. In this paper, we propose a progressive two-step algorithm called GIFTS to accelerate GCN inference on CPUs by making use of the dynamic sparsity in the feature matrix and the static sparsity in the adjacency matrix. The first step introduces an online adaptive compression approach that performs a selective value-level compression on the part of feature vectors that will be frequently accessed during GCN inference. To further reduce the redundant bit width in feature vectors, we employ a bit-level pruning approach to narrow down bandwidth. The second step designs an offline degree-aware scheduling approach, which aims at balancing workloads caused by the irregular sparsity in the adjacency matrix during the GCN training process, given the observation that the adjacency matrix remains unchanged during inference. This step distributes workloads based on the nodes' degree in a coarse-grained fashion, considering both workload distribution and data locality. Through comprehensive evaluation, we demonstrate that GIFTS consistently outperforms the PyTorch Geometric (PyG) and DistGNN frameworks in terms of execution time, while incurring negligible accuracy loss. The source code is available at https://github.com/ACA-Lab/GIFTS.

Neural Scene Representation (NSR) is a promising technique for representing real scenes.By learning from dozens of 2D photos captured from different viewpoints, NSR computes the 3D representation of real scenes.However, the performance of NSR processing running on GPU is insufficient for applications.Cambricon-R achieves high performance of more than 60 scenes per second, but at the cost of modeling quality.

Transformer-empowered Graph Neural Networks (TF-GNNs) are gaining significant attention in AI research because they leverage the front-end Transformer's ability to process textual data while also harnessing the back-end GNN's capacity to analyze graph structures.Typically, TF-GNNs follow the sequential execution mode, where the front-end Transformer first encodes vertex features, followed by subgraph sampling and subsequent processing by the back-end GNN.However, due to the massive computation workloads of Transformers and the irregular memory access patterns of GNNs, achieving efficient inference for TF-GNNs remains a challenge.Although architectures like FACT and MEGA have been proposed to separately accelerate the Transformer and GNN, they overlook the new opportunities arising from the coupling of the Transformer and GNN.To enable efficient TF-GNNs, we propose HEAT, a heterogeneous architecture with a Neural Processing Unit (NPU) and a DIMM-based Near-Data Processing (NDP).Such a heterogeneous architecture can utilize both the high computational power of NPU and the high internal bandwidth of NDP.To fully unleash the potential of the NPU-NDP architecture, HEAT makes the following three contributions: First, HEAT leverages graph topology to identify the importance of vertices and encodes their features in the Transformer using varying precision accordingly.Second, HEAT gives more flexibility to the execution granularity and execution order of * Zhuoran Song is the corresponding author.

3D Gaussian Splatting (3DGS) has emerged as a leading neural rendering technique for high-fidelity view synthesis, prompting the development of dedicated 3DGS accelerators for resource-constrained platforms.The conventional decoupled preprocessing-rendering dataflow in existing accelerators has two major limitations: 1) a

The main transformer of a substation has highly dynamic and nonlinear characteristics, and its load is affected by various nonlinear factors and fluctuation characteristics, making load forecasting complex and difficult to obtain ideal results. Therefore, an optimization study on load forecasting method for main transformers in substations based on EEMD-BP neural network is proposed. Firstly, decompose the transient and steady-state composite data acquisition logic, construct the historical load data sequence of the main transformer, and then select the horizontal processing method and mean filling method to preprocess the historical load data; Finally, the integrated empirical mode decomposition method is used to decompose the non-stationary daily load sequence into multiple sets of component sequences with frequencies ranging from low to high, and the remaining component sequences are randomly combined to optimize the nonlinear characteristics of the input load data in the traditional BP neural network model, effectively achieving the optimization prediction of the load of the main transformer in the substation.The experimental results show that the design method can accurately predict the seasonal substation main transformer load based on the environmental impact, and the average prediction error is low, the RMSE value is 0.089, the MAPE value is 2.19 MW, and the AE value is 4.43 MW, which can effectively realize the accurate prediction of the substation main transformer load under the influence of environmental factors.

Recently, graph convolutional networks (GCNs) have gained wide attention due to their ability to capture node relationships in graphs. One problem appears when full-batch GCN is trained on large graph datasets, where the computational and memory requirements are unacceptable. To address this issue, mini-batch GCN training is introduced to improve the scalability of GCN training for large datasets by sampling and training only a subset of the graph in each batch. Although several acceleration techniques have been designed for boosting the efficiency of full-batch GCN, they lack attention to mini-batch GCN, which differs from full-batch GCN in terms of the sampled dynamic graph structures. Based on our previous work, GCNTrain [ 28 ], which was originally excogitated for accelerating full-batch GCN training, we devise GCNTrain+—a universal accelerator to tackle the performance bottlenecks associated with both full-batch and mini-batch GCN training. GCNTrain+ is equipped with two engines to optimize computation and memory access in GCN training, respectively. To reduce the computation overhead, we propose to dynamically reconfigure the computation order based on the varying data dimensions involved in each training batch. Moreover, we build a unified computation engine to perform the sparse-dense matrix multiplications and sparse-sparse matrix multiplications discovered in GCN training uniformly. To alleviate the memory burden, we devise a two-phased dynamic clustering mechanism to capture data locality as well as customized hardware to reduce the clustering overhead. We evaluate GCNTrain+ on seven datasets, and the result shows that GCNTrain+ achieves 136.0×, 52.6×, 2.2×, and 1.5× speedup over CPU, GPU, GCNAX, and GCNTrain in full-batch GCN training. Additionally, GCNTrain+ outperforms them with speedups of 131.6×, 67.1×, 4.4×, and 1.5× in mini-batch GCN training.

In the realm of video understanding tasks, Video Transformer models (VidT) have recently exhibited impressive accuracy improvements in numerous edge devices. However, their deployment poses significant computational challenges for hardware. To address this, pruning has emerged as a promising approach to reduce computation and memory requirements by eliminating unimportant elements from the attention matrix. Unfortunately, existing pruning algorithms face a limitation in that they only optimize one of the two key modules on VidT's critical path: linear projection or self-attention. Regrettably, due to the variation in battery power in edge devices, the video resolution they generate will also change, which causes both linear projection and self-attention stages to potentially become bottlenecks, the existing approaches lack generality. Accordingly, we establish a Run-Through Sparse Attention (RTSA) framework that simultaneously sparsifies and accelerates two stages. On the algorithm side, unlike current methodologies conducting sparse linear projection by exploring redundancy within each frame, we extract extra redundancy naturally existing between frames. Moreover, for sparse self-attention, as existing pruning algorithms often provide either too coarse-grained or fine-grained sparsity patterns, these algorithms face limitations in simultaneously achieving high sparsity, low accuracy loss, and high speedup, resulting in either compromised accuracy or reduced efficiency. Thus, we prune the attention matrix at a medium granularity—sub-vector. The sub-vectors are generated by isolating each column of the attention matrix. On the hardware side, we observe that the use of distinct computational units for sparse linear projection and self-attention results in pipeline imbalances because of the bottleneck transformation between the two stages. To effectively eliminate pipeline stall, we design a RTSA architecture that supports sequential execution of both sparse linear projection and self-attention. To achieve this, we devised an atomic vector-scalar product computation underpinning all calculations in parse linear projection and self-attention, as well as evolving a spatial array architecture with augmented processing elements (PEs) tailored for the vector-scalar product. Experiments on VidT models show that RTSA can save 2.71$\boldsymbol{\times}$to 5.32$\boldsymbol{\times}$ideal computation with$ \lt 1\%$accuracy loss, achieving 105$\boldsymbol{\times}$, 56.8$\boldsymbol{\times}$, 3.59$\boldsymbol{\times}$, and 3.31$\boldsymbol{\times}$speedup compared to CPU, GPU, as well as the state-of-the-art ViT accelerators ViTCoD and HeatViT.

Vision Transformers (ViTs) have achieved remarkable success across various tasks. However, their deployment is hindered by challenges, such as high memory requirements, long inference latency, and significant power consumption. To address these challenges, existing works optimize one of the two key stages on ViT’s critical path: linear projection or self-attention. Regrettably, we have noticed that both linear projection and self-attention can potentially become bottlenecks as the input image resolution varies, which makes the existing approaches lack generality. Accordingly, in this article, we propose a versatile attention optimization framework. On the algorithm side, we present a SpQuant algorithm that sparsifies weight matrices offline and input matrices online during linear projection as well as tunes the bit-width of the probabilities matrix according to their importance. On the hardware side, we design SQArch architectures to improve the performance of the SpQuant algorithm. The proposed SQArch architecture offers a low-cost preprocess module that predicts and prunes nonkey elements of the input matrix on the fly. Moreover, we design a compute module that supports sparse-sparse matrix multiplications (SpMSpM) and multiple precision computations on a single systolic array for generality. Furthermore, we can address the underutilization and workload imbalance problems by 1) decoupling the rows in the systolic array for enough flexibility and 2) proposing a workload balance scheme for SpMSpM that allows the array to accept data of similar sparsity, thereby reducing synchronization between computing units. Extensive experiment results demonstrate that SQArch can achieve satisfactory performance speedups and energy saving compared to state-of-the-art designs.

Hybrid convolutional neural network (CNN) and Transformer networks are emerging in computer vision, combining convolutional, linear, and attention layers to achieve high accuracies with moderate model sizes. Developing the accelerators for hybrid networks is pivotal to simultaneously optimize the static matrix multiplication (MM) in convolutional and linear layers, as well as dynamic MM in attention layers. However, the existing accelerators are primarily designed for either CNNs or Transformers, resulting in increased data movement to support dynamic MM and potential under-utilization of hardware for static MM. To enhance computational performance and energy efficiency for hybrid networks, we propose HyCTor, an accelerator featuring flexible output-stationary (OS) and weight-stationary (WS) dataflows, along with a multicore extension for higher throughput. The parallel array of HyCTor supports interlayer slicing and intralayer splicing to improve the utilization for static MM, and enables seamless switching between OS and WS dataflow to minimize the data movement in dynamic MM. By leveraging structured sparsity in OS dataflow and unstructured sparsity in WS dataflow, the computational efficiency is further boosted for each layer through flexible dataflow selection based on the sparsity ratio. Besides, a novel QuadLoop-mesh topology is proposed to address the complex data dependencies in hybrid networks and minimize data transmission distances in the multicore HyCTor. Experimental results on ResNet-18, ViT-B, and TransIAR-AF show that the proposed single-core HyCTor achieves$1.83\times $,$1.65\times $, and$2.41\times $speedup than state-of-the-art (SOTA) accelerators with 100% utilization rate in most layers, and$3.82\times $–$38.5\times $speedup than RTX4090 GPU. The energy efficiency of HyCTor is improved by$1.81\times $–$8.77\times $compared with SOTA accelerators. Moreover, the 4-core HyCTor achieves speedups of$3.32\times $,$2.58\times $, and$2.91\times $, while the 16-core HyCTor achieves speedups of$7.05\times $,$4.05\times $, and$9.64\times $compared to 1-core HyCTor on three networks.

Video Transformers (VidTs) have reached the forefront of accuracy in various video understanding tasks. Despite their remarkable achievements, the processing requirements for a large number of video frames still present a significant performance bottleneck, impeding their deployment to resource-constrained platforms. While accelerators meticulously designed for Vision Transformers (ViTs) have emerged, they may not be the optimal solution for VidTs, primarily due to two reasons. These accelerators tend to overlook the inherent temporal redundancy that characterizes VidTs, limiting their chance for further performance enhancement. Moreover, incorporating a sparse attention prediction module within these accelerators incurs a considerable overhead.

Point Cloud Neural Network (PCNN) plays an essential role in various 3D applications, with some of them even being time-sensitive and safety-critical. However, the large scale of unordered points with lengthy features results in heavy computational workloads, making them far from real-time processing. To address this challenge, we propose MoC, a Morton-code-based fine-grained quantization for accelerating PCNNs. Specifically, we utilize Morton code to capture the spatial locality among points. Then, we gather nearby points with similar features into a region. Considering the similarity in features of nearby points, we propose to decompose features into base and offsets, where the offsets fall within a narrow range. Building upon this, we introduce a two-level mixed-precision quantization. In the first level, we quantize offsets with low precision, while keeping the base in high precision to ensure accuracy. For the second level, noticing the different data distribution of offsets across various regions, we employ two types of low precision at the region level, which provides opportunities to further accelerate feature computations. To support our algorithm, we design a hardware architecture that parallelizes the Morton code path with the critical path. In our extensive experiments on various datasets, our algorithm-architecture co-designed method demonstrates 12X, 6.3X, 4.7X, 3.8X, 3.4X and 2.8X speedup and 19.3X, 9.7X, 6.0X, 5.2X, 4.6X and 4.1X energy savings over CPU, Server and Edge GPUs, state-of-the-art ASICs (incl. PointAcc, MARS, PRADA) with negligible accuracy loss.

Deep Neural Network (DNN) inference consumes significant computing resources and development efforts due to the growing model size. Quantization is a promising technique to reduce the computation and memory cost of DNNs. Most existing quantization methods rely on fixed-point integers or floating-point types, which require more bits to maintain model accuracy. In contrast, variable-length quantization, which combines high precision for values with significant magnitudes (i.e., outliers) and low precision for normal values, offers algorithmic advantages but introduces significant hardware overhead due to variable-length encoding and decoding. Also, existing quantization methods are less effective for both (dynamic) activations and (static) weights due to the presence of outliers.

Attention-based models provide significant accuracy improvement to Natural Language Processing (NLP) and computer vision (CV) fields at the cost of heavy computational and memory demands. Previous works seek to alleviate the performance bottleneck by removing useless relations for each position. However, their attempts only focus on intra-sentence optimization and overlook the opportunity in the temporal domain. In this paper, we accelerate attention by leveraging the tempo-spatial similarity across successive sentences, given the observation that successive sentences tend to bear high similarity. This is rational owing to many semantic similar words (namely tokens) in the attention-based models. We first propose an online-offline prediction algorithm to identify similar tokens/heads. We then design a recovery algorithm so that we can skip the computation on similar tokens/heads in succeeding sentences and recover their results by copying other tokens/heads features in preceding sentences to reserve accuracy. From the hardware aspect, we propose a specialized architecture TSAcc that includes a prediction engine and recovery engine to translate the computational saving in the algorithm to real speedup. Experiments show that TSAcc can achieve 8.5X, 2.7X, 14.1X, and 64.9X speedup compared to SpAtten, Sanger, 1080TI GPU, and Xeon CPU, with negligible accuracy loss.

In the realm of video-oriented tasks, Video Transformer models (VidT), an evolution from vision Transformers (ViT), have demonstrated considerable success. However, their widespread application is constrained by substantial computational demands and high energy consumption. Addressing these limitations and thus improving VidT efficiency has become a hot topic. Current methodologies solve this challenge by dividing a video into several features and applying intra-feature sparsity. However, they neglect the crucial point of inter-feature redundancy and often entail prolonged latency in fine-tuning phases. In response, this paper introduces InterArch, a tailored framework designed to significantly enhance VidT efficiency. We first design a novel inter-feature sparsity algorithm consisting of hierarchical deduplication and recovery. The deduplication phase capitalizes on temporal similarities at both block and element levels, enabling the elimination of redundant computations across features in both coarse-grained and fine-grained manners. To prevent long-latency fine-tuning, we employ a lightweight recovery mechanism that constructs approximate features for the sparsified data. Furthermore, InterArch incorporates a regular dataflow strategy, which consolidates sparse features and effectively translates sparse computations into dense ones. Complementing this, we develop a spatial array architecture equipped with augmented processing elements (PEs), specifically optimized for our proposed dataflow. Extensive experiment results demonstrate that InterArch can achieve satisfactory performance speedups and energy saving.

Spiking neural networks (SNNs) are emerging as energy-efficient alternatives to traditional artificial neural networks (ANNs). Their event-driven information processing significantly reduces computational demands while maintaining competitive performance. However, as SNNs are increasingly deployed in edge devices, security concerns have emerged. While significant research efforts have been dedicated to addressing the security vulnerabilities stemming from malicious input, often referred to as adversarial examples, the security of SNN parameters remains relatively unexplored. This work introduces a novel attack methodology for SNNs known as Energy-Oriented SNN attack (EOS). EOS is designed to increase the energy consumption of SNNs through the malicious manipulation of binary bits within their memory systems (i.e., DRAM), where neuronal information is stored. The key insight of EOS lies in the observation that energy consumption in SNN implementations is intricately linked to spiking activity. The bit-flip operation, the well-known Row Hammer technique, is employed in EOS. It achieves this by identifying the most robust neurons in the SNN based on the spiking activity, particularly those related to the firing threshold, which is stored as binary bits in memory. EOS employs a combination of spiking activity analysis and a progressive search strategy to pinpoint the target neurons for bit-flip attacks. The primary objective is to incrementally increase the energy consumption of the SNN while ensuring that accuracy remains intact. With the implementation of EOS, successful attacks on SNNs can lead to an average of 43% energy increase with no drop in accuracy.

Point-based Point Cloud Neural Networks (PCNNs) have attracted much attention for their higher accuracy than voxel-based and multi-view-based PCNNs. Nevertheless, the increasing scale of point cloud data poses a challenge for real-time processing. Numerous previous works focus on accelerating PCNN inference but only optimize specific stages, limiting their generality to different networks with diverse performance bottlenecks. In this paper, we take nearly all stages of PCNNs into account, and propose 3 orthogonal algorithms, including Fusion-FPS, Fusion-Computation, and Fusion-Aggregation. We introduce Fusion-FPS to alter the sequential execution flow by reducing the Farthest Point Sampling (FPS) across layers to once and organize all neighbor search stages in parallel. To exclude redundant feature computations of “Filling Points”, we propose Fusion-Computation, identifying the presence and locations of “Filling Points” and directly borrowing the nearest neighbor features for them. To eliminate redundant memory accesses caused by shared neighbors in aggregation, we present Fusion-Aggregation, which clusters nearby centroids and coalesces their replicated accesses. In support of our algorithms, we co-design FusionArch, an architecture that implements our strategies and further optimizes memory access via a Local Fusion-Aggregation Table (LFT). We evaluate FusionArch on both server-level and edge-level platforms on 5 PCNNs across 4 applications and show remarkable accuracy and performance gains. On average, FusionArch achieves$2.6\times,5.6\times, 13.0\times$speedup and$17\times, 22\times, 62.4\times$energy savings over PointAcc.Server, NVIDIA AIOO GPU and Intel Xeon CPU, respectively. Moreover, it outperforms PRADA, PointAcc.Edge, Mesorasi and GPU with speedups of$2.4\times, 2.9\times, 5.3\times, 5.5\times$, and energy savings of$4.4\times, 7.2\times, 12.4\times, 11.5\times$, respectively.

Point Cloud Transformer is undergoing a rising trend in both industry and academia. It aligns traditional point cloud feature extraction methods with the latest transformer architecture and achieves remarkable performance. However, accelerators for traditional point cloud neural networks (PCNNs) and those solely for transformers fail to capture the characteristics of point cloud transformers, thus exhibiting poor performance. To address this challenge, we propose Sava, a co-designed accelerator that adopts a spatial- and value-aware hybrid pruning strategy for point cloud transformers. In terms of the spatial domain, we observe that points in regions of various densities exhibit different levels of importance. In the value space, a minor input contributes less to features, indicating lower importance. Considering both perspectives, we hybridize the information inherited from the spatial and value spaces to prune less significant values in attention, which converts data to sparse patterns and makes it readily accelerated. Furthermore, we adopt low-bit quantization to boost computations and apply varying quantization precisions across different network layers based on their sensitivity. In support of our algorithm, we propose an architecture that employs a configurable mixed-precision systolic array for various computing loads under diverse precisions. To address the workload imbalance of the unstructured sparse computations, we introduce a data rearrangement mechanism, which improves resource utilization while hiding latency. We evaluate our Sava on four point cloud transformer models and achieve notable accuracy and performance gains. In comparison with CPU, GPUs, and ASICs, our Sava offers 10.3×, 3.6×, 3.3×, 2.6×, 2.2× speedup, along with 20×, 8.8×, 6.9×, 3.2×, 2.4× energy savings on average.

Deep Neural Networks (DNNs) have demonstrated remarkable success; however, their increasing model size poses a challenge due to the widening gap between model size and hardware capacity. To address this, model compression techniques have been proposed, but existing compression methods struggle to effectively handle the significant parameter variations (activations and weights) within the model. Moreover, current variance-aware encoding solutions for compression introduce complex logic, leading to limited compression benefits and hardware efficiency. In this context, we present SPARK, a novel algorithm/architecture co-designed solution that utilizes variable-length data representation for local parameter value processing, offering low hardware overhead and high-performance gains. Our key insight is that the high-order part in quantized values are often sparse, allowing us to employ an identity bit to assign the appropriate encoding length, thereby eliminating redundant bit-length footprints. This reduction in data representation based on data characteristics enables a serialized structured data encoding scheme that seamlessly integrates with existing hardware accelerators, such as systolic arrays. We evaluate SPARK-based accelerators against some existing encoding-based accelerator, and our results demonstrate significant improvements. The SPARK-based accelerator achieves up to 4.65 × speedup and 74.7% energy reduction, while maintaining superior model accuracy.

Spiking neural networks (SNNs) have been promising applications in the image recognition domain, and their key component is the spiking neuron. SNN s mainly contain integration and firing processes, which are essentially weight accumulation and threshold comparison, respectively. However, spike trains of the neurons exhibit high sparsity and irregularity in both temporal and spatial domains, leading to inefficient memory access and computation. Therefore, designing an efficient accelerator for SNNs is urgent. This paper presents an elaborate accelerator Early in a software-hardware co-design way. At the software level: (i) Noticing the importance of weights, where larger weights disproportionately affect the membrane potential, we devise a weight importance-aware early firing solution for the firing neurons. It prioritizes the accumulation of these large weights, thereby accelerating the membrane potential's rise to surpass the threshold sooner. (ii) Meanwhile, given the observation that a large proportion of neurons do not eventually be fired even after experiencing a long delay of weight accumulation, we propose a weight importance-aware early exit mechanism. It preferentially accumulates large weights and compares the membrane potential with the predetermined threshold, which early halts the accumulation of neurons that are unlikely to be fired, enhancing efficiency. At the hardware level, we design a specialized processing element (PE) featuring the reorder engine for spikes and weights, tailored to realize the aforementioned strategies. Experimental results show that Early averagely achieves 20.3 x, 6.5 x, and 2.4 x speedup compared to the state-of-the-art accelerators Spinalflow, PTB, and SATO. Meanwhile, it averagely achieves 25.2x, 7.4x, and 3.2x energy savings with respect to the three accelerators.

Neural Radiance Field (NeRF) holds immense promise for generating photo-realistic images and videos. How-ever, the computation and memory demands significantly impede its applicability in real-time tasks such as virtual reality and augmented reality. While accelerators dedicatedly designed for NeRF have emerged, they may not be the optimal solution as they only consider the optimization of full-precision NeRF models, leading to unsatisfactory memory and computation saving. This paper proposes Sensitivity-Aware Dynamic Precision Rendering, abbreviated as SRender, an algorithm-hardware co-design framework that borrows the idea of Adaptive Rendering commonly used in conventional graphics rendering to accelerate the rendering process of NeRF exploiting the fine-grained per-ray and per-point levels sensitivity. The fundamental idea behind SRender is that rays and points with high sensitivity require high-precision data types, while the insensitive rays and points do not, allowing them to be sacrificed for greater acceleration. Accordingly, the algorithm part of SRender dynamically configures the precision of the hash table and the multilayer perceptron (MLP), and the hardware part features a specialized accelerator that exploits such mixed-precision memory access and computation patterns. Specifically, we predict the sensitive rays and points based on low-precision hash encoding and feature computation. Then, we trigger a high-precision rendering again on the sensitive ones to guarantee the rendering quality. The hardware design of SRender includes a sensitivity prediction engine, coarse-then-fine-grained encoding engines, and a recovery engine, which translates the computational saving in the algorithm to real speedup. Using the coarse-then-fine-grained encoding engine, we maximize the data reuse on-chip, thereby reducing the stress of off-chip memory bandwidth. Experiments show that the edge-level SRender can achieve 3.94x and 34.06x speedup over state-of-the-art NeRF accelerator NeuRex and Jetson AGX Xavier GPU with negligible rendering quality loss.

In the current landscape, high-resolution (HR) videos have gained immense popularity, promising an elevated viewing experience. Recent research has demonstrated that the video super-resolution (SR) algorithm, empowered by deep neural networks (DNNs), can substantially enhance the quality of HR videos by processing low-resolution (LR) frames. However, the existing DNN models demand significant computational resources, posing challenges for the deployment of SR algorithms on client devices. While numerous accelerators have proposed solutions, their primary focus remains on client-side optimization. In contrast, our research recognizes that the HR video is originally stored in the cloud server and presents an untapped opportunity for achieving both high accuracy and performance improvements. Building on this insight, this article introduces an end-to-end video CODEC-assisted super-resolution (E 2 SR+) algorithm, which tightly integrates the cloud server with the client device to deliver a seamless and real-time video viewing experience. We propose the motion vector search algorithm executed in a cloud server, which can search the motion vectors and residuals for a part of the HR video frames and then pack them as add-ons. We also design an auto-encoder algorithm to down-sample the residuals to save the bitstream cost while guaranteeing the quality of the residuals. Lastly, we propose a reconstruction algorithm performed in the client to quickly reconstruct the corresponding HR frames using the add-ons to skip part of the DNN computations. To implement the E 2 SR+ algorithm, we design corresponding E 2 SR+ architecture in the client, which achieves significant speedup with minimal hardware overhead. Given that the environmental condition varies in the server–client hierarchies, we believe that simply applying E 2 SR+ to all frames is irrational. Accordingly, we offer an environmental condition–aware system to chase the best performance while adapting to the diverse environment. In the system, we design a linear programming (LP) model to simulate the environment and allocate frames to three existing mechanisms. Our experimental results demonstrate that the E 2 SR+ algorithm enhances the peak signal-to-noise ratio by 1.2, 2.5, and 2.3 compared with the state-of-the-art (SOTA) methods EDVR, BasicVSR, and BasicVSR++, respectively. In terms of performance, the E 2 SR+ architecture offers significant improvements over existing SOTA methods. For instance, while BasicVSR++ requires 98 ms on an NVIDIA V100 graphics processing unit (GPU) to generate a 1,280 × 720 HR frame, the E 2 SR+ architecture reduces the execution time to just 39 ms, highlighting the efficiency and effectiveness of our proposed method. Overall, the E 2 SR+ architecture respectively achieves 1.4×, 2.2×, 4.6×, and 442.0× performance improvement compared with ADAS, ISRAcc, the NVIDIA V100 GPU, and a central processing unit. Lastly, the proposed system showcases its superiority and surpasses all the existing mechanisms in terms of execution time when varying environmental conditions.

Graph application is ever-growing in relational data analysis. However, the memory access patterns become the performance bottleneck in graph analytics and graph neural network (GNN) suffering from single-side and dual-side sparsity, separately. Existing resistive random access memory (RRAM)-based processing-in-memory accelerators reduce data movements but fail to handle both types of sparsity in graph data. To address these issues, our work introduces Janus, a flexible highly compact architecture that is capable of being configured to enable single-sparse mode and dual-sparse mode, to accelerate graph analytics and GNN workloads in compressed mapping, respectively. Upon performing graph analytics with single-side sparsity, Janus employs a tandem-isomorphic-crossbar design both to remove zero-stored footprint, and to eliminate redundant search and sequential indexing. To address the challenge of dual-side sparsity in GNN, Janus still takes a random index access mechanism to gather data rapidly and uses a semi-SPM2 compute paradigm to boost the RRAM-based analog multiplication-and-accumulation in the compressed format. Compared with the state-of-the-art works, Janus outperforms them in both performance and energy efficiency for graph analytics and GNN, respectively.

Bit-sparsity has shown its promise in CNN acceleration. However, prior bit-sparse accelerators have two drawbacks: 1) a large number of zero values are involved in the computation and data movement; 2) the distribution of non-zero bits is not considered in PE design. To address these issues, we propose AdaS. At the multiplier level, we dynamically serialize the operands that have fewer non-zero bits. At the dataflow level, we propose a group-wise bi-directional inner-join for workload extraction and balancing. Results show that AdaS can achieve 3.28×, 2.05× speedup, and 1.99×, 1.80× energy efficiency over Bit-Pragmatic and Laconic, respectively.

Recent point cloud recognition (PCR) tasks tend to utilize deep neural network (DNN) for better accuracy. Still, the computational intensity of DNN makes them far from real-time processing, given the fast-increasing number of points that need to be processed. Because the point cloud represents 3D-shaped discrete objects in the physical world using a mass of points, the points tend for an uneven distribution in the view space that exposes strong clustering possibility and local pairs' similarities. Based on this observation, this paper proposes PRADA, an algorithm-architecture co-design that can accelerate PCR while reserving its accuracy. We propose dynamic approximation, which can approximate and eliminate the similar local pairs' computations and recover their results by copying key local pairs' features for PCR speedup without losing accuracy. For accuracy good, we further propose an advanced re-clustering technique to maximize the similarity between local pairs. For performance good, we then propose a PRADA architecture that can be built on any conventional DNN accelerator to dynamically approximate the similarity and skip the redundant DNN computation with memory accesses at the same time. Our experiments on a wide variety of datasets show that PRADA averagely achieves 4.2×, 4.9×, 7.1×, and 12.2× speedup over Mesorasi, V100 GPU, 1080TI GPU, and Xeon CPU with negligible accuracy loss.

Vision Transformer (ViT) has achieved state-of-the-art performance in the computer vision field, showcasing the remarkable potential to become a dominant model in the future. However, the self-attention mechanism within ViT presents significant challenges in terms of computational requirements and storage demands. This limitation becomes particularly pronounced in video recognition tasks, where the computational complexity escalates proportionally with the number of input frames. Current efforts to enhance ViT’s efficiency mainly concentrate on exploiting sparsity within individual frames, which neglects the temporal redundancy across frames, leading to an unsatisfactory solution for video tasks. Alternatively, in this paper, we propose a Vision Transformer acceleration framework called ViTframe, which aims to omit temporal redundancy in the video by dynamically selecting informative frames fed into ViT for fast video recognition. We first introduce an informative frame selection algorithm to pick out the most representative frames from the video for a quick ViT inference and a result compensation mechanism to compensate for the accuracy loss incurred by the informative frame selection. Moreover, we offer a customized architecture to efficiently implement the ViTframe algorithm. We implement ViTframe in a 28nm technology node. Extensive evaluations verify the effectiveness of our proposal on speedup, energy, and accuracy.

Attention-based models, such as transformers, have achieved remarkable success across various tasks. However, their deployment is hindered by challenges such as high memory requirements, long inference latency, and significant power consumption. Quantization has emerged as an effective approach to address these challenges by reducing the bit-width of the model. However, existing quantization algorithms suffer from too coarse-grained quantization granularity or statically determining the bit-width of tokens, lacking the flexibility needed to achieve maximum performance improvement. Accordingly, in this paper, we present a Dynamic Element-wise Quantization (DEQ) algorithm that dynamically tunes tokens’ bit-width according to the importance of elements in the attention possibilities matrix.On the hardware side, we design three versions of DEQ architectures to progressively improve the performance of the DEQ algorithm. The proposed DEQ architecture can address the under-utilization and workload imbalance problems by 1) supporting multiple precision computations on a single systolic array for generality, 2) decoupling the rows in the systolic array for enough flexibility, 3) identifying and parallelizing the independent computations within one systolic array for high parallelism. Extensive experiment results demonstrate that DEQ can achieve satisfactory performance speedups and energy saving compared to state-of-the-art designs.

Nowadays, the significance of multi-tenant deep neural networks (DNNs) has grown exponentially, particularly for cloud providers who execute multiple DNN models on one server to fulfill the users’ requirements while reducing the computational overhead. To satisfy the heavy computation requirement of multi-tenant DNNs, a feasible approach is to establish a multi-core accelerator housing multiple sub-accelerators. Although many researchers have achieved a certain success by designing either offline schedulers for heterogeneous accelerators or real-time schedulers for homogeneous accelerators, they fail to schedule multi-tenant DNNs to both homogeneous and heterogeneous multi-core accelerators in real time given the large search space and restricted overhead constraint.In this paper, we propose RealArch, a novel real-time scheduler that efficiently schedules multi-tenant DNNs to both homogeneous and heterogeneous multi-core accelerators in real-time. The key idea of RealArch is to quickly find the minimal latency of mapping multi-tenant DNNs to sub-accelerators, considering the occupation of DRAM, sub-accelerators, and buffers. To support the key idea, we first establish lightweight estimation models for multiple sub-accelerators to evaluate the Data Movement (DM) and Execution (EX) time when mapping a layer to them. Then, we design a real-time scheduling algorithm to compute and select the mapping solution with minimal latency. Finally, we build a low-cost hardware scheduler to perform the estimation models and the real-time scheduling algorithm. Extensive experiment results verify that RealArch can exceed the baseline Round Robin scheduling algorithm and two state-of-the-art schedulers AI-MT and MAGMA with acceptable hardware overhead.

The deep learning recommendation model (DLRM) plays a crucial role in online services, whose key component is the embedding layer. The embedding layer is to gather and reduce several rows of embedding vectors from the large embedding tables given the input item IDs, which poses challenges due to its memory-intensive nature and becomes a focus of current DLRM accelerators. One potential solution for accelerating DLRM is the use of resistive random access memory (RRAM), which exploits process-in-memory (PIM) capability. However, current RRAM-based DLRM accelerators encounter issues with expensive serial embedding vector searches.Accordingly, this paper proposes a Hybrid CAM-MAC RRAM-based Accelerator (HyAcc) to address the challenges of the embedding layer. Firstly, we recognize that content-addressable-memory (CAM) crossbar can broadcast the input item IDs across all rows to gather the stored item IDs at one cycle. Hence, we design RRAM-based CAM crossbars to gather item IDs efficiently. In the meantime, we utilize the multiplication-and-accumulation (MAC) crossbars to implement the reduction operation in the embedding layer. Whereas, during the gather operation, the RRAM-based CAM crossbar inevitably encounters the access inefficiency problem because only one item ID can be gathered per cycle. To overcome this, we propose the hot/cold item engines containing fine-grained/coarse-grained CAM crossbars for the input item IDs with high-frequency/low-frequency (termed as hot/cold item IDs). Additionally, since the input cold item IDs are unevenly distributed in the coarse-grained CAM crossbars, they may cause the workload imbalance problem. To alleviate it, we present the access-aware dynamic pruning solution to dynamically prune the redundant input cold item IDs and average the workload of the coarse-grained CAM crossbars. Extensive experiments validate the effectiveness of the proposed HyAcc architecture.

Due to the restricted on-chip computing capability for deep neural network (DNN) processing, high-definition video recognition (VOR) task is not easily achievable as a real-time task in a consumer SoC. Despite the fact that many accelerators have been proposed for fast VOR, they remain isolated from a video decoder’s inherent video compression knowledge. Therefore, in this article, we propose a video decoder-assisted neural network acceleration framework for real-time video recognition. First, given the fact that the nonkey frames can be dynamically reconstructed by the key frames with high fidelity during video compression, we propose the VR-DANN algorithm that reconstructs the VOR results of nonkey frames in a similar way so as to save a large amount of NN computing power. In VR-DANN, we leverage motion vectors, the tempo-spatial information already available in the video decoding process to facilitate the recognition process, and propose a lightweight NN-based refinement scheme to suppress the nonpixel recognition noise. Moreover, we consider that there is numerous redundant information in the video frames because the objects of interest usually take a small portion in a video frame. We, therefore, propose the object-based acceleration algorithm (Jigsaw-VOR) to avoid unnecessary computation by dropping out the redundant information in the frames before going through the computing-intensive DNN process. Concretely, we adopt the motion vectors to track the rough position for the objects of interest and then merge them into a consolidated frame for DNN processing like a jigsaw game. The acceleration comes from the processing of much fewer consolidated frames compared to the raw frames in a video stream. The VR-DANN and Jigsaw-VOR can be integrated for further speedup. From the hardware side, we propose the VR-DANN and Jigsaw-VOR architectures to, respectively, accelerate the VR-DANN and Jigsaw-VOR algorithms. These two architectures can be combined to gain higher performance improvement. Our experimental results show that the VR-DANN architecture achieves$2.9\times $performance improvement with less than 1% accuracy loss compared with the state-of-the-art “FAVOS” scheme. In addition, the experimental results show that applying Jigsaw-VOR to all frames can achieve$2.4\times $performance improvement with comparable accuracy compared to FAVOS. By combining VR-DANN and Jigsaw-VOR schemes, the performance improvement can reach up to$3.6\times $.

Graph convolutional network (GCN) is a promising but computing- and memory-intensive learning model. Processing-in-memory (PIM) architecture based on the resistive random access memory-based crossbar (ReRAM crossbar) is a natural fit for GCN inference. It can reduce the data movements and compute the vector-matrix multiplication (VMM) in analog. However, it requires an unbearable crossbar cost to leverage the massive parallelism exhibited in GCNs. First, this article explores the design space for GCN inference on ReRAM crossbars and presents the first PIM-based GCN accelerator named PIMGCN, PIMGCN employs dense data mapping and a search-execute architecture to take full advantage of the intravertex parallelisms with acceptable crossbars cost. Two scheduling strategies for PIMGCN to maximize the intervertex parallelisms and optimize the pipeline are proposed. The optimal scheduling is reduced to a maximum independent set problem, which is solved by a novel node-grouping algorithm. Second, this article explores the task-irrelevant information in the graphs and proposes an adaptively sparsified GCN network targeted for PIMGCN, which is named as ASparGCN. ASparGCN exploits a multilayer perceptron (MLP)-based edge predictor to get edge selection strategies for each GCN layer separately and adaptively in the training stage, and only inferences with the selected edges in the test stage. We design two regularization terms to guide the selection strategies to achieve architecture-friendly sparse graphs for PIMGCN. The overall algorithm-architecture co-design is named as PASGCN. Compared to the state-of-the-art software framework running on Intel Xeon CPU and NVIDIA RTX8000 GPU, PASGCN achieves an average of$16455\times $and$110.7\times $speedup and 8.0E$+ 06\times $and 6.67E$+ 03\times $energy reduction, respectively. Compared with the ASIC accelerator HyGCN (Yan et al., 2020), PASGCN achieves$326.31\times $speedup and$124.8\times $energy reduction.

High-resolution video object recognition (VOR) evolves so fast but is very compute-intensive. This is because VOR leverages compute-intensive deep neural network (DNN) for better accuracy. Although many works have been proposed for speedup, they mostly focus on DNN algorithm and hardware acceleration on the edge side. We observe that most video streams need to be losslessly compressed before going online and an encoder should have all the video information. Moreover, as the cloud should have abundant computing power to handle sophisticated VOR algorithms, we propose to take a one-shot effort for a modified VOR algorithm at the encoding stage in cloud and integrate the full VOR regeneration into a slightly extended decoder on the device. The scheme can enable lightweight VOR with server-class accuracy by simply leveraging the classic and economic video decoder universal to any mobile device. Meanwhile, the scheme can save massive computing power for not repetitively processing the same video on different user devices that makes it extremely sustainable for green computing across the whole network. We propose E 2 -VOR, an end-to-end encoder and decoder architecture for efficient VOR. We carefully design the scheme to have minimum impact on the video bitstream transmitted. In the cloud, the VOR extended video encoder tracks on a macro-block basis and packs intelligent information into the video stream for increased VOR accuracy and fast regenerating process. On the edge device, we extend the traditional video decoder with a small piece of dedicated hardware to enable the efficient VOR regeneration. Our experiment shows that E 2 -VOR can achieve 5.0× performance improvement with less than 0.4% VOR accuracy loss compared to the state-of-the-art FAVOS scheme. On average, E 2 -VOR can run over 54 frames-per-second (FPS) for 480P videos on an edge device.

In point cloud video recognition (PVR) tasks, deep neural networks (DNNs) have been widely adopted to enhance accuracy. However, real-time processing is hindered due to the increasing volume of points and frames that require processes. Point clouds represent 3D-shaped discrete objects using a multitude of points. Consequently, these points often exhibit an uneven distribution in the view space, resulting in strong spatial similarity within each point cloud frame. Taking advantage of this observation, this article introduces PRADA, aPoint CloudRecognitionAcceleration algorithm viaDynamicApproximation. PRADA approximates and eliminates the similar local pairs’ computations and recovers their results by copying dissimilar local pairs’ features for speedup with negligible accuracy loss. Furthermore, considering the slow changes in point cloud frames that lead to the high temporal similarity among points across multiple frames, we design PointV, aPointCloudVideo Recognition Acceleration algorithm, to minimize unnecessary computations of similar points in the temporal domain. Moreover, we propose the PRADA and PointV architectures to accelerate the PRADA and PointV algorithms. These two architectures can be integrated to gain higher performance improvement. Our experiments on a wide variety of datasets show that PRADA averagely achieves about$7\times$speedup over 1080TI GPU. In addition, the experimental results show that the PointV architecture and the integrated architecture can respectively achieve$11.7\times$and$13.9\times$performance improvement with acceptable accuracy compared to the 1080TI GPU.

Nowadays high-resolution (HR) videos have been a popular choice for a better viewing experience. Recent works have shown that super-resolution (SR) algorithms can provide superior quality HR video by applying the deep neural network (DNN) to each low-resolution (LR) frame. Obviously, such per-frame DNN processing is compute-intensive and hampers the deployment of SR algorithms on mobile devices. Although many accelerators have proposed solutions, they only focus on mobile devices. Differently, we notice that the HR video is originally stored in the cloud server and should be well exploited to gain high accuracy and performance improvement. Based on this observation, this paper proposes an end-to-end video CODEC assisted system (E2SR), which tightly couples the cloud server with the device to deliver a smooth and real-time video viewing experience. We propose the motion vector search algorithm executed in the cloud server, which can search the motion vectors and residuals for part of HR video frames and then pack them as addons. We further propose the reconstruction algorithm executed in the device to fast reconstruct the corresponding HR frames using the addons to skip part of DNN computations. We design the corresponding E2SR architecture to enable the reconstruction algorithm in the device, which achieves significant speedup with minimal hardware overhead. Our experimental results show that the E2SR system achieves 3.4x performance improvement with less than 0.56 PSNR loss compared with the state-of-the-art "EDVR" scheme.

Models based on the attention mechanism, i.e. transformers, have shown extraordinary performance in Natural Language Processing (NLP) tasks. However, their memory footprint, inference latency, and power consumption are still prohibitive for efficient inference at edge devices, even at data centers. To tackle this issue, we present an algorithm-architecture co-design with dynamic and mixed-precision quantization, DTQAtten. We present empirically that the tolerance to the noise varies from token to token in attention-based NLP models. This finding leads us to quantize different tokens with mixed levels of bits. Thus, we design a compression framework that (i) dynamically quantizes tokens while they are forwarded in the models and (ii) jointly determines the ratio of each precision. Moreover, due to the dynamic mixed-precision tokens caused by our framework, previous matrix-multiplication accelerators (e.g. systolic array) cannot effectively exploit the benefit of the compressed attention computation. We thus design our accelerator with the variable-speed systolic array (VSSA) and propose an effective optimization strategy to alleviate the pipeline-stall problem in VSSA without hardware overhead. We conduct experiments with existing attention-based NLP models, including BERT and GPT-2 on various language tasks. Our results show that DTQAtten outperforms the previous neural network accelerator Eyeriss by 13.12× in terms of speedup and 3.8× in terms of energy-saving. Compared with the state-of-the-art attention accelerator SpAtten, our DTQAtten achieves at least 2.65× speedup and 3.38× energy efficiency improvement.

Graph application plays a significant role in real-world data computation. However, the memory access patterns become the performance bottleneck of the graph applications, which include low compute-to-communication ratio, poor temporal locality, and poor spatial locality. Existing RRAM-based processing-in-memory accelerators reduce the data movements but fail to address both sparsity and redundancy of graph data. In this work, we present Gzippo, a highly-compact design that supports graph computation in the compressed sparse format. Gzippo employs a tandem-isomorphic-crossbar architecture both to eliminate redundant searches and sequential indexing during iterations, and to remove sparsity leading to non-effective computation on zero values. Gzippo achieves a 3.0× (up to 17.4×) performance speedup, 23.9× (up to 163.2×) energy efficiency over state-of-the-art RRAM-based PIM accelerator, respectively.

Graph convolutional neural networks (GCNs) have been emerging as a promising category of neural network models for extending deep learning to graph data analytics. Serving as a type of semi-supervised models, GCNs need training before being used to extract any input graph’s features. The challenge is that the existing GCN accelerators often target the sparse-dense matrix multiplications (SpDM) in GCN inference while ignoring the compute-intensive GCN training. Obviously, this poses momentous performance demands and design challenges.In this paper, we categorize the computations of GCN training into sparse-sparse matrix multiplications (SpSpM) and sparse-dense matrix multiplications (SpDM); and then introduce the GCNTrain-v1 architecture that uniformly performs both SpSpM and SpDM by the column-wise-product-based method. To ad-dress the bank conflict problem in the GCNTrain-v1 architecture, we further propose the GCNTrain-v2 architecture with the conflict-free bank access strategy. This strategy is able to coalesce all requests to one bank by broadcasting elements. Moreover, to alleviate the workload imbalance problem in the GCNTrain-v2 architecture, we offer the GCNTrain-v3 architecture with the offline reshuffle technique that offline reshuffles and balances the non-zero elements in the matrix before GCN training. Overall, the GCNTrain-v3 architecture implements both SpSpM and SpDM for accelerating GCN training without bank conflict and work-load imbalance problems. On five graph datasets, experiment results demonstrate considerable performance speedups over CPU (80.47×), GPU (10.88×), and GCNAX (1.65×).

Low-precision quantization and sparsity have been widely explored in CNN acceleration due to their effectiveness in reducing computational complexity and memory requirements. However, to support variable numerical precision and sparse computation, prior accelerators design flexible multipliers or sparse dataflow separately. A uniform solution that simultaneously exploits mixed-precision and dual-sided irregular sparsity for CNN acceleration is still lacking. Through an in-depth review of existing precision-scalable and sparse accelerators, we observe that a direct combination of low-level multipliers and high-level sparse dataflow from both sides is challenging due to their orthogonal design spaces. To this end, in this paper, we propose condensed streaming computation. By representing non-zero weights and activations as atomized streams, the low-level mixed-precision multiplication and high-level sparse convolution can be unified into a shared dataflow through hierarchical data reuse. Based on the condensed streaming computation, we propose Ristretto, an atomized architecture that exploits both mixed-precision and dual-sided irregular sparsity for CNN inference. We implement Ristretto in a 28nm technology node. Extensive evaluations show that Ristretto consistently outperforms three state-of-the-art CNN accelerators, including Bit Fusion, Laconic, and SparTen, in terms of performance and energy efficiency.

Deep Neural Networks (DNNs) have gained a strong momentum across various applications in recent years. Meanwhile, they are compute- and memory-intensive as the deep layers induce massive matrix-multiplication operations. The Resistive Random Access Memory (ReRAM) can naturally carry out the matrix-multiplication in memory. Therefore, ReRAM-based accelerators are widely used for deploying DNN applications. Researchers strive to compress DNNs to accelerate DNNs on the ReRAM-based accelerators. However, the existing works focus on ReRAM-crossbar level compression. Such coarse-grained pruning lacks the flexibility for a higher compression rate. In this paper, we present our ReRAM-Sharing, a softwarehardware co-design scheme, to explore fined-grained weight sharing compression for ReRAM-based accelerators. Due to the limits of ADC bandwidth and ADC numbers, DNN computation on ReRAM crossbars is conducted in a smaller granularity, denoted as Operation Unit (OU). Motivated by this, we propose ReRAM-Sharing algorithm that applies weight-sharing on OU- level to exploit fine-grained sparsity. Our proposed ReRAM- Sharing reduces the redundancy of DNNs while maintaining the representation capability. Moreover, as the ReRAM-Sharing algorithm is orthogonal with the traditional pruning techniques, we can integrate them to shrink NN model size further. We then propose the ReRAM-Sharing architecture, which introduces the index table and adders to the traditional ReRAM-based accelerator, to support the ReRAM-Sharing algorithm. Experiment results show that our proposed ReRAM-Sharing achieves up to 59.39x and 14.47x compression ratio with negligible accuracy loss on CIFAR-10 and ImageNet datasets, respectively.

Deep neural networks (DNNs) have gained a strong momentum among various applications. The enormous matrix-multiplication exhibited in the above DNNs is computation and memory intensive. Resistive random-access memory crossbar (RRAM-crossbar) consisting of memristor cells can naturally carry out the matrix-vector multiplication. RRAM-crossbar-based accelerator, therefore, has two orders of magnitude of higher energy-efficiency than conventional accelerators. The imperfect fabrication process of RRAM-crossbars, however, causes various defects and process variations. These fabrication imperfections not only result in significant yield loss but also degrade the accuracy of DNNs executed on the RRAM-crossbars. In this article, we first propose an accelerator-friendly neural-network training method, by leveraging the inherent self-healing capability of the neural network, to prevent the large-weight synapses from being mapped to the imperfect memristors. Next, we propose a dynamic adjustment mechanism to extend the above method for DNNs, such as multilayer perceptrons (MLPs), wherein the imperfect-memristor induced errors can accumulate and magnify through multiple layers. Such off-device training method is a pure software solution, and it is unable to provide enough accuracy for convolutional neural networks (CNNs). Several works propose error-tolerable hardware design by allowing the retraining of CNNs on the RRAM-crossbar. Although this hardware-based on-device training method is effective, the frequent write operation on RRAM-crossbar hurt the endurance of RRAM-crossbars. Consequently, we propose a software and hardware co-design methodology to effectively preserve the classification accuracy of CNN with few on-device training iterations. The experimental results show that the proposed method can guarantee ≤1.1% loss of accuracy for resistance variations in MLP and CNN. Moreover, the proposed method can guarantee ≤1% loss of accuracy even when stuck-at-faults (SAFs) rate = 20%.

To tailor for DNN (Deep Neural Network) acceleration, GPU has migrated to new architectures such as NVIDIA Volta and Turing that incorporate dedicated Tensor Cores. Although good at GEMM (generic matrix-matrix multiplication), Tensor Cores still have inefficiency facing convolutions with certain layer structures. This paper proposes a GPNPU (General-Purpose Neural-network Processing Unit) architecture, which offers another option of direct convolution in GPU. It stitches the direct convolution dataflow into the Tensor Cores with little hardware support, and resorts to regulated data layout with stripe-mined convolution execution to achieve higher performance and power efficiency, while retaining the general programability as GPU. We further apply a unified core design to support varied operand types and precision for higher computing throughput. The evaluation shows that GPNPU can outperform Tensor Cores on typical DNNs by 1.4X for inference (FP16) and 1.2X for training with much reduced power. The INT8 performance even increases to 2.4X. Our study demonstrates that it is possible and appealing to refine the Tensor Cores for greater DNN acceleration, while conforming to GPU architecture for the programmability necessary in future DNN evolution.

The sparsity in the deep neural networks (DNNs) can be leveraged by methods such as pruning and quantization to assist the energy-efficient deployment of large-scale deep neural networks onto hardware platforms, such as GPU and ASIC, for better performance and power efficiency. However, for the metal-oxide resistive random access memory (ReRAM) architecture, the study of energy-efficient methods still shrink the model size or constrain the precision of DNN by leveraging the DNN sparsity. Due to the circuit features of ReRAM, reading bit-0 naturally consumes less energy than reading bit-1. In this paper, we exploit the fine-grained tuning method on the bit-level to reduce energy consumption of ReRAM. Specifically, we present the gradient-search and the weight-group update algorithm, which can significantly unbalance the proportion of bit-1 and bit-0 inside the weights of DNN with negligible NN accuracy loss. Experiments demonstrate that the percentage of bit-0, in some typical convolutional neural networks (CNNs), increases to 33.8%, with less than 0.5% degradation in NN accuracy. The energy reduction can be up to 65%.

Quantization is an effective technique for Deep Neural Network (DNN) inference acceleration. However, conventional quantization techniques are either applied at network or layer level that may fail to exploit fine-grained quantization for further speedup, or only applied on kernel weights without paying attention to the feature map dynamics that may lead to lower NN accuracy. In this paper, we propose a dynamic region-based quantization, namely DRQ, which can change the precision of a DNN model dynamically based on the sensitive regions in the feature map to achieve greater acceleration while reserving better NN accuracy. We propose an algorithm to identify the sensitive regions and an architecture that utilizes a variable-speed mixed-precision convolution array to enable the algorithm with better performance and energy efficiency. Our experiments on a wide variety of networks show that compared to a coarse-grained quantization accelerator like “Eyeriss”, DRQ can achieve 92% performance gain and 72% energy reduction with less then 1% accuracy loss. Compared to the state-of-the-art mixed-precision quantization accelerator “OLAccel”, DRQ can also achieve 21% performance gain and 33% energy reduction with 3% prediction accuracy improvement which is quite impressive for inference.

Nowadays, high-definition video object recognition (segmentation and detection) is not within the easy reach of a real-time task in a consumer SoC due to the limited on-chip computing power for neural network (NN) processing. Although many accelerators have been optimized heavily, they are still isolated from the intrinsic video compression expertise in a decoder. Given the fact that a great portion of frames can be dynamically reconstructed by a few key frames with high fidelity in a video, we envision that the recognition can also be reconstructed in a similar way so as to save a large amount of NN computing power. In this paper, we study the feasibility and efficiency of a novel decoder-assisted NN accelerator architecture for video recognition (VR-DANN) in a conventional SoC-styled design, which for the first time tightly couples the working principle of a video decoder with the NN accelerator to provide smooth high-definition video recognition experience. We leverage motion vectors, the simple tempo-spatial information already available in the decoding process to facilitate the recognition process, and propose a lightweight NN-based refinement scheme to suppress the non-pixel recognition noise. We also propose the corresponding microarchitecture design, which can be built upon any existing commercial IPs with minimal hardware overhead but significant speedup. Our experimental results show that the VR-DANN-parallel architecture achieves 2.9× performance improvement with less than 1% accuracy loss compared with the state-of-the-art "FAVOS" scheme widely used for video recognition. Compared with optical flow assisted "DFF" scheme, it can achieve 2.2× performance gain and 3% accuracy improvement. As to another "Euphrates" scheme, VR-DANN can achieve 40% performance gain and comparable accuracy.

The training phases of Deep neural network (DNN) consumes enormous processing time and energy. Compression techniques utilizing the sparsity of DNNs can effectively accelerate the inference phase of DNNs. However, it can be hardly used in the training phase because the training phase involves dense matrix-multiplication using General Purpose Computation on Graphics Processors (GPGPU), which endorse regular and structural data layout. In this paper, we propose the Approximate Random Dropout that replaces the conventional random dropout of neurons and synapses with a regular and online generated patterns to eliminate the unnecessary computation and data access. We develop a SGD-based Search Algorithm that producing the distribution of dropout patterns to compensate the potential accuracy loss. We prove our approach is statistically equivalent to the previous dropout method. Experiments results on multilayer perceptron (MLP) and long short-term memory (LSTM) using well-known benchmarks show that the speedup rate brought by the proposed Approximate Random Dropout ranges from 1.18-2.16 (1.24-1.85) when dropout rate is 0.3-0.7 on MLP (LSTM) with negligible accuracy drop.

Approximate computing is a promising design paradigm that introduces a new dimension—error—into the original design space. By allowing the inexact computation in error-tolerance applications, approximate computing can gain both performance and energy efficiency. A neural network (NN) is a universal approximator in theory and possesses a high level of parallelism. The emerging deep neural network accelerators deployed with NN-based approximator is thereby a promising candidate for approximate computing. Nevertheless, the approximation result must satisfy the users’ requirement, and the approximation result varies across different applications. We normally deploy an NN-based classifier to ensure the approximation quality. Only the inputs predicted to meet the quality requirement can be executed by the approximator. The potential of these two NNs, however, is fully explored; the involving of two NNs in approximate computing imposes critical optimization questions, such as two NNs’ distinct views of the input data space, how to train the two correlated NNs, and what are their topologies. In this article, we propose a novel NN-based approximate computing framework with quality insurance. We advocate a co-training approach that trains the classifier and the approximator alternately to maximize the agreement of the two NNs on the input space. In each iteration, we coordinate the training of the two NNs with a judicious selection of training data. Next, we explore different selection policies and propose to select training data from multiple iterations, which can enhance the invocation of the approximate accelerator. In addition, we optimize the classifier by integrating a dynamic threshold tuning algorithm to improve the invocation of the approximate accelerator further. The increased invocation of accelerator leads to higher energy efficiency under the same quality requirement. We propose two efficient algorithms to explore the smallest topology of the NN-based approximator and the classifier to achieve the quality requirement. The first algorithm straightforward searches the minimum topology using a greedy strategy. However, the first algorithm incurs too much training overhead. To solve this issue, the second one gradually grows the topology of NNs to match the quality requirement by transferring the learned parameters. Experimental results show significant improvement on the quality and the energy efficiency compared to the existing NN-based approximate computing frameworks.

Neural approximate computing gains enormous energy-efficiency at the cost of tolerable quality-loss. A neural approximator can map the input data to output while a classifier determines whether the input data are safe to approximate with quality guarantee. However, existing works cannot maximize the invocation of the approximator, resulting in limited speedup and energy saving. By exploring the mapping space of those target functions, in this paper, we observe a nonuniform distribution of the approximation error incurred by the same approximator. We thus propose a novel approximate computing architecture with a Multiclass-Classifier and Multiple Approximators (MCMA). These approximators have identica network topologies, and thus can share the same hardware resource in an neural processing unit(NPU) clip. In the runtime, MCMA can swap in the invoked approximator by merely shipping the synapse weights from the on-chip memory to the buffers near MAC within a cycle. We also propose efficient co-training methods for such MCMA architecture. Experimental results show a more substantial invocation of MCMA as well as the gain of energy-efficiency.