Neural Vector Search (NVS) has exhibited superior search quality over traditional key-based strategies for information retrieval tasks. An effective NVS architecture requires high recall, low latency, and high throughput to enhance user experience and cost-efficiency. However, implementing NVS on existing neural network accelerators and vector search accelerators is sub-optimal due to the separation between the embedding stage and vector search stage at both algorithm and architecture levels. Fortunately, we unveil that Product Quantization (PQ) opens up an opportunity to break separation. However, existing PQ algorithms and accelerators still focus on either the embedding stage or the vector search stage, rather than both simultaneously. Simply combining existing solutions still follows the beaten track of separation and suffers from insufficient parallelization, frequent data access conflicts, and the absence of scheduling, thus failing to reach optimal recall, latency, and throughput. To this end, we propose a unified and efficient NVS accelerator dubbed NeuVSA based on algorithm and architecture co-design philosophy. Specifically, on the algorithm level, we propose a learned PQ-based unified NVS algorithm that consolidates two separate stages into the same computing and memory access paradigm. It integrates an end-to-end joint training strategy to learn the optimal codebook and index for enhanced recall and reduced PQ complexity, thus achieving smoother acceleration. On the architecture level, we customize a homogeneous NVS accelerator based on the unified NVS algorithm. Each sub-accelerator is optimized to exploit all parallelism exposed by unified NVS, incorporating a structured index assignment strategy and an elastic on-chip buffer to alleviate buffer conflicts for reduced latency. All sub-accelerators are coordinated using a hardware-aware scheduling strategy for boosted throughput. Experimental results show that the joint training strategy improves recall by 4.6% over the separated strategy and accuracy by 43.5% over LUT-NN. NeuVSA achieves $2.82 \times$ to $416.17 \times$ lower latency over CPU, GPU, DFX+ANNA, and PQA+ANNA, and up to $49.60 \times$ and $10.57 \times$ higher average throughput over CPU and GPU, respectively. NeuVSA also reduces chip area by 65.2% over PQA+ANNA.

The advancements in learning-based embedding models have propelled the embedding vector to become a vital data structure for various semantic search applications. The graph-based vector search method, which relies on the k-nearest neighbor (k-NN) graph, has gained immense popularity due to its superior performance in recall and throughput compared to other vector search methods. However, the construction of the k-NN graph is a time-consuming process, which not only affects the efficiency of identifying the optimal k-NN graph but also impacts data updates, especially for large-scale vector datasets. Despite the availability of various software-level optimization methods, such as iterative, refinement, and incremental methods, their peak performance is still limited to the underlying CPU platform. Furthermore, the existing GPU and specialized accelerator solutions only focus on partial k-NN graph construction methods and still encounter low efficiency when executing iterative methods due to irregular memory access, low data reuse, and redundant distance calculations.

Most existing NAS-based multi-modal classification (MMC-NAS) methods are optimized using the classification accuracy.They can not simultaneously provide multiple models with diverse perferences such as model complex and classification performance for meeting different users' demands. Combining NAS-MMC with multi-objective optimization is a nature way for this issue. However, the challenge problem of this solution is the high computation cost. For multi-objective optimization, the computing bottleneck is pareto front search. Some higher-quality MMC models (namely core structures, CSs) consisting of high-quality features and fusion operators are easier to identify. We find that CSs have a close relation with the pareto front (PF), i.e., the individuals lying in PF contain the CSs. Based on the finding, we propose an efficient multi-objective neural architecture search for multi-modal classification by applying CSs to guide the PF search (CoMO-NAS). In conclusion, experimental results thoroughly demonstrate the effectiveness of our CoMO-NAS. Compared to state-of-the-art competitors on benchmark multi-modal tasks, we achieve comparable performance with lower model complexity in shorter search time.

Deep neural networks have shown outstanding performance on complex tasks. Recently, various researches have been developed to pursue fast and energy-efficient deep learning accelerators. However, devices may suffer from hard defects and hardware variability during its lifetime, which poses severe challenges to deep learning accelerators. To protect edge deep learning accelerators from fault-induced failures, we leverage the adversarial deep learning technique to tailor a lightweight online fault detection method for neural network accelerator chips. The proposed Adversarial Testing scheme (AT) is a function-level testing method outcompeting conventional test in several ways: negligible run-time overhead, super sensitivity to subtle hardware variations, which reduces chip over-kills and also the unnecessary diagnosis operations. The evaluation results show that AT can accurately detect fault occurrence and ensure the normal use of deep learning accelerator during its lifetime.

It has recently become trendy for neural network accelerators to scale from single-core to chiplet-based multichip architecture, as the growth of neural network depth and complexity are calling for the promotion of computation and memory capabilities. However, the unintended extensive inter-chip communication of chiplet-based accelerator may bottleneck the parallelism of deep learning inference, which is undesirable for many real-time applications and energy-efficient devices. Although it is imperative for novel schemes to be devised to alleviate this problem, related works are scarce. In this work, we present CampQ, a fine-grained communication-aware mixed-precision quantization method to accelerate inference parallelization by reducing the major inter-chiplet communication overhead. By leveraging the AutoML technique, CampQ is capable of determining different bit-width to activation groups according to thier transmission distances in on-package network. The experimental results show 1.4×-2.6× performance benefits and 29%-60% energy reduction over the 16-bit models for various neural networks and parallelism approaches.

The emerging Resistive RAM (ReRAM) technology significantly boosts the performance and the energy efficiency of the deep learning accelerators (DLAs) via the Computing-in-Memory (CiM) architecture. However, ReRAM-based DLA also suffers a high occurrence rate of memory faults. How to detect and protect against the faults in ReRAM devices poses great challenges to ReRAM-based DLA design. In this work, we propose RRAMedy, an in-situ fault detection and network remedy framework for ReRAM-based DLAs. With the proposed Adversarial Example Testing, which is a lifetime on-device and on-line fault detection technique, it achieves high detection coverage of both hard faults and soft faults at a low run-time cost. In addition, it employs an edge-cloud collaborative model retraining method to tolerate the detected faults by leveraging the inherent fault-adaptive capability of DNNs. Meanwhile, to enable in-situ model remedy when the cloud assistance is absent due to security or overhead issues, we propose to accelerate the fault-masking retraining process on edge devices with parallelized Knowledge Transfer. Our experimental results show that the proposed fault detection technique achieves high fault detection accuracy and delivers real-time testing performance. Meanwhile, the proposed retraining approach greatly alleviates the accuracy degradation problem and achieves excellent performance speedups over the baselines.

This work is oriented at the edge computing scenario that terminal deep learning accelerators use pre-trained neural network models distributed from third-party providers (e.g. from data center clouds) to process the private data instead of sending it to the cloud. In this scenario, the network model is exposed to the risk of being attacked in the unverified devices if the parameters and hyper-parameters are transmitted and processed in an unencrypted way. Our work tackles this security problem by using on-chip memory Physical Unclonable Functions (PUFs) and Processing-In-Memory (PIM). We allow the model execution only on authorized devices and protect the model from white-box attacks, black-box attacks and model tampering attacks. The proposed PUFs-and-PIM based Protection method for neural Models (P3M), can utilize unstable PUFs to protect the neural models in edge deep learning accelerators with negligible performance overhead. The experimental results show considerable performance improvement over two state-of-the-art solutions we evaluated.

The emerging memristor technology is considered a promising solution to the edge-oriented deep learning and neuromorphic processor chips because it enables power-efficient Computing-in-Memory (CiM) and normally-off architecture simultaneously. However, as the analog nature and the immature nano-scale fabrication technology, the memristive cells suffer from manufacturing defects, process variations and aging-induced variations, which may incur system and function failures in applications. How to detect and rescue from the permanent and soft faults poses a significant challenge to the edge ReRAM-based deep learning or neuromorphic chips. In this work, we propose an edge-cloud collaborative framework, RRAMedy, to achieve in-situ fault detection and network remedy for memristor-based neural accelerators. In this framework, we present Adversarial Example Testing, a lifetime on-device fault detection technique, which can accurately detect defected cells and memristor soft faults with high probability and at a low cost. Furthermore, the model accuracy can be restored by the proposed edge-cloud collaborative fault-masking retraining and model updating mechanism with a minimized edge-cloud communication overhead. The experimental results show that RRAMedy can effectively detect the memristor permanent and soft faults, protecting the neural accelerator from accuracy and performance degradation in its life cycle.

There is a sheer growth of attention on the issue of deep learning attacks launched by adversaries. Especially with the spread of edge computing devices that cooperate with the central cloud, how to protect the neural network models and private data from being attacked becomes a hot topic. In this paper, we consider two collaborative edge-cloud deep learning scenarios. The first is that deep learning models are trained on resource-rich cloud and deployed on the terminal deep learning accelerators for delay-sensitive tasks. While in the second scenario, edge collected data is offloaded to the cloud for computationally-intensive tasks. In both scenarios, the valuable pre-trained neural network models and the private data are exposed to the risks of being attacked if they are transmitted and processed in an unencrypted way. To tackle this security problem, we present a lightweight protection scheme towards data-oriented attacks and model-oriented attacks. With the use of on-chip memory Physical Unclonable Functions (PUFs) and Processing-In-Memory (PIM), our method limits the model execution only on specific edge devices and also prevents the unauthorized analysis of private data. Experiments on state-of-the-art deep learning networks show that our method secures edge deep learning models and user-data with a negligible performance overhead.

The shared memory controllers of single-chip cloud computing (SCC) processors are vulnerable to timing-channel attacks. Existing protection strategies based on fixed memory bandwidth assignment degrade processor performance and severely harm the experience of cloud-users. This paper proposes a novel light-weight timing channel protection scheme against both side channel and covert channel attacks to the shared memory controllers of cloud chip multi-processors (CMPs). Instead of enforcing a fixed time-slot assignment to the applications, we go an entirely different technical route and utilize the background DRAM refresh as a free noise source to eliminate the time correlation between victim and attacker applications. The proposed protection framework, MemJam, relies on the emerging fine-grained refresh technology to achieve the effects of timing channel obfuscation. Multi-programmed workloads running in a cloud CMP were used to evaluate the protection method. The results show that the light-weight refresh-based noise can effectively block the timing-channel between user applications, and achieve up to 1.61~2.49X memory performance boost compared to prior solutions.

The shared memory controllers of Single-Chip Cloud computing processors are vulnerable to timing-channel attacks. Existing protection strategies based on fixed memory bandwidth assignment degrade processor performance and severely harm the experience of cloud-users. This paper proposes a novel light-weight timing channel protection scheme against both side channel and covert channel attacks to the shared memory controllers of cloud Chip Multi-Processors (CMPs). Instead of enforcing a fixed time-slot assignment to the applications, we go an entirely different technical route and utilize the background DRAM refresh as a free noise source to eliminate the time correlation between victim and attacker applications. The proposed protection framework, MemJam, relies on emerging Fine-Grained Refresh technology to achieve the effects of timing channel obfuscation. Multi-programmed workloads running in a cloud CMP were used to evaluate the protection method. The results show that the light-weight refresh-based noise can effectively block the timing-channel between user applications, and achieve up to 1.69~3X memory performance boost compared to prior solutions.