Transformers have increasingly become the backbone of modern AI, excelling in tasks across natural language processing (NLP) and computer vision (CV). Despite their effectiveness, calculating them is costly, especially on source-constrained platforms like FPGAs. Among the three main computation modules in a Transformer model (QKV generation, attention computation, FFN), most prior research has primarily focused on the attention computation. However, it is typically the QKV generation and the FFN that dominate the overall computational cost in most cases. To address this gap, we propose SiST, an innovative algorithm-architecture co-design to optimize all three modules of Transformer with both sparsity and token similarity. We first propose a cross-stage sparsity prediction mechanism before QKV generation to minimize computations involved in both QKV generation and attention computing. The prediction mechanism avoids large overhead through off-line weight clustering. Further, we leverage token similarity to adapt token clustering in FFN module, thereby reducing computational load. Moreover, the sparsity of weights in FFN is explored to further optimize the computational load. Our dedicated hardware architecture can effectively handle the unstructured sparsity and token clustering, ensuring high hardware resource utilization. Comprehensive evaluations across multiple benchmarks demonstrate that SiST reduces computational overhead by 45.9% to 52.2% with less than 2% loss in accuracy. It achieves a 2.35× to 8.44× speedup, 1.50× to 14.09× normalized throughput improvement and 2.43× to 3.44× improvement in energy efficiency compared with other FPGA-based Transformer accelerators.

High-performance analog and mixed-signal (AMS) circuits are mainly full-custom designed, which is time-consuming and labor-intensive. A significant portion of the effort is experience-driven, which makes the automation of AMS circuit design a formidable challenge. Large language models (LLMs) have emerged as powerful tools for electronic design automation (EDA) applications, fostering advancements in the automatic design process for large-scale AMS circuits. However, the absence of high-quality datasets has led to issues such as model hallucination, which undermines the robustness of automatically generated circuit designs. To address this issue, this article introduces AMSnet-KG, a dataset encompassing various AMS circuit schematics and netlists. We construct a knowledge graph with annotations on detailed functional and performance characteristics. Facilitated by AMSnet-KG, we propose an automated AMS circuit generation framework that utilizes the comprehensive knowledge embedded in LLMs. The flow first formulate a design strategy (e.g., circuit architecture using a number of circuit components) based on required specifications. Next, matched subcircuits are retrieved and assembled into a complete topology, and transistor sizing is obtained through Bayesian optimization. Simulation results of the netlist are automatically fed back to the LLM for further topology refinement, ensuring the circuit design specifications are met. We perform case studies of operational amplifier and comparator design to verify the automatic design flow from specifications to netlists with minimal human effort. The dataset used in this article is available at https://ams-net.github.io/ .

Transformer models have become popular in various AI applications due to their exceptional performance. However, their impressive performance comes with significant computing and memory costs, hindering efficient deployment of Transformer-based applications. Many solutions focus on leveraging sparsity in weight matrix and attention computation. However, previous studies fail to exploit unified sparse pattern to accelerate all three modules of Transformer (QKV generation, attention computation and FFN). In this paper, we propose FNM-Trans, an adaptable and efficient algorithm-hardware co-design aimed at optimizing all three modules of the Transformer by fully harnessing N : M sparsity. At the algorithm level, we fully explore the interplay of dynamic pruning with static pruning under high N : M sparsity. At the hardware level, we develop a dedicated hardware architecture featuring a custom computing engine and a softmax module, tailored to support varying levels of N : M sparsity. Experiment results show that, our algorithm optimizes accuracy by 11.03% under 2:16 attention sparsity and 4:16 weight sparsity, compared to other methods. Additionally, FNM-Trans achieves speedups of 27.13× and 21.24× over Intel i9-9900X and NVIDIA RTX 2080 Ti, respectively, and outpaces current FPGA-based Transformers by 1.88× to 36.51×.