Vision Language Models (VLMs) have shown strong performance in multimodal understanding, offering promise for the circuit-to-netlist translation task. However, the diverse component symbols and complex connections in circuit images challenge VLMs in understanding physical layouts and reasoning for electrical connection logic. To address these, we propose Circuit-Think, the first multimodal reasoning framework for the automated circuit-to-netlist translation task, which employs a Trajectory-Guided Reinforcement Learning (TGRL) paradigm for structured logical reasoning on circuit images. Circuit-Think initializes reasoning capabilities through supervised fine-tuning (SFT) on image-netlist pairs, then optimizes reasoning trajectories and netlist generation decisions using TGRL. Firstly, TGRL introduces a step-by-step reasoning paradigm, which guides the model with stepwise reward functions to simulate the human cognitive trajectory of ``identifying ports, recognizing devices, and inferring connections''. Secondly, we customize a multi-level reward that maps reasoning and answers into graph structures and node sets, jointly optimizing logical consistency and netlist accuracy via graph similarity and set matching. Thirdly, TGRL contains a reflective learning mechanism for low-scoring samples, which corrects the reasoning trajectory through reference answers as hints, avoiding local optima caused by sparse reward signals or erroneous reasoning paths. Moreover, we construct a circuit image-netlist reasoning dataset with 3,100 samples, offering step-by-step annotations for converting circuit images to netlists. Extensive experiments demonstrate that Circuit-Think achieves SOTA netlist accuracy and significantly improves the accuracy of downstream tasks.

Convolutional neural networks (CNNs) have been widely applied in lithography hotspot detection due to their strong feature extraction capability; however, low computational efficiency remains a critical bottleneck. Recently, graph neural networks (GNNs) have emerged as a promising alternative, offering both high inference speed and strong scalability to variable-sized inputs. Nevertheless, existing approaches model layouts by decomposing polygons into rectangles, which introduces redundant boundaries and struggles to handle complex non-Manhattan layouts. In this paper, we propose a novel graph representation that accurately extracts the critical geometric features of non-Manhattan layouts by modeling polygon contours. To capture the long-range interactions induced by optical effects, we introduce a hierarchical message-passing mechanism to encode both local and global layout structures efficiently. Furthermore, building on the graph representation, the clip-level labels of non-hotspots can be transformed into edge-level supervision. Accordingly, we incorporate multiple instance learning (MIL) to leverage the fine-grained supervision from non-hotspot clips, thereby enhancing the ability to distinguish between hotspot and non-hotspot clips. Experiments on industrial non-Manhattan datasets demonstrate that our method yields a 3.6% higher recall, 10.8% fewer false alarms, and a 1.7% increase in F1 score compared with the state-of-the-art (SOTA) methods. The industrial non-Manhattan layout used in this work is available at https://github.com/yb-hitsz/DATE2026-GNN4LSD.

Optimizing Electronic Design Automation (EDA) tool parameters with only dozens of affordable evaluations represents one of the most challenging problems in today’s EDA flow management, where each experiment costs hours to days yet directly impacts final PPA outcomes. While Bayesian Optimization (BO) naturally fits such sample-constrained scenarios, it models the entire EDA flow as a monolithic formulation, blindly ignoring the sequential structure that each stage in the EDA flow affects the next. In this work, we propose a stage-aware optimization framework that fundamentally rethinks EDA parameter tuning. The proposed stage-aware Gaussian process explicitly models cascading relationships between EDA stages through interconnected GP layers, extracting abundant information from each expensive evaluation. To better meet realistic needs, we further introduce Expected Hypervolume Improvement (EHVI)-Efficiency, a time-aware acquisition function that exploits evaluation runtime estimation and EDA tools’ checkpoint reuse to balance design metrics’ expected improvement against EDA flow’s computational cost. Experiments and ablation studies on 6 designs across 3 process nodes demonstrate the effectiveness of our proposed method.

Technology node scaling is challenged in many aspects, including pitch reduction, patterning flexibility, and lithography process variability during manufacturing. Without exception, layout hotspot detection, one of the critical steps to achieving design closure, also requires upgrading the associated techniques. With the rapid development of deep learning techniques, the detector exploiting convolutional neural network (CNN) is superior to ones based on pattern matching and classical machine learning algorithms. However, due to the local nature of CNN, the traditional CNN-based detector fails to model the relationship between the patterns in a large-sized layout, resulting in ignoring the impact of light propagation and some optical effects during photolithography. Even worse, another challenge arises from the fact that engineers cannot fully trust the results of learning model-based detectors, especially when handling some complicated layout patterns in practice. This makes it very difficult to deploy the detectors. Observing the facts, we propose a vision transformer (ViT) model-based layout hotspot detector with a deformed attention mechanism, where the training paradigm is inspired by the large pre-trained foundation model (e.g., OpenAI’s GPT-n series) and fine-tuning. Considering the light diffraction during photolithography, the hybrid domain (i.e., spatial and spectral domain) layout inputs via multi-channel are leveraged. Besides, our proposed detector integrates a selective option where the model can choose to do prediction or send to engineers based on the misclassification risk level. Experimental results on the ICCAD2012 metal layer benchmarks and ICCAD2020 via layer benchmarks demonstrate the effectiveness and efficiency of our approach. We have made the ICCAD2020 dataset publicly available to support further research in hotspot detection, enable benchmarking across different process nodes and layout types, and facilitate reproducibility in the field. The dataset is accessible at https: //github.com/shadowior/ICCAD2020.

Modern neural networks have undergone extensive hardware optimization to address the increasing computational demands. While most existing acceleration strategies concentrate on linear operations, the relative cost of these nonlinear operations has become a critical efficiency bottleneck. This work presents QUNF+, a hardware-centric, quadratic-based approximation framework that offers a universal, scalable, and accurate approach for accelerating a wide range of nonlinear functions. Unlike conventional piecewise linear methods, QUNF+ segments functions uniformly and applies second-order Taylor expansions, yielding superior accuracy with fewer segments. QUNF+ also introduces a hardware-efficient approximation scheme with adjustable precision and an optional remainder compensation mechanism. In addition, we propose a set of novel function remapping techniques further reducing approximation error with minimal overhead. We design a fully integrated hardware architecture incorporating Canonical Signed Digit encoding and logic pruning to minimize resource usage without accuracy loss. Experimental results demonstrate that QUNF+ achieves up to 4.1x, 1.5x and 1.7x improvements in power, area, and latency, respectively, over state-of-the-art PWL methods. Application to real-world Transformer models results in an accuracy degradation of less than 0.72% and 0.10% on NLP and CV tasks, with system-level evaluations showing up to 24.38% reduction in total energy consumption and 92.39% in nonlinear operation energy. These results establish QUNF+ as a robust and scalable solution for nonlinear acceleration in modern AI hardware.

Accurate analysis of nanoscale defects in semiconductor manufacturing is essential for optimizing yield and reliability. Existing methods heavily rely on large, labor-intensive datasets and primarily focus on macroscopic defect distributions rather than finer nanoscale defect morphology. In this study, we introduce a novel hybrid weakly supervised segmentation framework for scanning electron microscope (SEM) images, which significantly reduces labeling demands while maintaining high precision. Our approach consists of two interconnected subnetworks: the first is dedicated to precise defect localization and image cropping, and the second performs detailed segmentation of the localized regions. Additionally, we propose an enhanced H-WSSNet that employs Leaky ReLU and a novel multi-level feature fusion mechanism, addressing gradient vanishing during training and improving the model’s adaptive feature fusion and selection capabilities. Extensive validation on a dataset of 1,328 real-world SEM images shows that our model achieves accuracy comparable to fully supervised methods, but with only 10% of the labeling workload. This advancement opens up new possibilities for efficient and scalable high-precision defect segmentation in semiconductor manufacturing.

Large language models (LLMs) have been the driving force behind significant advancements in artificial intelligence. However, their unique self-attention mechanism leads to difficulties in accelerating the inference. Softmax, with its complex nonlinear operations and low parallelism, significantly limits the efficiency of LLMs for long sequences. This work proposes a novel high-parallelism hardware/software co-design Softmax solution. By incorporating a sum estimation algorithm, we eliminate the need for complex exponential on all elements. A high-speed low-energy hardware architecture is introduced by applying high-parallelism statistical module and simplified division module. Experimental results demonstrate that our approach achieves a 9.90–44.75% reduction in latency and up to a 17.36% reduction in energy consumption compared with state-of-the-art Softmax hardware, with minimal impact on model inference accuracy.

Subcircuit matching is widely applied in logic synthesis, design verification, hardware security, etc. Previous works employ redundant circuit representations, coupled with timeconsuming enumerative search methods. Subsequent works use a hybrid “approximate filtering - exact verification” framework, but the numerous false negatives predicted by the graph neural network (GNN) based filtering lead to severe matching failure. In this paper, an improved hybrid method named H3Match is proposed to achieve a better tradeoff between runtime, accuracy, and false negative rate. First, we model the circuits as hypergraphs to fully capture the topology and construct diverse heterogeneous hyperedge features to facilitate the learning of circuit topologies. Second, to reduce the false negatives, we reformulate the subgraph matching problem as matching directed acyclic graphs (DAGs) with embedded circular structure information and develop a directed GNN-based approximate matching approach to identify potential matching subcircuits. Finally, we propose a general mixed integer nonlinear programming (MINLP) formulation for exact verification, with convergency speed accelerated by extracting the initial solution from the results in approximate matching. Experimental results show that our approximate method outperforms state-of-the-art (SOTA) methods by 4.31% in accuracy while achieving virtually zero false negatives. Our exact verification is on average $4.16 \times$ faster than SOTA exact methods. Overall, the end-to-end flow achieves a $7.08 \times$ speedup compared to existing approaches.

A significant challenge in microarchitecture design space exploration (DSE) lies in the time-intensive synthesis and simulation process, making rapid design exploration infeasible. While the simulation tools offer reports on performance, power, and area (PPA) in the different stages, the PPA reports at early stages may fail to reflect the true relative qualities for various designs, i.e., with low fidelities. To address these limitations, we propose a novel multi-fidelity optimization algorithm tailored for multi-stage optimization problems. The proposed method employs a non-linear Gaussian process model to effectively fuse data from different stages with different fidelities, minimizing the need for expensive high-fidelity data while maximizing accuracy. Furthermore, a logical regression function and a multi-objective partial order relation are introduced to evaluate the reliability of low-fidelity data, mitigating their potential inaccuracies. Experiments demonstrate that our proposed multi-fidelity optimization algorithm can approximate the Pareto front of the direct design space in a shorter time with better performance.

The growing complexity of modern hardware has created vast design spaces that are difficult to explore efficiently. Current design space exploration (DSE) methods treat designs as flat parameter vectors, failing to leverage the rich structural information inherent in hardware architectures. This paper presents a novel RTL hierarchy aware approach to microarchitecture DSE that exploits the natural structure of hardware designs. We propose an RTL hierarchy aware kernel that enables direct comparison of RTL hierarchies, preserving their structural characteristics. Our method incorporates module importance derived from hierarchical synthesis reports through a weighted kernel extension. Additionally, we introduce a clustering method that leverages the proposed kernel to identify distinct architectural patterns, enabling efficient parallel evaluation. Experimental results and ablation studies on a Gemmini-based RISC-V SoC demonstrate the superiority of our approach.

Nonlinear functions (NFs) in Transformers require high-precision computation consuming significant time and energy, despite the aggressive quantization schemes for other components. Piece-wise Linear (PWL) approximation-based methods offer more efficient processing schemes for NFs but fall short in dealing with functions with high nonlinearities. Moreover, PWL-based methods still suffer from inevitably high latency introduced by the Multiply-And-Add (MADD) unit. To address these issues, this paper proposes a novel quadratic approximation scheme and a highly integrated, multiplier-less hardware structure, as a unified method to accelerate any unary nonlinear function. We also demonstrate implementation examples for GELU, Softmax, and LayerNorm. The experimental results show that the proposed method achieves up to 5.41% higher inference accuracy and 60.12% lower area-delay product.

Retrieval-Augmented Generation (RAG) incorporates external knowledge to support Large Language Models (LLMs) in generating more accurate, fact-based answers. However, standard RAG methods applied in LLMs lack adaptation to specific domains, limiting their effectiveness in handling complex and specialized knowledge in wafer manufacturing, such as defect root cause analysis, which leads to lower retrieval accuracy and increased large model hallucinations. We propose a wafer-domain-tailored Multimodal Large Language Model (MLLM), FabThink, which aims to optimize the RAG process through a unique multimodal Chain-of-Thought (CoT) framework to address the above issues. Specifically, we propose a "logical decomposition, cross-modal integration, multi-turn retrieval" strategy to refine the process of solving complex queries and enhance the precision of document retrieval. In addition, we introduce an adaptive weighted ranking for critical document selection and fine-tune a text generator to answer wafer-related questions. Experimental results on fab data show that FabThink excels in detection, retrieval, and generation tasks, strongly supporting defect analysis in the integrated circuits (IC) domain.

Reliable, generalizable data foundations are critical for enabling large-scale models in computational lithography. However, essential tasks—mask generation, rule violation detection, and layout optimization—are often handled in isolation, hindered by scarce datasets and limited modeling approaches. To address these challenges, we introduce Unitho, a unified multi-task large vision model built upon the Transformer architecture. Trained on a large-scale industrial lithography simulation dataset with hundreds of thousands of cases, Unitho supports end-to-end mask generation, lithography simulation, and rule violation detection. By enabling agile and high-fidelity lithography simulation, Unitho further facilitates the construction of robust data foundations for intelligent EDA. Experimental results validate its effectiveness and generalizability, with performance substantially surpassing academic baselines.

The increasing complexity of modern SoC designs creates challenges in efficiently exploring vast design spaces. Current approaches often reduce microarchitectures to simple parameter vectors, overlooking their rich information embedded in both functional behaviors and implementation details. This paper proposes an LLM-augmented multi-modal fusion method that captures this dual nature of microarchitecture design. By recombining design parameters with their natural language descriptors, we leverage a domain-knowledge-enhanced LLM to extract semantic features that represent functional behavior. Simultaneously, we process Chisel-compiled RTL through a graph neural network to capture structural implementation details. This multi-modal approach enables more effective feature extraction from limited evaluation data. We integrate these rich features into an MLP enhanced with Monte Carlo dropout. This approach provides uncertainty quantification while enabling end-to-end training, allowing the pre-trained feature extractors to be fine-tuned during exploration through Bayesian optimization. Experimental results and ablation studies on a Gemmini-based RISC-V SoC demonstrate that our approach significantly improves exploration efficiency and prediction quality under data limitations.

The complexity of EDA tools necessitates the development of advanced documentation query answering systems to enhance user efficiency and reduce the associated learning curve. Recent innovations in the use of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) for EDA tool documentation have demonstrated significant progress; however, these approaches typically lack the multi-modal capabilities required to effectively handle visual data, such as circuit layout and GUI screenshots provided through user input. To address the concern, we introduce a multi-modal RAG system that incorporates two domain-customized modules: a multi-modal retriever model finetuned by the customized bilevel hard negative mining (BHNM) strategy, and a vision large language model (VLLM) finetuned using a tailored extract-score-answer pipeline. Moreover, we have manually curated ORD-MMBench, a multi-modal QA benchmark comprising 120 high-quality question-document-answer triplets based on OpenROAD documentation. Experimental results demonstrate that our customized RAG framework outperforms state-of-the-art multi-modal RAG flows and models on ORD-MMBench.

As IC fabrication advances toward smaller process nodes, design technology co-optimization (DTCO) has emerged as a critical enabler of chip performance advancements. Lithography simulation, vital for bridging design and manufacturing, now plays an indispensable role in designing litho-friendly layouts/masks and developing resolution enhancement techniques (RETs). While academia and industry have explored statistical techniques and machine learning models for simulators, the computing paradigm and hardware prevent these solutions from efficiently and accurately simulating the complicated optical imaging coupled with resist film imaging. In this paper, we propose a new simulation paradigm: LMLitho (large vision model-driven lithography simulator), trained on circa one hundred thousand triplets of illumination maps, masks, and resist images. The cross-attention mechanism in our simulator inherently captures diffraction patterns akin to light wave interference within mask features, while hierarchical attention layers enable the modeling of long-range diffraction effects (e.g., proximity effects). A comprehensive dataset encompassing diverse classical types of source and mask patterns, including both metal-1 and via layers, is generated to meet the requirements of training our large vision model-based simulator1. The experimental results demonstrate that our simulator achieves over 120× speedup compared to existing commercial solutions while preserving comparable high fidelity, and exhibits superior generalization to advanced process nodes. When deployed in inverse lithography technology (ILT)-guided mask optimization workflows, masks of higher quality are generated than existing solutions.

Retrieval augmented generation (RAG) improves the accuracy and dependability of generative AI models by integrating factual information from external databases. This technique is widely used in tasks involving document-grounded question answering (QA). While these RAG systems are extensively pretrained on general-purpose documents, they face considerable limitations when applied to specialized, knowledgeintensive fields such as electronic design automation (EDA). This paper addresses such issue by proposing a customized RAG framework along with three domain-specific techniques for EDA tool documentation QA, including a contrastive learning scheme for text embedding model fine-tuning, a reranker distilled from proprietary LLM, and a generative LLM fine-tuned with highquality domain corpus. To further unleash the extraordinary language capacity of LLMs in the domain of EDA-tool documentation QA, we propose to train LLMs as the reranker model with our customized two-stage traning scheme, which consists of the point-wise instruction tuning stage and the pairwise learn-to-rank (LTR) stage. Finally, we have developed and released a documentation QA evaluation benchmark, ORD-QA, for OpenROAD, an advanced RTL-to-GDSII design platform. Experimental results demonstrate that our proposed RAG flow and techniques have achieved superior performance on ORD-QA as well as on a commercial tool, compared with state-of-thearts. Furthermore, compared with the SOTA reranker models, our LLM reranker prominently improves the document retrieval accuracy and thus leads to better QA quality. The ORD-QA benchmark and the training dataset for our customized RAG flow are open-source at https://github.com/lesliepy99/RAG-EDA.

High-speed serial links are essential for low-latency, high-bandwidth communication in data-intensive systems. Signal integrity (SI) of transmitters (TXs) directly impacts transmission quality of the links, while TXs' delay also introduces timing mismatches that degrade link integrity. In this paper, we propose LiTformer, a Transformer-based model for efficient SI analysis of high-speed link TXs, featuring a non-sequential encoder and a multi-head Transformer decoder to incorporate link parameters and capture long-range dependencies. By adopting a nonautoregressive approach, it enables parallel sequence prediction. We also introduce an ANN-based delay model for fast TX delay estimation. Considering link factors including crosstalk in multiple-link systems, LiTformer enables accurate and fast long-sequence signal prediction at high data rates, achieving efficient SI analysis for TXs. Experimental results show LiTformer achieves 437-996 × speedup in eye diagram prediction over SPICE, with mean errors of 0.15-1.57%. It supports 4-bit signals at Gbps data rates for single-ended and differential TXs, including NRZ and PAM4 formats. The delay model predicts TX delay achieving a speedup of four orders of magnitude with errors of 0.86-2.69%.

Emerging domains, such as artificial intelligence, 5G mobile, and automotive, are increasingly reliant on high-speed circuits for efficient processing, in which achieving high operating frequencies and data rates is crucial to enable productive data exchange and rapid responses. High-speed data as well as low noise margin in the high-speed serial links call for efficient models of drivers. In this article, we propose a fast machine-learning-driven macromodel for high-speed drivers, which can efficiently capture the nonlinear characteristics of drivers considering dynamic supply noise with low model complexity. A decoupling-superposition strategy is employed to effectively calculate the impact of power supply noise. Additionally, we introduce a piecewise-segmented method for macromodel solving to further enhance the speed of model utilization. Experimental results demonstrate that compared to HSPICE, the proposed macromodel achieves up to$50\times $–$1200\times $speedup while maintaining sufficient accuracy, even for signals with GHz data rate.

Recent rapid advancements in intelligent vehicular systems and deep learning techniques have led to the emergence of diverse applications utilizing high-quality automotive videos in the Internet-of-Vehicles (IoV), often assisted by uncrewed aerial vehicles (UAVs). These applications aim to provide convenience and security for users. However, transmitting automotive videos with high-quality and low-bit-rate poses a challenge due to the inherent lossiness of traditional compression codecs in current UAV-assisted IoV systems, thereby affecting the performance of subsequent tasks. To address this, we propose a spatial-temporal hybrid video compression framework (STHVC), which integrates Space-Time Super-Resolution (STSR) with conventional codecs to enhance the compression efficiency on automotive videos. In our hybrid design, the encoder generates a low-frame-rate and low-resolution version of the source video, which is then compressed using a traditional codec. During the decoding stage, an effective STSR network is developed to increase both the resolution and the frame rate, and mitigate compression artifacts for automotive videos simultaneously. Additionally, we introduce a rectified intermediate flow estimation technique (RecIFE) within the proposed STSR network to address the challenge of noisy and inaccurate motions during the compression pipeline. Extensive experiments on various benchmark datasets demonstrate that our approach achieves bit-rate reductions of 29.97% compared to H.265 (slow) and 31.27% compared to H.266, while also exhibiting superior restoration performance compared to other state-of-the-art learning-based approaches.

Approximate computing emerges as a viable solution to enhance energy efficiency in applications sensitive to human perception, particularly on edge devices. This work introduces a novel piece-wise approximate floating-point divider that boasts resource efficiency and runtime configurability. Our method leverages a piece-wise approximation algorithm for computing 1/ y by exploiting powers of 2, complemented by an error compensation technique grounded in thorough mathematical analysis. This approach facilitates the realization of a reciprocal-based floating-point divider devoid of multipliers, which not only mitigates hardware resource consumption but also reduces latency. Additionally, we unveil a multi-level runtime configurable hardware architecture that significantly improves flexibility across diverse application contexts. Compared to the existing state-of-the-art approximate dividers and truncated exact dividers, our proposed solution achieves a superior compromise between precision and resource efficiency. Application-level evaluations reveal that our design provides over 87.7% energy saving while maintaining a negligible impact on output quality.

High-level synthesis is an effective methodology that accelerates early-stage circuit design. The optimization of HLS directives has been a critical yet challenging endeavor, with prevailing research primarily concentrating on custom feature engineering and dedicated model designs. However, these conventional approaches often fall short of fully harnessing the intricate latent information embedded within raw HLS directives, potentially limiting the scope and efficiency of optimization processes. In response to these challenges, this article pioneers the integration of large language model (LLM) into the HLS optimization workflow, leveraging their capabilities as both sophisticated feature extractors and autonomous agents. This application of LLM marks a significant departure from traditional methods, introducing a more nuanced and effective strategy for navigating the complex landscape of HLS directive optimization, enabling a more efficient exploration of the design space and prioritization of search strategies. Specifically, our approach makes a significant improvement to the Pareto frontier in directive design, enabling a more rapid and efficient design space exploration. This demonstrates not only an increase in optimization performance but also a decrease in computational overhead, thereby promising significant time savings in the circuit design process. This work not only enhances the current state of HLS directive optimization but also makes new avenues for the application of language models in the field of EDA. Our work makes the following key achievements: We propose an LLM-based framework for effective HLS directives design space exploration; We utilize the prior knowledge of LLM and fine-tune an LLM for HLS directives optimization; Empirical results demonstrate this LLM-based approach’s effectiveness. Specifically, we obtain 15% improvement on the normalized ADRS metric, demonstrating superior performance with limited sampling steps compared with current leading algorithms.

AI chip scales expediently in the large language models (LLMs) era. In contrast, the existing chip design space exploration (DSE) methods, aimed at discovering optimal yet often infeasible or un-produceable Pareto-front designs, are hindered by neglect of design specifications. In this paper, we propose a novel Spec-driven transformed Bayesian optimization framework to find expected optimal RISC-V SoC architecture designs for LLM tasks. The highlights of our framework lie in a tailored transformed Gaussian process (GP) model prioritizing specified target metrics and a customized acquisition function (EHRM) in multi-objective optimization. Extensive experiments on large-scale RISC-V SoC architecture design explorations for LLMs, such as Transformer, BERT, and GPT-1, demonstrate that our method not only can effectively find the design according to QoR values from the spec, but also outperforms 34.59% in ADRS over state-of-the-art approach with only 66.67% runtime overhead.

In semiconductor manufacturing, pinpointing nanoscale wafer defects is crucial for yield and reliability. Deep learning methods for defect segmentation rely heavily on large, labor-intensive datasets and focus mainly on macroscopic wafer defects, not nanoscale morphology. Our research introduces a hybrid weakly supervised scanning electron microscope (SEM) defect segmentation system with two sub-networks: one for accurate defect localization and image cropping, another for detailed segmentation. Validated on 1,328 SEM image defects from a real facility, our model surpasses existing weakly supervised methods and equals fully supervised models in accuracy, with 10% labeling effort, providing a novel approach for high-precision defect segmentation.

As the technology node scales down to 5nml3nm, the consequent difficulty has been widely lamented. The defects on the surface of wafers are much more prone to emerge during manufacturing than ever. What's worse, various single-type defect patterns may be coupled on a wafer and thus shape a mixed-type pattern. To improve yield during the design cycle, mixed-type wafer defect pattern recognition is required to perform to identify the failure mechanisms. Based on these issues, we revisit failure dies on wafer maps by treating them as point sets in two-dimensional space and propose a two-stage classification framework, PoLM. The challenge of noise reduction is considerably improved by first using an adaptive alpha-shapes algorithm to extract intricate geometric features of mixed-type patterns. Unlike sophisticated frameworks based on CNNs or Transformers, PoLM only completes classification within a point cloud cluster for aggregating and dispatching features. Furthermore, recognizing the remarkable success of large pre-trained foundation models (e.g., OpenAI's GPT-n series) in various visual tasks, this paper also introduces a training paradigm leveraging these pre-trained models and fine-tuning to improve the final recognition. Experiments demonstrate that our proposed framework significantly surpasses the state-of-the-art methodologies in classifying mixed-type wafer defect patterns.

Improving the outcomes of very-large-scale integration design without altering the underlying design enablement, such as process, device, interconnect, and IPs, is critical for integrated circuit (IC) designers. Parameter tuning for electronic design automation (EDA) tools is an emerging technology for improving the final design Quality-of-Result (QoR). However, many complex heuristics have been accreted upon previous complex heuristics integrated into tools, resulting in a vast number of tunable parameters. Even worse, these parameters include both continuous and discrete ones, making the parameter tuning process laborious and challenging. In this paper, we propose an attention-based EDA tool parameter explorer. A self-attention mechanism is developed to navigate the parameter importance. A hybrid space Gaussian process model is leveraged to optimize continuous and discrete parameters jointly, capturing their complex interactions. In addition, considering multiple QoR metrics and the large amount of time required to invoke EDA tools, a customized acquisition function based on expected hypervolume improvement (EHVI) is proposed to enable multi-objective optimization and parallel evaluation. Experimental results on a set of IWLS2005 benchmarks demonstrate the effectiveness and efficiency of our method.

Approximate computing is a promising alternative to improve energy efficiency for human perception related applications on the edge. This work proposes a piece-wise approximate floating-point divider, which is resource-efficient and run-time configurable. We provide a piece-wise approximation algorithm for 1/ y, utilizing powers of 2. This approach enables the implementation of a reciprocal-based floating-point divider that is independent of multipliers, which not only reduces hardware consumption but also results in shorter latency. Furthermore, a multi-level run-time configurable hardware structure is intro-duced, enhancing the adaptability to various application scenarios. When compared to the prior state-of-the-art approximate divider, the proposed divider strikes an advantageous balance between accuracy and resource efficiency. The application-level evaluation of the proposed dividers demonstrates manageable and minimal degradation of the output quality when compared to the exact divider.

Large Language Models (LLMs) have revolutionized many domains with exceptional performance while their large sizes hinder their broad applicability, especially in the edge computation scenarios. Designing large-scale LLM-specific accelerators is also challenging, suffering from the complicated, cumbersome, and time-consuming design, simulation, and optimization process. This paper meticulously proposes an agile framework for accelerator development, supporting efficient LLM inference. Firstly, we investigate the architecture of LLMs, uncover performance bottlenecks, and design an optimized binarized accelerator and a configurable RISC-V-based SoC to boost the inference of binary LLMs. Further, a novel fidelity-driven method is proposed to learn the multi-fidelity representation, solving the modeling and accuracy issues due to the lack of accurate later-stage data in the EDA flow, by capturing complex relationships among simulation metrics in and across different fidelities. Tailored strategies across model preparation, backend kernel implementations, agile accelerator and SoC design, and inference simulation are incorporated into our framework to refine the development workflow. Our method significantly accelerates the hardware design, simulation, and optimization processes. Experimental results illustrate the impressive speed and effectiveness of our framework in designing edge LLM accelerators and optimizing LLM inference.

In the tide of explosive development in artificial intelligence (AI), the design of AI System-on-Chips (SoCs) is an urgently pressing issue that needs to be addressed. The application of Design Space Exploration (DSE) methods is paramount in pursuing a sound microarchitecture design and improving the quality of results. However, the high-dimensional design parameters and huge design space, which normally occur in the complicated SoCs for Large Language Model (LLM) tasks, pose a great challenge to existing techniques. In this paper, a novel and explainable Bayesian optimization-based framework MCT-Explorer is proposed. A Monte Carlo Tree Search (MCTS)-based method is utilized to analyze the importance of design parameters, guide the sampling directions, mitigate low-quality performance modeling issues, and further improve optimization efficiency. Besides, an information-guided multi-objective optimization function is adopted to balance the multiple metrics (e.g., Cycle. Area, and Power) for SoC design. Our approach can provide guiding opinions and deeper insights for parameter optimization, thus transcending previous arts and achieving an explainable model. Experiment results demonstrate the extraordinary performance of our framework in various high-dimensional (up to hundreds of parameters) and complicated LLM SoC designs.

Intelligence is key to advancing integrated circuit (IC) fabrication. Recent breakthroughs in Large Multimodal Models (LMMs) have unlocked extraditionary abilities in understanding images and text, fostering intelligent fabrication. Leveraging the power of LMMs, we introduce FabGPT, a customized IC fabrication large multimodal model for wafer defect knowledge query. FabGPT manifests expertise in conducting defect detection in Scanning Electron Microscope (SEM) images, performing root cause analysis, and providing expert Q&A on fabrication processes. FabGPT matches enhanced multimodal features to automatically detect minute defects under complex wafer backgrounds and reduce the subjectivity of manual threshold settings. Besides, the proposed modulation module and interactive corpus training strategy embed wafer defect knowledge into the pre-trained model, effectively balancing Q&A queries related to defect knowledge and original knowledge and mitigating the modality bias issues. Experiments on in-house fab data show that FabGPT achieves significant performance improvement in wafer defect detection and knowledge querying.

In the field of integrated circuit manufacturing, the detection and classification of nanoscale wafer defects are critical for subsequent root cause analysis and yield enhancement. The complex background patterns observed in scanning electron microscope (SEM) images and the diverse textures of the defects pose significant challenges. Traditional methods usually suffer from insufficient data, labels, and poor transferability. In this paper, we propose a novel few-shot learning approach, SEM-CLIP, for accurate defect classification and segmentation. SEM-CLIP customizes the Contrastive Language-Image Pretraining (CLIP) model to better focus on defect areas and minimize background distractions, thereby enhancing segmentation accuracy. We employ text prompts enriched with domain knowledge as prior information to assist in precise analysis. Additionally, our approach incorporates feature engineering with textual guidance to categorize defects more effectively. SEM-CLIP requires little annotated data, substantially reducing labor demands in the semiconductor industry. Extensive experimental validation demonstrates that our model achieves impressive classification and segmentation results under few-shot learning scenarios.

Machine learning on point clouds is increasingly accessible at the edge, notably in applications such as autonomous driving. However, the sparse and irregular nature of point clouds presents significant latency challenges on general-purpose hardware. RISC-V, with its evolving ecosystem, offers a promising platform for embedding intelligence at the edge due to its full-stack scalability. This paper focuses on the advanced point cloud operation known as submanifold convolution (SC), deploying submanifold sparse convolutional networks (SSCNs) on a RISC-V System-on-Chip (SoC) designed within the Chipyard framework. We address three critical bottlenecks of SSCNs- Rule Map Construction (Mapping), Gather-MatMul-Scatter (GMS), and uncombined operation - to meet the real-time inference requirement for the on-chip implementation. By leveraging the RISC-V Vector extension and Gemmini, an open-source full-stack DNN accelerator generator, we vectorize the Mapping process, offload GEMM-related operations to the Gemmini Systolic Array, and cooperatively use the Systolic Array and vector processing units to reduce the memory footprint. Our evaluations show that the RISC-V-based SSCNs implementation achieves an average of 11.73× and 13.1× overall speedups with a small workload compared to TorchSparse on Edge-CPU, a state-of-the-art point cloud inference engine, for 3D segmentation and detection tasks, respectively. When contrasting with TorchSparse on an Edge-GPU, our implementation still delivers a notable improvement, with average speedups of 1.63× for 3D segmentation and 1.07× for detection tasks.

Nowadays, Large Language Models (LLMs) mostly possess billions of parameters, bringing significant challenges to hardware platforms. Although quantization is an efficient approach to reduce computation and memory overhead for inference optimization, we stress the challenge that mainstream low-bit quantization approaches still suffer from either various data distribution outliers or a lack of hardware efficiency. We also find that low-bit data format has further potential expressiveness to cover the atypical language data distribution. In this paper, we propose a novel numerical representation, Bi-Exponent Block Floating Point (BiE), and a new quantization flow. BiE quantization shows accuracy superiority and hardware friendliness on various models and benchmarks.

Traditional server failure prediction methods predominantly rely on single-modality data such as system logs or system status curves. This reliance may lead to an incomplete understanding of system health and impending issues, proving inadequate for the complex and dynamic landscape of contemporary cloud computing environments. The potential of multimodal data to provide comprehensive insights is widely acknowledged, yet the lack of a holistic dataset and the challenges inherent in integrating features from both structured and unstructured data have impeded the exploration of multimodal-based server failure prediction. Addressing these challenges, this paper presents an industrial-scale, comprehensive dataset for server failure prediction, comprising nearly 80 types of structured and unstructured data sourced from real-world industrial cloud systems 1. Building on this resource, we introduce MISP, a model that leverages multimodal fusion techniques for server failure prediction. MISP transforms multimodal data into multi-dimensional sequences, extracts and encodes features both within and across the modalities, and ultimately computes the failure probability from the synthesized features. Experiments demonstrate that MISP significantly outperforms existing methods, enhancing prediction accuracy by approximately 25% over previous state-of-the-art approaches.

Multi-modality image fusion (MMIF) aims to integrate the complementary features of source images into the fused image, including target saliency and texture specifics. Recently, image fusion methods leveraging diffusion models have demonstrated commendable results. Despite their strengths, diffusion models reduce the capability to perceive local features. Additionally, their inherent working mechanism, introducing noise to the inputs, consequently leads to a loss of original information. To overcome this problem, we propose a novel Diffusion-CNN feature Aggregation Fusion (DCAFuse) network that can extract complementary features from the dual branches and aggregate them effectively. Specifically, we utilize the denoising diffusion probabilistic model (DDPM) in the diffusion-based branch to construct global information, and multi-scale convolutional kernels in the CNN-based branch to extract local detailed features. Afterward, we design a novel complementary feature aggregation module (CFAM). By constructing coordinate attention maps for features, CFAM captures long-range dependencies in both horizontal and vertical directions, thereby dynamically guiding the aggregation weights of branches. In addition, to further improve the complementarity of dual-branch features, we introduce a novel loss function based on cosine similarity and a unique denoising timestep selection strategy. Extensive experimental results show that our proposed DCAFuse outperforms other state-of-the-art methods in multiple image fusion tasks, including infrared and visible image fusion (IVF) and medical image fusion (MIF).

Deep learning frameworks or compilers optimize the operators in computation graph using fixed templates via significant engineering efforts, which may miss potential optimizations such as operator fusion. Therefore, automatically implementing and optimizing the emerging new combinations of operators on a specific hardware accelerator is of importance. In this article, we introduce GTCO, a tensor compilation system designed to accelerate transformer-based vision models’ inference on GPUs. GTCO tackles the operator fusion techniques in the transformer-based model using a novel dynamic programming algorithm and proposes a search policy with new sketch generation rules for the fused batch matrix multiplication and softmax operators. Tensor programs are sampled from an effective search space, and a hardware abstraction with hierarchical mapping from tensor computation to domain-specific accelerators (Tensor Cores) is formally defined. Finally, our framework can map and transform tensor expression into efficient CUDA kernels with hardware intrinsics on GPU. Our experimental results demonstrate that GTCO improves the end-to-end execution performance by up to$1.73\times $relative to the cutting-edge deep learning library TensorRT on NVIDIA GPUs with Tensor Cores.

Microarchitecture parameters tuning is critical in the microprocessor design cycle. It is a non-trivial design space exploration (DSE) problem due to the large solution space, cycle-accurate simulators’ modeling inaccuracy, and high simulation runtime for performance evaluations. Previous methods require massive expert efforts to construct interpretable equations or high computing resource demands to train black-box prediction models. This article follows the black-box methods due to better solution qualities than analytical methods in general. We summarize two learned lessons and propose BOOM-Explorer accordingly. First, embedding microarchitecture domain knowledge in the DSE improves the solution quality. Second, BOOM-Explorer makes the microarchitecture DSE for register-transfer-level designs within the limited time budget feasible. We enhance BOOM-Explorer with the diversity-guidance, further improving the algorithm performance. Experimental results with RISC-V Berkeley-Out-of-Order Machine under 7-nm technology show that our proposed methodology achieves an average of 18.75% higher Pareto hypervolume, 35.47% less average distance to reference set, and 65.38% less overall running time compared to previous approaches.

Transistor aging leads to the deterioration of analog circuit performance over time. The worst aging degradation is used to evaluate the circuit reliability. It is extremely expensive to obtain it since several circuit stimuli need to be simulated. The worst degradation collection cost reduction brings an inaccurate training dataset when a machine learning (ML) model is used to fast perform the estimation. Motivated by the fact that there are many similar subcircuits in large-scale analog circuits, in this article we propose Wages to train an ML model on an inaccurate dataset for the worst aging degradation estimation via a domain generalization technique. A sampling-based method on the feature space of the transistor and its neighborhood subcircuit is developed to replace inaccurate labels. A consistent estimation for the worst degradation is enforced to update model parameters. Label updating and model updating are performed alternately to train an ML model on the inaccurate dataset. Experimental results on the very advanced 5 nm technology node show that our Wages can significantly reduce the label collection cost with a negligible estimation error for the worst aging degradations compared to the traditional methods.

Deep learning frameworks optimize the computation graphs and intra-operator computations to boost the inference performance on GPUs, while inter-operator parallelism is usually ignored. In this paper, a unified framework, AutoGraph, is proposed to obtain highly optimized computation graphs in favor of parallel executions of GPU kernels. A novel dynamic programming algorithm, combined with backtracking search, is adopted to explore the optimal graph optimization solution, with the fast performance estimation from the mixed critical path cost. Accurate runtime information based on GPU Multi-Stream launched with CUDA Graph is utilized to determine the convergence of the optimization. Experimental results demonstrate that our method achieves up to 3.47x speedup over existing graph optimization methods. Moreover, AutoGraph outperforms state-of-the-art parallel kernel launch frameworks by up to 1.26x.

The ongoing evolution in process fabrication enables us to step below the 5nm technology node. Although foundries can pattern and etch smaller but more complex circuits on silicon wafers, a multitude of challenges persist. For example, defects on the surface of wafers are inevitable during manufacturing. To increase the yield rate and reduce time-to-market, it is vital to recognize these failures and identify the failure mechanisms of these defects. Recently, applying machine learning-powered methods to combat single defect pattern classification has made significant progress. However, as the processes become increasingly complicated, various single-type defect patterns may emerge and be coupled on a wafer and thus shape a mixed-type pattern. In this paper, we will survey the recent pace of progress on advanced methodologies for wafer failure pattern recognition, especially for mixed-type one. We sincerely hope this literature review can highlight the future directions and promote the advancement of the wafer failure pattern recognition.

In recent years, memristive crossbar-based neuromorphic computing systems (NCS) have provided a promising solution to the acceleration of neural networks. However, stuck-at faults (SAFs) in the memristor devices significantly degrade the computing accuracy of NCS. Besides, the memristor suffers from the process variations, causing deviation of the actual programming resistance from its target resistance. In this paper, we propose a reliability-driven design framework for a memristive crossbar-based NCS in combination with general and chip-specific design optimizations. First, we design a general reliability-aware training scheme to enhance the robustness of NCS to SAFs and device variations; a dropconnect-inspired approach is developed to alleviate the impact of SAFs; a new weighted error function, including cross-entropy error (CEE), the$l_{2}$-norm of weights, and the sum of squares of first-order derivatives of CEE with respect to weights, is proposed to obtain a smooth error curve, where the effects of variations are suppressed. Second, given the neural network model generated by the reliability-aware training scheme, we exploit chip-specific mapping and re- training to further improve computation accuracy loss incurred by SAFs. Experimental results show that the proposed method can boost the computation accuracy of NCS and improve the NCS robustness. Note to Practitioners—This work is motivated by the manufacturing reliability problem in a memristive crossbar-based NCS. To enhance the robustness of an NCS to SAFs and device variations, this paper presents a reliability-driven design framework with taking account of both general and chip-specific design optimizations. The experimental results have demonstrated that the proposed framework is superior to the prior arts, and can be easily integrated with existing industrial hardware-based fault tolerance solutions for higher accuracy at lower overhead. Memristive crossbar-based computing system gives hope for the anticipated efficient implementation of artificial neuromorphic networks. With the help of the reliability-driven designs, the computation accuracy is restored, and hence we can expect the wide use of memristive crossbar-based computing system in neuromorphic computing applications.

Over the past few years, super-resolution (SR) processing has achieved astonishing progress along with the development of deep learning. Nevertheless, the rigorous requirement for real-time inference, especially for video tasks, leaves a harsh challenge for both the model architecture design and the hardware-level implementation. In this article, we propose a hardware-aware acceleration on embedded GPU devices as a full-stack SR deployment framework. The most critical stage with dictionary learning applied in SR flow was analyzed in details and optimized with a tailored dictionary slimming strategy. Moreover, we also delve into the programming architecture of hardware while analyzing the model structure to optimize the computation kernels to reduce inference latency and maximize the throughput given restricted computing power. In addition, we further accelerate the model with 8-bit integer inference by quantizing the weights in the compressed model. An adaptive 8-bit quantization flow for SR task enables the quantized model to achieve a comparable result with the full-precision baselines. With the help of our approaches, the computation and communication bottlenecks in the deep dictionary learning-based SR models can be overcome effectively. The experiments on both edge embedded device NVIDIA NX and 2080Ti prove that our framework exceeds the performance of state-of-the-art NVIDIA TensorRT significantly and can achieve real-time performance.

Pursuing accurate and robust recognizers has been a long-lasting goal for scene text recognition (STR) researchers. Recently, attention-based methods have demonstrated their effectiveness and achieved impressive results on public benchmarks. The attention mechanism enables models to recognize scene text with severe visual distortions by leveraging contextual information. However, recent studies revealed that the implicit over-reliance of context leads to catastrophic out-of-vocabulary performance. On the contrary to the superior accuracy of the seen text, models are prone to misrecognize unseen text even with good image quality. We propose a novel framework, Context-based contrastive learning (ConCLR), to alleviate this issue. Our proposed method first generates characters with different contexts via simple image concatenation operations and then optimizes contrastive loss on their embeddings. By pulling together clusters of identical characters within various contexts and pushing apart clusters of different characters in embedding space, ConCLR suppresses the side-effect of overfitting to specific contexts and learns a more robust representation. Experiments show that ConCLR significantly improves out-of-vocabulary generalization and achieves state-of-the-art performance on public benchmarks together with attention-based recognizers.

As the technology node of integrated circuits rapidly goes beyond 5nm, synthesis-centric modern very large-scale integration (VLSI) design flow is facing ever-increasing design complexity and suffering the pressure of time-to-market. During the past decades, synthesis tools have become progressively sophisticated and offer countless tunable parameters that can significantly influence design quality. Nevertheless, owing to the time-consuming tool evaluation plus a limitation to one possible parameter combination per synthesis run, manually searching for optimal configurations of numerous parameters proves to be elusive. What's worse, tiny perturbations to these parameters can result in very large variations in the Quality-of-Results (QoR). Therefore, automatic tool parameter tuning to reduce human cost and tool evaluation cost is in demand. Machine-learning techniques provide chances to enable the auto-tuning process of tool parameters. In this paper, we will survey the recent pace of progress on advanced parameter auto-tuning flows of physical synthesis tools. We sincerely expect this survey can enlighten the future development of parameter auto-tuning methodologies.

Domain generalization refers to the problem of training a model from a collection of different source domains that can directly generalize to the unseen target domains. A promising solution is contrastive learning, which attempts to learn domain-invariant representations by exploiting rich semantic relations among sample-to-sample pairs from different domains. A simple approach is to pull positive sample pairs from different domains closer while pushing other negative pairs further apart. In this paper, we find that directly applying contrastive-based methods (e.g., supervised contrastive learning) are not effective in domain generalization. We argue that aligning positive sample-to-sample pairs tends to hinder the model generalization due to the significant distribution gaps between different domains. To address this issue, we propose a novel proxy-based contrastive learning method, which replaces the original sample-to-sample relations with proxy-to-sample relations, significantly alleviating the positive alignment issue. Experiments on the four standard benchmarks demonstrate the effectiveness of the proposed method. Furthermore, we also consider a more complex scenario where no ImageNet pre-trained models are provided. Our method consistently shows better performance.

It is an open problem to compile DNN models on GPU and improve the performance. A novel framework, GTuner, is proposed to jointly learn from the structures of computational graphs and the statistical features of codes to find the optimal code implementations. A Graph ATtention network (GAT) is designed as the performance estimator in GTuner. In GAT, graph neural layers are used to propagate the information in the graph and a multi-head self-attention module is designed to learn the complicated relationships between the features. Under the guidance of GAT, the GPU codes are generated through auto-tuning. Experimental results demonstrate that our method outperforms the previous arts remarkably.

Continuous scaling of the very-large-scale integration system leaves a significant challenge on manufacturing; thus optical proximity correction (OPC) is widely applied in conventional design flow for manufacturability optimization. Traditional techniques conduct OPC by leveraging a lithography model but may suffer from prohibitive computational overhead. In addition, most of them focus on optimizing a single and local clip instead of addressing how to tackle the full-chip scale. In this article, we present DAMO, a high-performance and scalable deep-learning-enabled OPC system for full-chip scale. It is an end-to-end mask optimization paradigm that contains a deep lithography simulator (DLS) for lithography modeling and a deep mask generator (DMG) for mask pattern generation. Moreover, a novel layout splitting algorithm customized for DAMO is proposed, composed of DBSCAN clustering and KMeans++ clustering, to handle the full-chip OPC problem. Further, graph-based computation and parallelism techniques are proposed to deploy our GPU algorithms to accelerate computations. Extensive experiments show that DAMO outperforms state-of-the-art OPC solutions in both academia and industrial commercial toolkit.

With continued scaling, the transistor aging induced by hot carrier injection (HCI) and bias temperature instability (BTI) causes an increasing failure of nanometer-scale integrated circuits (ICs). Compared to digital ICs, analog ICs are more susceptible to aging effects. The industrial large-scale analog ICs bring grand challenges in the efficiency of aging verification. In this article, we propose a heterogeneous graph convolutional network (H-GCN) to fast estimate aging-induced transistor degradation in analog ICs. To characterize the multityped devices and connection pins, a heterogeneous directed multigraph is adopted to efficiently represent the topology of analog ICs. A latent space mapping method is used to transform the feature vector of all typed devices into a unified latent space. We further extend the proposed H-GCN to be a deep version via initial residual connections and identity mappings. The extended deep H-GCN can extract information from multihop devices without an oversmoothing issue. A probability-based neighborhood sampling method on the bipartite graph is adopted to ease the model training on large-scale graphs and achieve good scalability. Experiments on very advanced 5-nm industrial benchmarks show that, compared to traditional graph learning methods and static aging reliability simulations by an industrial design-for-reliability (DFR) tool, the proposed deep H-GCN can achieve more accurate estimations of aging-induced transistor degradation. Compared to the dynamic and static aging reliability simulations, our extended deep H-GCN, on average, can achieve$241\times $and$39\times $speedup, respectively.

This article studies the robust deep stereo vision in autonomous driving systems and counteracting adversarial attacks. The autonomous system operation requires real-time processing of measurement data which often contain significant uncertainties and noise. Adversarial attacks have been widely studied to simulate these perturbations in recent years. To counteract the practical attacks in autonomous systems, novel methods based on simulated attacks are proposed in this article. Univariate and multivariate functions are adopted to represent the relationships between the left and right input images and the deep stereo model. A stereo regularizer is proposed to guide the model to learn the implicit relationship between the images and characterize the loss function’s local smoothness. The attacks are generated by maximizing the regularizer term to break the linearity and smoothness. The model then defends the attacks by minimizing the loss and regularization terms. Two techniques are developed in this article. The first technique,SmoothStereo, explores the basic knowledge from the physical world and smoothness, while the second technique,SmoothStereoV2, improvesSmoothStereothrough leveraging the smooth activation functions during the defense.SmoothStereoV2can learn and utilize the gradient information concerning the attacks. The gradients of the smooth activation functions can handle attacks for improving the model robustness. Numerical experiments on KITTI datasets demonstrate that the proposed methods offer superior performance.

Compared with traditional convolutions, grouped convolutional neural networks are promising for both model performance and network parameters. However, existing models with the grouped convolution still have parameter redundancy. In this article, concerning the grouped convolution, we propose a sharing grouped convolution structure to reduce parameters. To efficiently eliminate parameter redundancy and improve model performance, we propose a Bayesian sharing framework to transfer the vanilla grouped convolution to be the sharing structure. Intragroup correlation and intergroup importance are introduced into the prior of the parameters. We handle the Maximum Type II likelihood estimation problem of the intragroup correlation and intergroup importance by a group LASSO-type algorithm. The prior mean of the sharing kernels is iteratively updated. Extensive experiments are conducted to demonstrate that on different grouped convolutional neural networks, the proposed sharing grouped convolution structure with the Bayesian sharing framework can reduce parameters and improve prediction accuracy. The proposed sharing framework can reduce parameters up to 64.17%. For ResNeXt-50 with the sharing grouped convolution on ImageNet dataset, network parameters can be reduced by 96.875% in all grouped convolutional layers, and accuracies are improved to 78.86% and 94.54% for top-1 and top-5, respectively.

High-level synthesis (HLS) tools have gained great attention in recent years because it emancipates engineers from the complicated and heavy hardware description language writing and facilitates the implementations of modern applications (e.g., deep learning models) on Field-programmable Gate Array (FPGA) , by using high-level languages and HLS directives. However, finding good HLS directives is challenging, due to the time-consuming design processes, the balances among different design objectives, and the diverse fidelities (accuracies of data) of the performance values between the consecutive FPGA design stages. To find good HLS directives, a novel automatic optimization algorithm is proposed to explore the Pareto designs of the multiple objectives while making full use of the data with different fidelities from different FPGA design stages. Firstly, a non-linear Gaussian process (GP) is proposed to model the relationships among the different FPGA design stages. Secondly, for the first time, the GP model is enhanced as correlated GP (CGP) by considering the correlations between the multiple design objectives, to find better Pareto designs. Furthermore, we extend our model to be a deep version deep CGP (DCGP) by using the deep neural network to improve the kernel functions in Gaussian process models, to improve the characterization capability of the models, and learn better feature representations. We test our design method on some public benchmarks (including general matrix multiplication and sparse matrix-vector multiplication) and deep learning-based object detection model iSmart2 on FPGA. Experimental results show that our methods outperform the baselines significantly and facilitate the deep learning designs on FPGA.

With continued scaling, transistor aging induced by Hot Carrier Injection and Bias Temperature Instability causes a gradual failure of nanometer-scale integrated circuits (ICs). In this paper, to characterize the multi-typed devices and connection ports, a heterogeneous directed multigraph is adopted to efficiently represent analog IC post-layout netlists. We investigate a heterogeneous graph convolutional network (H-GCN) to fast and accurately estimate aging-induced transistor degradation. In the proposed H-GCN, an embedding generation algorithm with a latent space mapping method is developed to aggregate information from the node itself and its multi-typed neighboring nodes through multi-typed edges. Since our proposed H-GCN is independent of dynamic stress conditions, it can replace static aging analysis. We conduct experiments on very advanced 5nm industrial designs. Compared to traditional machine learning and graph learning methods, our proposed H-GCN can achieve more accurate estimations of aging-induced transistor degradation. Compared to an industrial reliability tool, our proposed H-GCN can achieve 24.623x speedup on average.

Placement and routing (PnR) is the most time-consuming part of the physical design flow. Recognizing the routing performance ahead of time can assist designers and design tools to optimize placement results in advance. In this paper, we propose a fully convolutional network model to predict congestion hotspots and then incorporate this prediction model into a placement engine, DREAMPlace, to get a more route-friendly result. The experimental results on ISPD2015 benchmarks show that with the superior accuracy of the prediction model, our proposed approach can achieve up to 9.05% reduction in congestion rate and 5.30% reduction in routed wirelength compared with the state-of-the-art.

Recent years have witnessed the great successes of deep neural network (DNN) models while deploying DNN models on hardware platforms is still challenging and widely discussed. Some works proposed dedicatedly designed accelerators for some specific DNN models, while some others proposed general-purpose deployment frameworks that can optimize the hardware configurations on various hardware platforms automatically. However, the extremely large design space and the very time-consuming on-chip tests bring great challenges to the hardware configuration optimization process. In this paper, to optimize the hardware deployment, we propose an advanced active learning framework which is composed of batch transductive experiment design (BTED) and Bootstrap-guided adaptive optimization (BAO). The BTED method generates a diverse initial configuration set filled with representative configurations. Based on the Bootstrap method and adaptive sampling, the BAO method guides the selection of hardware configurations during the searching process. To the best of our knowledge, these two methods are both introduced into general DNN deployment frameworks for the first time. We embed our advanced framework into AutoTVM, and the experimental results show that our methods reduce the model inference latency by up to 28.08% and decrease the variance of inference latency by up to 92.74%.

High-level synthesis (HLS) tools have gained great attention in recent years because it emancipates engineers from the complicated and heavy hardware description language writing, by using high-level languages and HLS directives. However, previous works seem powerless, due to the time-consuming design processes, the contradictions among design objectives, and the accuracy difference between the three stages (fidelities). To find good HLS directives, in this paper, a novel correlated multi-objective non-linear optimization algorithm is proposed to explore the Pareto solutions while making full use of data from different fidelities. A non-linear Gaussian process is proposed to model relationships among the analysis reports from different fidelities for the same objective. For the first time, correlated multivariate Gaussian process models are introduced into this domain to characterize the complex relationships of multiple objectives in each design fidelity. A tree-based method is proposed to erase invalid solutions and obviously non-optimal solutions. Experimental results show that our non-linear and pioneering correlated models can approximate the Pareto-frontier of the directive design space in a shorter time with much better performance and good stability, compared with the state-of-the-art.

The microarchitecture design of a processor has been increasingly difficult due to the large design space and time-consuming verification flow. Previously, researchers rely on prior knowledge and cycle-accurate simulators to analyze the performance of different microarchitecture designs but lack sufficient discussions on methodologies to strike a good balance between power and performance. This work proposes an automatic framework to explore microarchitecture designs of the RISC-V Berkeley Out-of-Order Machine (BOOM), termed as BOOM-Explorer, achieving a good trade-off on power and performance. Firstly, the framework utilizes an advanced microarchitecture-aware active learning (MicroAL) algorithm to generate a diverse and representative initial design set. Secondly, a Gaussian process model with deep kernel learning functions (DKL-GP) is built to characterize the design space. Thirdly, correlated multi-objective Bayesian optimization is leveraged to explore Pareto-optimal designs. Experimental results show that BOOM-Explorer can search for designs that dominate previous arts and designs developed by senior engineers in terms of power and performance within a much shorter time.

Performance optimization is the art of continuously seeking an effective mapping between algorithm and hardware. Existing deep learning compilers or frameworks optimize the computation graph via adapting transformations manually designed by expert efforts. We argue that these methods ignore some possible graph-level optimizations, thus it is difficult to generalize to emerging deep learning models or new operators. In this work, we propose AutoGTCO, a tensor program generation system for vision tasks with the transformer architecture on GPU. Compared with existing fusion strategies, AutoGTCO explores the optimization of operator fusion in the transformer model through a novel dynamic programming algorithm. Specifically, to construct an effective search space of the sampled programs, new sketch generation rules and a search policy are proposed for the batch matrix multiplication and softmax operators in each subgraph, which are capable of fusing them into large computation units, it can then map and transform them into efficient CUDA kernels. Overall, our evaluation on three real-world transformer-based vision tasks shows that AutoGTCO improves the execution performance relative to deep learning engine TensorRT by up to 1.38 ×.

Recent years have witnessed impressive progress in super-resolution (SR) processing. However, its real-time inference requirement sets a challenge not only for the model design but also for the on-chip implementation. In this paper, we implement a full-stack SR acceleration framework on embedded GPU devices. The special dictionary learning algorithm used in SR models was analyzed in detail and accelerated via a novel dictionary selective strategy. Besides, the hardware programming architecture together with the model structure is analyzed to guide the optimal design of computation kernels to minimize the inference latency under the resource constraints. With these novel techniques, the communication and computation bottlenecks in the deep dictionary learning-based SR models are tackled perfectly. The experiments on the edge embedded NVIDIA NX and 2080Ti show that our method outperforms the state-of-the-art NVIDIA TensorRT significantly and can achieve real-time performance.

Deep neural networks (DNNs) have been widely used recently while their hardware deployment optimizations are very time-consuming and the historical deployment knowledge is not utilized efficiently. In this paper, to accelerate the optimization process and find better deployment configurations, we propose a novel transfer learning method based on deep Gaussian processes (DGPs). Firstly, a deep Gaussian process (DGP) model is built on the historical data to learn empirical knowledge. Secondly, to transfer knowledge to a new task, a tuning set is sampled for the new task under the guidance of the DGP model. Then DGP is tuned according to the tuning set via maximum-a-posteriori (MAP) estimation to accommodate for the new task and finally used to guide the deployments of the task. The experiments show that our method achieves the best inference latencies of convolutions while accelerating the optimization process significantly, compared with previous arts.

Multiple patterning lithography has been widely adopted in advanced technology nodes of VLSI manufacturing. As a key step in the design flow, multiple patterning layout decomposition (MPLD) is critical to design closure. Due to the$\mathcal {N} \mathcal {P} $-hardness of the general decomposition problem, various efficient algorithms have been proposed with high-quality solutions. However, with increasingly complicated design flow and peripheral processing steps, developing a high-quality layout decomposer becomes more and more difficult, slowing down further advancement in this field. This article presents$\mathsf {OpenMPL}$(2020), an open-source layout decomposition framework, with well-separated peripheral processing and core solving steps. Besides, previous algorithms or techniques are inspected and several issues are discovered. We then propose corresponding new algorithms to resolve these issues. The experiments demonstrate the effectiveness of our proposed algorithms and the efficiency of$\mathsf {OpenMPL}$.

In this paper, we focus on studying robust deep stereo vision of autonomous driving systems and counteracting adversarial attacks against it. Autonomous system operation requires real-time processing of measurement data which often contain significant uncertainties and noise. Adversarial attacks have been widely studied to simulate these perturbations in recent years. To counteract these attacks in autonomous systems, a novel defense method is proposed in this paper. A stereo-regularizer is proposed to guide the model to learn the implicit relationship between the left and right images of the stereo-vision system. Univariate and multivariate functions are adopted to characterize the relationships between the two input images and the object detection model. The regularizer is then relaxed to its upper bound to improve adversarial robustness. Furthermore, the upper bound is approximated by the remainder of its Taylor expansion to improve the local smoothness of the loss surface. The model parameters are trained via adversarial training with the novel regularization term. Our method exploits basic knowledge from the physical world, i.e., the mutual constraints of the two images in the stereo-based system. As such, outliers can be detected and defended with high accuracy and efficiency. Numerical experiments demonstrate that the proposed method offers superior performance when compared with traditional adversarial training methods in state-of-the-art stereo-based 3D object detection models for autonomous vehicles.

Deep neural networks (DNNs) have been proven to achieve unprecedented success on modern artificial intelligence (AI) tasks, which have also greatly motivated the rapid developments of novel DNN models and hardware accelerators. Many challenges still remain towards the design of power efficient DNN accelerator due to the intrinsically intensive data computation and transmission in DNN algorithms. However, most existing efforts in the domain have taken latency as the sole optimization objective, which may often result in sub-optimality in power consumption. In this paper, we propose a framework to optimize the power efficiency of DNN dataflow on FPGA while maximally minimizing the impact on latency. We first propose power and latency models that are built upon different dataflow configurations. Then a power-driven dataflow formulation is proposed, which enables a hierarchical exploration strategy on the dataflow configurations, leading to efficient power consumption at limited latency loss. Experimental results have demonstrated the effectiveness of our proposed models and exploration strategies, where power improvement has shown up to 31% with latency degradation of no worse than 6.5%.