Large-language models (LLMs) have exhibited great potential to assist chip designs and analysis. Recent research and efforts are mainly focusing on text-based tasks including general QA, debugging, design tool scripting, and so on. However, chip design and implementation workflow usually require a visual understanding of diagrams, flow charts, graphs, schematics, waveforms, etc, which demands the development of multimodality foundation models. In this paper, we propose ChipVQA, a benchmark designed to evaluate the capability of visual language models for chip design. ChipVQA includes 142 carefully designed and collected VQA questions covering five chip design disciplines: Digital Design, Analog Design, Architecture, Physical Design and Semiconductor Manufacturing. Unlike existing VQA benchmarks, ChipVQA questions are carefully designed by chip design experts and require indepth domain knowledge and reasoning to solve. We conduct comprehensive evaluations on both open-source and proprietary multimodal models that are greatly challenged by the benchmark suit. ChipVQA is available at https://github.com/phdyang007/chipvqa.

Despite recent advancements in code generation with large language models (LLMs), generating hardware code such as Verilog remains a significant challenge due to the scarcity of large-scale, high-quality datasets in the hardware domain. Existing approaches, including scraping open-source repositories and relying on manually curated datasets, often suffer from limited diversity, quality, and scalability. To address these limitations, we introduce LLM4Verilog, an exploratory, collaborative initiative aimed at constructing a large-scale, high-quality, open-source Verilog dataset. Our initiative integrates a community-driven data collection pipeline with a two-stage data filtering technique to ensure high dataset quality. The first stage removes duplicates and low-quality samples, resulting in a large-scale dataset called LLM4Verilog-complete. The second stage applies an LLM-driven quality scoring method, VeriScore, to perform fine-grained filtering and produce a high-quality, ready-to-use dataset called LLM4Verilog-filtered. We evaluate the effectiveness of these datasets through fine-tuning three different LLMs on our dataset, achieving 6.6%~11.2% and 5.3%~13.2% improvements in pass@1 scores on VerilogEval-human and VerilogEval-Machine, respectively, compared to models fine-tuned with prior state-of-the-art datasets. Notably, these improvements are achieved without relying on complex fine-tuning or data augmentation techniques, highlighting our dataset’s strong potential as a foundational resource for enhancing LLMs’ Verilog code generation capabilities. For more information about our initiative and resulting dataset, please refer to https://nvlabs.github.io/LLM4HWDesign/.

Despite the significant progress made in code generation with large language models, challenges persist, especially with hardware description languages such as Verilog. This paper first presents an analysis of fine-tuned LLMs on Verilog coding, with synthetic data from prior methods. We identify two main issues: difficulties in handling non-textual representations (Karnaugh maps, state-transition diagrams and waveforms) and significant variability during training with models randomly making ''minor'' mistakes. To address these limitations, we enhance data curation by creating correct-by-construction data targeting non-textual representations. Additionally, we introduce an automated framework that generates error reports from various model checkpoints and injects these errors into open-source code to create targeted code repair data. Our fine-tuned Starcoder2-15B outperforms prior state-of-the-art results by 3.8\%, 10.9\%, 6.6\% for pass@1 on VerilogEval-Machine, VerilogEval-Human, and RTLLM.

Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wojciech Maciej Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, Dragomir Radev. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

Large Language Models (LLMs) show promise in streamlining hardware design, particularly in hardware code generation. However, the development of LLMs for this domain is severely hindered by the scarcity of large-scale, high-quality, and publicly accessible hardware code datasets. This shortage limits the effective fine-tuning of LLMs, impeding their ability to acquire hardware domain knowledge and generate practical designs. To address this challenge, we have organized the first-of-its-kind LLM4HWDesign contest, a community-driven initiative aimed at constructing a large-scale, high-quality dataset for hardware code generation. The contest adopts a two-phase approach, focusing on expanding the scale and quality of an existing hardware code generation dataset, respectively. By harnessing the collective efforts of the hardware design community, the LLM4HWDesign contest seeks to establish a critical resource for advancing LLM-assisted hardware design workflows. The primary goal of this initiative is to deliver a comprehensive dataset compiled from participants' submissions. We hope the released dataset will significantly advance the field of LLM-assisted hardware design and provide substantial benefits to the broader hardware community.