Natural language is not enough: Benchmarking multi-modal generative AI for Verilog generation
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences（中国科学院大学杭州高等研究院）；Shanghai Jiao Tong University（上海交通大学）
Natural language interfaces have exhibited considerable potential in the automation of Verilog generation derived from high-level specifications through the utilization of large language models, garnering significant attention. Nevertheless, this paper elucidates that visual representations contribute essential contextual information critical to design intent for hardware architectures possessing spatial complexity, potentially surpassing the efficacy of natural-language-only inputs. Expanding upon this premise, our paper introduces an open-source benchmark for multi-modal generative models tailored for Verilog synthesis from visual-linguistic inputs, addressing both singular and complex modules. Additionally, we introduce an open-source visual and natural language Verilog query language framework to facilitate efficient and user-friendly multi-modal queries. To evaluate the performance of the proposed multi-modal hardware generative AI in Verilog generation tasks, we compare it with a popular method that relies solely on natural language. Our results demonstrate a significant accuracy improvement in the multi-modal generated Verilog compared to queries based solely on natural language. We hope to reveal a new approach to hardware design in the large-hardware-design-model era, thereby fostering a more diversified and productive approach to hardware design.
RealBench: Benchmarking Verilog Generation Models with Real-World IP Designs
State Key Lab of Processors, Institute of Computing Technology, CAS（处理器国家重点实验室，计算技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；University of Science and Technology of China（中国科学技术大学）；Shanghai Innovation Center for Processor Technologies（上海处理器技术创新中心）；Cambricon Technologies（Cambricon技术公司）
School of Technology Management and Engineering, NMIMS（NMIMS技术管理与工程学院）；Indian Institute of Technology, Roorkee（鲁尔基印度理工学院）；Birla Institute of Technology and Science, Pilani (BITS Pilani)（皮拉尼贝拉理工学院（BITS Pilani））
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
AI Singapore, National University of Singapore（新加坡国立大学新加坡人工智能）；School of Computing, National University of Singapore（新加坡国立大学计算学院）；Institute of Advanced Intelligence and Computing, A*STAR（新加坡科技研究局先进智能与计算研究所）
Multimodal Large Language Models for End-to-End Affective Computing: Benchmarking and Boosting with Generative Knowledge Prompting
School of Computer Science, South China Normal University（华南师范大学计算机科学学院）；School of Information Technology in Education, South China Normal University（华南师范大学教育信息技术学院）
