This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data-including textual design requirements, CAD images, and engineering drawings-derived from the Formula SAE student competition. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different sources. The benchmark features automatic evaluation metrics and is divided into segments-Rule Comprehension, Rule Compliance, and Rule Extraction-based on tasks that engineers perform when designing according to requirements. We evaluate state-of-the-art models (at the time of writing) like GPT-4o, GPT-4, Claude-Opus, Gemini-1.0, and LLaVA-1.5 against the benchmark, and our study uncovers the existing gaps in MLLMs' abilities to interpret complex engineering documentation. The MLLMs tested, while promising, struggle to reliably retrieve relevant rules from the Formula SAE documentation, face challenges in recognizing technical components in CAD images, and encounter difficulty in analyzing engineering drawings. These findings underscore the need for multimodal models that can better handle the multifaceted questions characteristic of design according to technical documentation. This benchmark sets a foundation for future advancements in AI-supported engineering design processes. DesignQA is publicly available at: https://github.com/anniedoris/design_qa/.
MCERF: Advancing Multimodal LLM Evaluation of Engineering Documentation with Enhanced Retrieval
School of Mechanical, Aerospace, and Manufacturing Engineering, University of Connecticut, Storrs, CT 06269(机械、航空航天与制造工程学院,康涅狄格大学,斯托尔斯,CT 06269);Department of Mechanical Engineering, Massachusetts Institute of Technology, Cambridge, MA 02139, USA(机械工程系,麻省理工学院,剑桥,MA 02139,美国)
Evaluating the Use of LLMs for Documentation to Code Traceability
Data-driven Analysis of Software (DAS) Lab Department of Computer Science \& Software Engineering Concordia University Montréal QC Canada;Concordia University
Evaluating Fine-Tuned LLM Model For Medical Transcription With Small Low-Resource Languages Validated Dataset
School of ICT and Industrial Management(信息通信技术与工业管理学院);Metropolia University of Applied Sciences(Metropolia应用科学大学);Rensselaer Polytechnic Institute(伦斯勒理工学院)
Large language models are vulnerable to incidental information in clinical documentation and reasoning
NYU Langone Health(纽约大学朗格尼健康中心);Johns Hopkins University School of Medicine(约翰霍普金斯大学医学院);Washington University School of Medicine in St. Louis(圣路易斯华盛顿大学医学院);University of Alabama at Birmingham Heersink School of Medicine(阿拉巴马大学伯明翰分校希尔斯金医学院);Perelman School of Medicine at the University of Pennsylvania(宾夕法尼亚大学佩雷尔曼医学院);New York University(纽约大学)
Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
University of Wisconsin-Madison(威斯康星大学麦迪逊分校);Loyola University Chicago(芝加哥洛约拉大学);Boston Children’s Hospital Harvard Medical School(波士顿儿童医院哈佛医学院);University of Colorado-Anschutz(科罗拉多大学安舒茨分校)
SkillScriptBench: Benchmarking Self-Evolution of Executable Agent Skill Packages Beyond Markdown
The Hong Kong University of Science and Technology(香港科技大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳));Harbin Institute of Technology, Harbin(哈尔滨工业大学)
From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Paper
School of Informatics and Creative Arts(信息学与创意艺术学院);Dundalk Institute of Technology(邓达克理工学院);The Centre for Research in Engineering Surface Technology (CREST)(工程表面技术研究中心);TU Dublin(都柏林大学技术学院);Zendesk(Zendesk公司);Department of Computing Science and Mathematics(计算科学与数学系);ADAPT Centre(ADAPT中心);National College of Ireland(国家学院);School of Computer Science, University College Dublin (UCD)(计算机科学学院,都柏林大学学院(UCD))
Query2Diagram: Answering Developer Queries with UML Diagrams
HSE University(俄罗斯高等经济学院);St. Petersburg Department of Steklov Mathematical Institute, RAS(圣彼得堡斯捷克洛夫数学研究所(俄罗斯科学院));St. Petersburg State University(圣彼得堡国立大学)
FactEHR: A Dataset for Evaluating Factuality in Clinical Notes Using LLMs
Khoury College of Computer Sciences, Northeastern University(东北大学克劳尔计算机科学学院);Center for Biomedical Informatics Research, Stanford University(斯坦福大学生物医学信息学研究中心);Department of Biomedical Data Science, Stanford School of Medicine(斯坦福医学院生物医学数据科学系);Stanford Health Care(斯坦福健康系统);Department of Medicine, Stanford School of Medicine(斯坦福医学院医学系);Clinical Excellence Research Center, Stanford School of Medicine(斯坦福医学院临床卓越研究中心);Department of Anesthesiology, Perioperative & Pain Medicine, Stanford School of Medicine(斯坦福医学院麻醉学、手术及疼痛医学系);Department of Dermatology, Stanford School of Medicine(斯坦福医学院皮肤病学系);Technology and Digital Solutions, Stanford Health Care(斯坦福健康系统技术与数字解决方案)
Evaluating Large Language Models for Multimodal Simulated Ophthalmic Decision-Making in Diabetic Retinopathy and Glaucoma Screening
Ophthalmology Department, São Paulo Federal University(圣保罗联邦大学眼科部);Université Paris-Saclay, CentraleSupélec, Mathematics and Computer Science Laboratory(巴黎萨克莱大学、中央超导实验室数学与计算机科学实验室);Laboratory for Computational Physiology, Massachusetts Institute of Technology(计算生理学实验室、麻省理工学院)
MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMs
Department of Computer Science and Engineering, The Chinese University of Hong Kong(中国香港中文大学计算机科学与工程系);School of Electronic Science and Engineering, Nanjing University(南京大学电子科学与工程学院);School of Integrated Circuits, Peking University(北京大学集成电路学院);School of Intergrated Circuits, Southeast University(东南大学集成电路学院);School of Computer Science and Engineering, Southeast University(东南大学计算机科学与工程学院);Department of Computer Science and Technology, University of Chinese Academy of Sciences(中国科学院大学计算机科学与技术系);National Center of Technology Innovation for EDA(EDA技术创新国家中心)