arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.07917cs.AIcs.CL
收藏

DesignQA:评估大型语言模型对工程文档理解的多模态基准

DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation

Massachusetts Institute of Technology(麻省理工学院);Autodesk Research(欧特克研究中心);MIT Motorsports(麻省理工学院赛车队)

Anna C. Doris, Daniele Grandi, Ryan Tomich, Md Ferdous Alam, Mohammadmehdi Ataei, Hyunmin Cheong, Faez Ahmed

AI总结:

本文提出DesignQA基准,通过源自Formula SAE的多模态工程文档数据评估GPT-4o等MLLMs在规则理解、合规与提取方面的能力,揭示了现有模型在检索规则、识别CAD组件及分析工程图纸上的不足,为AI辅助工程设计奠定基础。

摘要:

本研究介绍了DesignQA,这是一个旨在评估多模态大型语言模型(MLLMs)在理解和应用技术文档中工程要求方面能力的新型基准。DesignQA聚焦现实世界的工程挑战,独特地结合了源自Formula SAE学生竞赛的多模态数据,包括文本设计要求、CAD图像和工程图纸。与许多现有的MLLM基准不同,DesignQA包含基于文档的视觉问答,其中输入图像和输入文档来自不同的来源。该基准具有自动评估指标,并根据工程师按需求进行设计时执行的任务,被划分为规则理解、规则合规和规则提取三个部分。我们针对该基准评估了(撰写本文时的)最先进模型,如GPT-4o、GPT-4、Claude-Opus、Gemini-1.0和LLaVA-1.5,我们的研究揭示了MLLMs在解释复杂工程文档方面的现有差距。受测的MLLMs虽然前景广阔,但难以可靠地从Formula SAE文档中检索相关规则,在识别CAD图像中的技术组件方面面临挑战,并且在分析工程图纸时遇到困难。这些发现强调了对能够更好地处理按技术文档进行设计所特有的多方面问题的多模态模型的需求。该基准为AI支持的工程设计过程的未来进步奠定了基础。DesignQA公开于:https://github.com/anniedoris/design_qa/。

英文摘要:

This research introduces DesignQA, a novel benchmark aimed at evaluating the proficiency of multimodal large language models (MLLMs) in comprehending and applying engineering requirements in technical documentation. Developed with a focus on real-world engineering challenges, DesignQA uniquely combines multimodal data-including textual design requirements, CAD images, and engineering drawings-derived from the Formula SAE student competition. Different from many existing MLLM benchmarks, DesignQA contains document-grounded visual questions where the input image and input document come from different sources. The benchmark features automatic evaluation metrics and is divided into segments-Rule Comprehension, Rule Compliance, and Rule Extraction-based on tasks that engineers perform when designing according to requirements. We evaluate state-of-the-art models (at the time of writing) like GPT-4o, GPT-4, Claude-Opus, Gemini-1.0, and LLaVA-1.5 against the benchmark, and our study uncovers the existing gaps in MLLMs' abilities to interpret complex engineering documentation. The MLLMs tested, while promising, struggle to reliably retrieve relevant rules from the Formula SAE documentation, face challenges in recognizing technical components in CAD images, and encounter difficulty in analyzing engineering drawings. These findings underscore the need for multimodal models that can better handle the multifaceted questions characteristic of design according to technical documentation. This benchmark sets a foundation for future advancements in AI-supported engineering design processes. DesignQA is publicly available at: https://github.com/anniedoris/design_qa/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑