arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.03410cs.CV
收藏

Enginuity:工程图纸视觉语言理解的数据集与基准

Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams

Predii;Oak Ridge National Laboratory(橡树岭国家实验室);Independent Researcher(独立研究员)

Abhishek Kumar, Isha Motiyani, Tilak Kasturi, Ethan Seefried, Prahitha Movva, Tirthankar Ghosal

AI总结:

针对工程图纸领域缺乏公开基准的问题,提出首个开放数据集Enginuity,通过结构化零件表提取和自由形式视觉问答两项任务评估前沿VLM,揭示零件识别与描述保真度之间的系统性差距。

摘要:

工程图纸对视觉语言模型提出了独特的挑战:与自然图像或通用文档不同,它们通过密集的空间布局、领域特定符号以及视觉标注与结构化零件表之间的交叉引用来编码信息。尽管工程图纸在服务、维修和设计工作流中至关重要,但目前尚无公开基准来衡量该领域VLM的能力;现有数据集主要关注流程图、科学图表或商业文档。为填补这一空白,我们引入了Enginuity,这是首个用于评估复杂工程图纸上VLM的开放数据集和基准。我们在美国军用服务和维修手册语料库上定义了两项任务:结构化零件表提取(任务1)和自由形式视觉图问答(VQA)(任务2)用于基准测试。我们在零样本和思维链提示下评估了四种前沿VLM(GPT-5.2 Chat、Claude Opus 4.7、Gemma 4、Qwen3-VL-32B-Instruct)。在任务1上,模型达到了0.61-0.87的Recall@all,但Token F1pen仅为0.03-0.18,暴露了零件识别与描述保真度之间的系统性差距。任务2揭示了所有模型在事实推理上的一致差距。一项支持性分析表明,相对于语义相似性,token重叠指标将技术描述上的模型能力低估了2-6倍,这促使在领域特定评估中进行LLM作为评判者的校准。我们发布了数据集、注释、评估框架以及每个样本的模型输出,以支持对工程内容上VLM能力的可重复研究。

英文摘要:

Engineering diagrams pose a distinct challenge for vision-language models: unlike natural images or general documents, they encode information through dense spatial layouts, domain-specific symbols, and cross-references between visual callouts and structured parts tables. Despite their centrality to service, repair, and design workflows, there is no public benchmark for measuring VLM capabilities in this domain; existing datasets primarily focus on flowcharts, scientific figures, or business documents. To address this gap, we introduce Enginuity, the first open dataset and benchmark for evaluating VLMs on complex engineering diagrams. We define two tasks over a corpus of U.S. military service and repair manuals: structured parts-table extraction (Task 1) and free-form visual diagram question answering (VQA)(Task 2) for benchmarking. We evaluate four frontier VLMs (GPT-5.2 Chat, Claude Opus 4.7, Gemma 4, Qwen3-VL-32B-Instruct) under zero-shot and chain-of-thought prompting. On Task 1, models reach Recall@all of 0.61-0.87 but Token F1pen of only 0.03-0.18, exposing a systematic gap between part identification and description fidelity. Task 2 reveals a consistent factual-reasoning gap across all models. A supporting analysis shows that token-overlap metrics under-report model capability on technical descriptions by 2-6x relative to semantic similarity, motivating LLM-as-judge calibration for domain-specific evaluation. We release the dataset, annotations, evaluation harness, and per-sample model outputs to support a reproducible study of VLM capability on engineering content.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑