arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.04819cs.AI
收藏

AECV-Bench:在建筑和工程图纸理解上的多模态模型评估基准

AECV-Bench: Benchmarking Multimodal Models on Architectural and Engineering Drawings Understanding

AECFoundry;University of New South Wales(新南威尔士大学)

Aleksei Kondratenko, Mussie Birhane, Houssame E. Hsain, Guido Maciocci

AI总结:

AECV-Bench评估多模态模型在建筑图纸理解上的能力,发现OCR和文本问答表现最佳,但符号理解尤其是门窗计数仍存在较大误差,需改进领域特定表示和人机协作流程。

摘要:

建筑和工程图纸通过符号、布局规范和密集标注来编码几何和语义,但现代多模态和视觉-语言模型是否能够可靠地解读这种图形语言仍不清楚。我们提出了AECV-Bench,这是一个用于评估多模态和视觉-语言模型在真实AEC制品上的基准,通过两个互补的应用场景:(i) 对120份高质量平面图进行物体计数(门、窗、卧室、卫生间等),以及(ii) 覆盖192个问题-答案对的图纸基础文档问答,测试文本提取(OCR)、实例计数、空间推理和常见图纸区域的比较推理。物体计数性能报告使用每字段精确匹配准确率和MAPE结果,而文档问答性能报告使用总体准确率和按类别分解,结合LLM作为裁判的评分流程和针对边缘情况的人类裁决。在统一协议下评估一系列最先进的模型,我们观察到一个稳定的能力建梯度;OCR和文本中心的文档问答最强(最高可达0.95准确率),空间推理中等,而符号中心的图纸理解——尤其是门和窗的可靠计数——仍处于未解决状态(通常准确率在0.40-0.55之间),存在显著的比例误差。这些结果表明,当前系统在文档助手方面表现良好,但缺乏稳健的图纸素养,推动了领域特定的表示和工具增强、人机协作的工作流,以实现高效的AEC自动化。

英文摘要:

AEC drawings encode geometry and semantics through symbols, layout conventions, and dense annotation, yet it remains unclear whether modern multimodal and vision-language models can reliably interpret this graphical language. We present AECV-Bench, a benchmark for evaluating multimodal and vision-language models on realistic AEC artefacts via two complementary use cases: (i) object counting on 120 high-quality floor plans (doors, windows, bedrooms, toilets), and (ii) drawing-grounded document QA spanning 192 question-answer pairs that test text extraction (OCR), instance counting, spatial reasoning, and comparative reasoning over common drawing regions. Object-counting performance is reported using per-field exact-match accuracy and MAPE results, while document-QA performance is reported using overall accuracy and per-category breakdowns with an LLM-as-a-judge scoring pipeline and targeted human adjudication for edge cases. Evaluating a broad set of state-of-the-art models under a unified protocol, we observe a stable capability gradient; OCR and text-centric document QA are strongest (up to 0.95 accuracy), spatial reasoning is moderate, and symbol-centric drawing understanding - especially reliable counting of doors and windows - remains unsolved (often 0.40-0.55 accuracy) with substantial proportional errors. These results suggest that current systems function well as document assistants but lack robust drawing literacy, motivating domain-specific representations and tool-augmented, human-in-the-loop workflows for an efficient AEC automation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑