一种多阶段混合框架用于利用视觉语言模型自动解释多视图工程图
A Multi-Stage Hybrid Framework for Automated Interpretation of Multi-View Engineering Drawings Using Vision Language Model
AI总结：
本文提出一种多阶段混合框架，利用视觉语言模型自动解析多视图工程图，通过布局分割、方向感知检测和语义解析提升工程图解读效率。
摘要：
工程图是制造沟通的基础，是传达设计意图、公差和生产细节的主要媒介。然而，使用手动方法、通用光学字符识别（OCR）系统或传统深度学习方法来解释复杂的多视图图仍然具有挑战性，因为布局、方向和混合的符号-文本内容各不相同。为了解决这些挑战，本文提出了一种三阶段混合框架，用于使用现代检测和视觉语言模型（VLMs）自动解释二维多视图工程图。第一阶段由YOLOv11-det执行布局分割，以定位关键区域，如视图、标题块和注释。第二阶段使用YOLOv11-obb进行面向方向的、细粒度的注释检测，包括测量、GD&T符号和表面粗糙度指标。第三阶段使用两种基于Donut的、无OCR的VLMs进行语义内容解析：Alphabetical VLM从标题块和注释中提取文本和分类信息，而Numerical VLM解释定量数据，如测量、GD&T框架和表面粗糙度。为确保鲁棒性和泛化能力，开发了两个专门的数据集：1000张用于布局检测的图纸和1406张用于注释级训练的图纸。Alphabetical VLM的总体F1得分为0.672，而Numerical VLM达到0.963，分别在文本和定量解释方面表现出色。统一的JSON输出使CAD和制造数据库的无缝集成成为可能，为智能工程图分析提供了可扩展的解决方案。
英文摘要：
Engineering drawings are fundamental to manufacturing communication, serving as the primary medium for conveying design intent, tolerances, and production details. However, interpreting complex multi-view drawings with dense annotations remains challenging using manual methods, generic optical character recognition (OCR) systems, or traditional deep learning approaches, due to varied layouts, orientations, and mixed symbolic-textual content. To address these challenges, this paper proposes a three-stage hybrid framework for the automated interpretation of 2D multi-view engineering drawings using modern detection and vision language models (VLMs). In the first stage, YOLOv11-det performs layout segmentation to localize key regions such as views, title blocks, and notes. The second stage uses YOLOv11-obb for orientation-aware, fine-grained detection of annotations, including measures, GD&T symbols, and surface roughness indicators. The third stage employs two Donut-based, OCR-free VLMs for semantic content parsing: the Alphabetical VLM extracts textual and categorical information from title blocks and notes, while the Numerical VLM interprets quantitative data such as measures, GD&T frames, and surface roughness. Two specialized datasets were developed to ensure robustness and generalization: 1,000 drawings for layout detection and 1,406 for annotation-level training. The Alphabetical VLM achieved an overall F1 score of 0.672, while the Numerical VLM reached 0.963, demonstrating strong performance in textual and quantitative interpretation, respectively. The unified JSON output enables seamless integration with CAD and manufacturing databases, providing a scalable solution for intelligent engineering drawing analysis.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
