电信领域视觉理解度量：使用VLM进行图像到UML转换的性能指标
Measuring Visual Understanding in Telecom domain: Performance Metrics for Image-to-UML conversion using VLMs
AI总结：
本文针对电信领域3GPP文档图像转PlantUML的评估空白，提出性能指标对比Claude Sonnet和GPT-4V的转换效果，发现VLM能准确捕获节点和消息但难以处理复杂结构，表明需在训练数据中改进复杂组件表示。
摘要：
电信领域的3GPP文档中充斥着包含序列图的图像。视觉语言大模型（VLMs）的进展简化了将此类图像转换为机器可读的PlantUML（puml）格式的过程。然而，对此类转换的评估存在空白——现有工作并未针对各种组件比较puml脚本。在这项工作中，我们提出了性能指标来衡量此类转换的有效性。从3GPP文档中提取的序列图数据集被选作特定领域实际场景的代表。我们将两种VLM——Claude Sonnet和GPT-4V——生成的puml输出与人工创建的真实表示进行了比较。我们使用版本控制工具来捕获差异，并引入标准性能指标来衡量各个组件的准确性：参与者识别、消息流准确性、顺序排序以及分组结构保留。我们展示了所提指标在量化puml脚本各组件转换错误方面的有效性。结果表明，节点、边和消息能被准确捕获。然而，我们观察到VLM在处理诸如注释、框、组等复杂结构时表现不佳。我们的实验和性能指标表明，在微调VLM的训练数据中需要更好地表示这些组件。
英文摘要：
Telecom domain 3GPP documents are replete with images containing sequence diagrams. Advances in Vision-Language Large Models (VLMs) have eased conversion of such images to machine-readable PlantUML (puml) formats. However, there is a gap in evaluation of such conversions - existing works do not compare puml scripts for various components. In this work, we propose performance metrics to measure the effectiveness of such conversions. A dataset of sequence diagrams from 3GPP documents is chosen to be representative of domain-specific actual scenarios. We compare puml outputs from two VLMs - Claude Sonnet and GPT-4V - against manually created ground truth representations. We use version control tools to capture differences and introduce standard performance metrics to measure accuracies along various components: participant identification, message flow accuracy, sequence ordering, and grouping construct preservation. We demonstrate effectiveness of proposed metrics in quantifying conversion errors across various components of puml scripts. The results show that nodes, edges and messages are accurately captured. However, we observe that VLMs do not necessarily perform well on complex structures such as notes, box, groups. Our experiments and performance metrics indicates a need for better representation of these components in training data for fine-tuned VLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
