SCORE：生成式文档解析的语义评估框架
SCORE: A Semantic Evaluation Framework for Generative Document Parsing
AI总结：
SCORE提出一个解释无关的语义评估框架，通过编辑距离、令牌诊断、表格空间容差和层级一致性检查，纠正传统指标对生成式文档解析中有效多样性的误罚，实现公平且可解释的基准测试。
摘要：
多模态生成式文档解析系统对传统评估提出了挑战：与确定性的OCR或布局模型不同，它们通常产生语义正确但结构多样的输出。传统指标——CER、WER、IoU或TEDS——将这种多样性误分类为错误，惩罚了有效的解释并掩盖了系统行为。我们引入SCORE（结构和内容鲁棒评估），一个解释无关的框架，整合了(i)调整后的编辑距离以评估鲁棒的内容保真度，(ii)令牌级诊断以区分幻觉与遗漏，(iii)具有空间容差和语义对齐的表格评估，以及(iv)层级感知的一致性检查。这些维度共同使得评估既能包容表示多样性，又能强制执行语义严谨性。在涵盖一个整体基准和一个领域数据集的1,114页中，SCORE一致地揭示了标准指标遗漏的跨数据集性能模式。在2-5%具有模糊表格结构的页面中，传统指标平均惩罚系统12-25%，导致排名扭曲。SCORE纠正了这些情况，恢复了替代但有效解释之间的等价性。此外，通过将生成输出规范化为格式无关的表示，SCORE无需目标检测管道即可复现传统分数（例如表格F1高达0.93），证明仅生成式解析就足以进行全面评估。通过揭示解释多样性如何影响评估结果并提供多维、可解释的诊断，SCORE为现代文档解析系统的语义基础、公平和实用基准测试确立了基本原则。
英文摘要：
Multi-modal generative document parsing systems challenge traditional evaluation: unlike deterministic OCR or layout models, they often produce semantically correct yet structurally divergent outputs. Conventional metrics-CER, WER, IoU, or TEDS-misclassify such diversity as error, penalizing valid interpretations and obscuring system behavior. We introduce SCORE (Structural and COntent Robust Evaluation), an interpretation-agnostic framework that integrates (i) adjusted edit distance for robust content fidelity, (ii) token-level diagnostics to distinguish hallucinations from omissions, (iii) table evaluation with spatial tolerance and semantic alignment, and (iv) hierarchy-aware consistency checks. Together, these dimensions enable evaluation that embraces representational diversity while enforcing semantic rigor. Across 1,114 pages spanning a holistic benchmark and a field dataset, SCORE consistently revealed cross-dataset performance patterns missed by standard metrics. In 2-5% of pages with ambiguous table structures, traditional metrics penalized systems by 12-25% on average, leading to distorted rankings. SCORE corrected these cases, recovering equivalence between alternative but valid interpretations. Moreover, by normalizing generative outputs into a format-agnostic representation, SCORE reproduces traditional scores (e.g., table F1 up to 0.93) without requiring object-detection pipelines, demonstrating that generative parsing alone suffices for comprehensive evaluation. By exposing how interpretive diversity impacts evaluation outcomes and providing multi-dimensional, interpretable diagnostics, SCORE establishes foundational principles for semantically grounded, fair, and practical benchmarking of modern document parsing systems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
