arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.18652cs.CVcs.AIcs.IR
收藏

超越字符串匹配:PDF表格提取的语义评估

Beyond String Matching: Semantic Evaluation of PDF Table Extraction

Institute for Machine Learning and Analytics (IMLA)(机器学习与分析研究所);Offenburg University(奥芬堡大学);University of Mannheim(曼海姆大学)

Pius Horn, Janis Keuper

AI总结:

提出基于LLM-as-a-judge的语义评估框架,通过合成PDF和人工验证,显著优于现有规则指标(TEDS、GriTS),并评估了21种PDF解析器。

摘要:

从PDF中可靠地提取表格对于大规模科学数据挖掘和知识库构建至关重要,然而现有的评估方法依赖于基于规则的指标,无法捕捉表格内容的语义等价性。我们提出了一个基于合成PDF的基准测试框架,这些PDF具有精确的LaTeX真实标注,并使用来自arXiv的表格以确保现实的复杂性和多样性。作为我们的核心方法论贡献,我们将LLM-as-a-judge应用于语义表格评估,并将其集成到一个能够适应解析器输出不一致性的匹配流水线中。通过一项包含超过1500个提取表格对的人工验证研究,我们表明基于LLM的评估与人类判断的相关性(Pearson r=0.93)显著高于当前使用的基于树编辑距离的相似度(TEDS, r=0.68)和网格表格相似度(GriTS, r=0.70)。对21个当代PDF解析器在包含451个表格的100个合成文档上的评估揭示了显著的性能差异。我们的结果为选择用于表格数据提取的解析器提供了实用指导,并为这一关键任务建立了一种可重复、可扩展的评估方法。代码和数据:https://github.com/phorn1/pdf-parse-bench 指标研究和人工评估:https://github.com/phorn1/table-metric-study

英文摘要:

Reliably extracting tables from PDFs is essential for large-scale scientific data mining and knowledge base construction, yet existing evaluation approaches rely on rule-based metrics that fail to capture semantic equivalence of table content. We present a benchmarking framework based on synthetically generated PDFs with precise LaTeX ground truth, using tables sourced from arXiv to ensure realistic complexity and diversity. As our central methodological contribution, we apply LLM-as-a-judge for semantic table evaluation, integrated into a matching pipeline that accommodates inconsistencies in parser outputs. Through a human validation study comprising over 1,500 quality judgments on extracted table pairs, we show that LLM-based evaluation achieves substantially higher correlation with human judgment (Pearson r=0.93) compared to currently used Tree Edit Distance-based Similarity (TEDS, r=0.68) and Grid Table Similarity (GriTS, r=0.70). Evaluating 21 contemporary PDF parsers across 100 synthetic documents containing 451 tables reveals significant performance disparities. Our results offer practical guidance for selecting parsers for tabular data extraction and establish a reproducible, scalable evaluation methodology for this critical task. Code and data: https://github.com/phorn1/pdf-parse-bench Metric study and human evaluation: https://github.com/phorn1/table-metric-study

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑