arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.13812cs.DBcs.AIcs.MA
收藏

DTBench:文档到表格提取的合成基准

DTBench: A Synthetic Benchmark for Document-to-Table Extraction

Zhejiang University(浙江大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));The Hong Kong University of Science(香港科学与技术大学)

Yuxiang Guo, Zhuoran Du, Nan Tang, Kezheng Tang, Congcong Ge, Yunjun Gao

AI总结:

提出DTBench合成基准,通过反向Table2Doc范式和多智能体合成流程生成文档,系统评估LLM在文档到表格提取中的多种能力。

摘要:

文档到表格(Doc2Table)提取是在目标模式从非结构化文档中导出结构化表格,实现可靠且可验证的基于SQL的数据分析。尽管大型语言模型(LLM)在灵活信息提取方面显示出潜力,但其生成精确结构化表格的能力仍未被充分理解,特别是对于需要推理和冲突解决等复杂能力的间接提取。现有基准既没有明确区分也没有全面覆盖Doc2Table提取所需的各种能力。我们认为,一个能力感知的基准对于系统评估至关重要。然而,使用人工标注的文档-表格对构建此类基准成本高、难以扩展且能力覆盖有限。为解决此问题,我们采用反向Table2Doc范式,并设计多智能体合成工作流,从真实表格生成文档。基于此方法,我们提出DTBench,一个合成基准,采用提出的Doc2Table能力的两级分类法,涵盖5个主要类别和13个子类别。我们在DTBench上评估了几种主流LLM,展示了模型间的显著性能差距,以及在推理、忠实性和冲突解决方面的持续挑战。DTBench为数据生成和评估提供了全面的测试平台,促进了Doc2Table提取的未来研究。该基准公开于https://github.com/ZJU-DAILY/DTBench。

英文摘要:

Document-to-table (Doc2Table) extraction derives structured tables from unstructured documents under a target schema, enabling reliable and verifiable SQL-based data analytics. Although large language models (LLMs) have shown promise in flexible information extraction, their ability to produce precisely structured tables remains insufficiently understood, particularly for indirect extraction that requires complex capabilities such as reasoning and conflict resolution. Existing benchmarks neither explicitly distinguish nor comprehensively cover the diverse capabilities required in Doc2Table extraction. We argue that a capability-aware benchmark is essential for systematic evaluation. However, constructing such benchmarks using human-annotated document-table pairs is costly, difficult to scale, and limited in capability coverage. To address this, we adopt a reverse Table2Doc paradigm and design a multi-agent synthesis workflow to generate documents from ground-truth tables. Based on this approach, we present DTBench, a synthetic benchmark that adopts a proposed two-level taxonomy of Doc2Table capabilities, covering 5 major categories and 13 subcategories. We evaluate several mainstream LLMs on DTBench, and demonstrate substantial performance gaps across models, as well as persistent challenges in reasoning, faithfulness, and conflict resolution. DTBench provides a comprehensive testbed for data generation and evaluation, facilitating future research on Doc2Table extraction. The benchmark is publicly available at https://github.com/ZJU-DAILY/DTBench.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑