ExtractBench: 一种复杂结构提取的基准和评估方法
ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction
AI总结:
ExtractBench提出了一种开源基准和评估框架,用于评估PDF到JSON结构提取的准确性和可靠性,揭示了前沿模型在复杂模式下的性能不足。
摘要:
无结构文档如PDF包含有价值的数据,但下游系统需要可靠标准化的格式。LLM越来越多地被用于自动化提取,准确性至关重要。然而,进展受两个瓶颈限制。首先,没有端到端的基准评估PDF到JSON提取在企业规模的模式广度。其次,没有系统的方法捕捉嵌套提取的语义,其中字段需要不同的正确性概念(标识符的精确匹配,数量的容忍,名称的语义等价),数组需要对齐,遗漏必须区分于幻觉。我们通过ExtractBench,一个开源的PDF到JSON结构提取基准和评估框架来解决这两个缺口。基准将35个PDF文档与JSON模式配对,并在经济上有价值的领域中配对人工标注的黄金标签,产生12,867个可评估的字段,涵盖从几十到几百个字段的模式复杂性。评估框架将模式视为可执行规范:每个字段声明其评分指标。基准评估揭示了前沿模型(GPT-5/5.2,Gemini-3 Flash/Pro,Claude 4.5 Opus/Sonnet)在现实模式上仍然不可靠。性能随着模式广度的增加而急剧下降,最终在所有测试模型上369个字段的财务报告模式中得出0%的有效输出。我们发布ExtractBench在https://github.com/ContextualAI/extract-bench。
英文摘要:
Unstructured documents like PDFs contain valuable structured information, but downstream systems require this data in reliable, standardized formats. LLMs are increasingly deployed to automate this extraction, making accuracy and reliability paramount. However, progress is bottlenecked by two gaps. First, no end-to-end benchmark evaluates PDF-to-JSON extraction under enterprise-scale schema breadth. Second, no principled methodology captures the semantics of nested extraction, where fields demand different notions of correctness (exact match for identifiers, tolerance for quantities, semantic equivalence for names), arrays require alignment, and omission must be distinguished from hallucination. We address both gaps with ExtractBench, an open-source benchmark and evaluation framework for PDF-to-JSON structured extraction. The benchmark pairs 35 PDF documents with JSON Schemas and human-annotated gold labels across economically valuable domains, yielding 12,867 evaluatable fields spanning schema complexities from tens to hundreds of fields. The evaluation framework treats the schema as an executable specification: each field declares its scoring metric. Baseline evaluations reveal that frontier models (GPT-5/5.2, Gemini-3 Flash/Pro, Claude 4.5 Opus/Sonnet) remain unreliable on realistic schemas. Performance degrades sharply with schema breadth, culminating in 0% valid output on a 369-field financial reporting schema across all tested models. We release ExtractBench at https://github.com/ContextualAI/extract-bench.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。