arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.00716cs.CVcs.LG
收藏

对齐用于表格结构识别的基准数据集

Aligning benchmark datasets for table structure recognition

Microsoft(微软)

Brandon Smock, Rohith Pesala, Robin Abraham

AI总结:

本文通过固定Table Transformer (TATR)模型架构,采用以数据为中心的方法,消除基准数据集间的标注错误与不一致性,使ICDAR-2013上的评估准确率大幅提升,证明了数据集对齐对TSR模型性能和基准设计的重要意义。

摘要:

用于表格结构识别(TSR)的基准数据集必须经过仔细处理,以确保其标注一致。然而,即使某个数据集的标注自身一致,不同数据集之间也可能存在显著的不一致性,这会损害在这些数据集上训练和评估的模型的性能。在这项工作中,我们展示了通过对齐这些基准数据集——即消除它们之间的错误和不一致性——能够显著提升模型性能。我们通过一种以数据为中心的方法对此进行了证明,在该方法中,我们采用了一种模型架构 Table Transformer (TATR),并在整个过程中保持其固定不变。在 PubTables-1M 上训练时,在 ICDAR-2013 基准上评估的 TATR 的基线精确匹配准确率为 65%;在 FinTabNet 上训练时为 42%;合并后为 69%。在减少了标注错误和数据集间的不一致性之后,在 PubTables-1M 上训练时,在 ICDAR-2013 上评估的 TATR 性能大幅提升至 75%;在 FinTabNet 上训练时提升至 65%;合并后提升至 81%。我们通过对各项修改步骤进行消融实验表明,表格标注的规范化对性能有显著的正向影响,而其他选择则在确定基准数据集最终构成时平衡了必要的权衡。总体而言,我们认为这项工作对 TSR 以及潜在其他任务的基准设计具有重要意义。数据集处理和训练代码将发布于 https://github.com/microsoft/table-transformer。

英文摘要:

Benchmark datasets for table structure recognition (TSR) must be carefully processed to ensure they are annotated consistently. However, even if a dataset's annotations are self-consistent, there may be significant inconsistency across datasets, which can harm the performance of models trained and evaluated on them. In this work, we show that aligning these benchmarks$\unicode{x2014}$removing both errors and inconsistency between them$\unicode{x2014}$improves model performance significantly. We demonstrate this through a data-centric approach where we adopt one model architecture, the Table Transformer (TATR), that we hold fixed throughout. Baseline exact match accuracy for TATR evaluated on the ICDAR-2013 benchmark is 65% when trained on PubTables-1M, 42% when trained on FinTabNet, and 69% combined. After reducing annotation mistakes and inter-dataset inconsistency, performance of TATR evaluated on ICDAR-2013 increases substantially to 75% when trained on PubTables-1M, 65% when trained on FinTabNet, and 81% combined. We show through ablations over the modification steps that canonicalization of the table annotations has a significantly positive effect on performance, while other choices balance necessary trade-offs that arise when deciding a benchmark dataset's final composition. Overall we believe our work has significant implications for benchmark design for TSR and potentially other tasks as well. Dataset processing and training code will be released at https://github.com/microsoft/table-transformer.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑