DocSplit: 一个全面的基准数据集和评估方法用于文档包识别与分割
DocSplit: A Comprehensive Benchmark Dataset and Evaluation Approach for Document Packet Recognition and Splitting
AI总结:
DocSplit提出首个全面的文档包识别与分割基准数据集及评估方法,涵盖多类型文档和复杂场景,揭示当前模型在处理复杂文档分割任务中的性能差距。
摘要:
现实世界中的文档理解往往需要处理异构的、多页文档包,其中包含多个拼接在一起的文档。尽管在视觉文档理解方面取得了最近的进步,但文档包分割这一基本任务,即将文档包分离成单独的单元,仍然几乎没有得到解决。我们提出了第一个全面的基准数据集DocSplit,以及用于评估大型语言模型文档包分割能力的新型评估指标。DocSplit包含五个复杂程度不同的数据集,涵盖多种文档类型、布局和多模态设置。我们正式定义了DocSplit任务,要求模型识别文档边界、分类文档类型,并在文档包内保持正确的页面顺序。该基准解决了现实中的挑战,包括页面顺序混乱、交错文档以及缺乏明显分界线的文档。我们进行了广泛的实验,评估了多模态LLM在我们数据集上的表现,揭示了当前模型在处理复杂文档分割任务时存在显著的性能差距。DocSplit基准数据集和提出的新型评估指标提供了一个系统框架,用于推进文档理解能力,这对法律、金融、医疗等文档密集型领域至关重要。我们发布了这些数据集,以促进未来文档包处理的研究。
英文摘要:
Document understanding in real-world applications often requires processing heterogeneous, multi-page document packets containing multiple documents stitched together. Despite recent advances in visual document understanding, the fundamental task of document packet splitting, which involves separating a document packet into individual units, remains largely unaddressed. We present the first comprehensive benchmark dataset, DocSplit, along with novel evaluation metrics for assessing the document packet splitting capabilities of large language models. DocSplit comprises five datasets of varying complexity, covering diverse document types, layouts, and multimodal settings. We formalize the DocSplit task, which requires models to identify document boundaries, classify document types, and maintain correct page ordering within a document packet. The benchmark addresses real-world challenges, including out-of-order pages, interleaved documents, and documents lacking clear demarcations. We conduct extensive experiments evaluating multimodal LLMs on our datasets, revealing significant performance gaps in current models' ability to handle complex document splitting tasks. The DocSplit benchmark datasets and proposed novel evaluation metrics provide a systematic framework for advancing document understanding capabilities essential for legal, financial, healthcare, and other document-intensive domains. We release the datasets to facilitate future research in document packet processing.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。