arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2610.14531cs.CL
收藏

LongHisDoc:面向中文长历史文档理解的综合基准测试集

LongHisDoc: A Comprehensive Benchmark for Chinese Long Historical Document Understanding

School of Electronic and Information Engineering, South China University of Technology(华南理工大学电子与信息工程学院);College of Literature and Journalism, Sichuan University(四川大学文学与新闻学院)

Zeyu Shan, Yongxin Shi, Shihang Wu, Peirong Zhang, Yuanyuan Fang, Zhaopeng Wang, Lianwen Jin

AI总结:

LongHisDoc是首个中文长历史文档理解综合基准,含1012个专家标注问答对,评估显示顶尖LVLM、LLM准确率仅62.9%、69.9%,还评估了模态检索器与RAG性能,可推动相关研究。

摘要:

中文历史文档是极具价值的文化瑰宝,但其复杂性给非专业人士带来了巨大的理解挑战。尽管大语言模型(LLMs)和大视觉语言模型(LVLMs)在现代文档任务中取得了成功,但它们在长文档(全书)历史文档理解方面的能力仍未得到充分探索。为了弥合这一差距,我们推出了首个专为长历史文档理解定制的基准测试集LongHisDoc。LongHisDoc包含1012个由专家标注的问答对,这些问答对源自101份PDF文档,每份文档平均有85.1页、42776.4个token。它涵盖10个类别,具有四种问答类型:单页、跨页、多跳和无法回答。回答这些问题需要整合文本、布局和视觉元素的多模态证据。我们的评估显示,表现最佳的LVLM和LLM分别仅达到62.9%和69.9%的准确率,凸显出巨大的性能差距。此外,我们评估了各种模态检索器和端到端RAG性能,为检索增强生成(RAG)系统提供参考。我们认为LongHisDoc是一个综合基准,可推动未来全书历史文档理解的研究,该基准和评估代码可在指定的两个URL获取。

英文摘要:

Chinese historical documents are invaluable cultural treasures, but their complexity poses significant comprehension challenges for non-specialists. Despite the success of Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) in modern document tasks, their capabilities in Long (full-book) Historical Document understanding remain largely unexplored. To bridge this gap, we introduce LongHisDoc, the first benchmark tailored for long historical document understanding. LongHisDoc comprises 1,012 expert-annotated QA pairs derived from 101 PDF documents, with an average of 85.1 pages and 42,776.4 tokens per document. It spans 10 categories and features four QA types: single-page, cross-page, multi-hop, and unanswerable. Answering these questions requires integrating multimodal evidence from text, layout, and visual elements. Our evaluation reveals that the top-performing LVLM and LLM achieve only 62.9% and 69.9% accuracy, respectively, highlighting a substantial performance gap. Furthermore, we assess various modal retrievers and the end-to-end RAG performance to provide references for Retrieval-Augmented Generation (RAG) systems. We believe that LongHisDoc serves as a comprehensive benchmark to catalyze future research in full-book historical document understanding. The benchmark and evaluation code are available at https://huggingface.co/datasets/qweq12433454/LongHisDoc and https://github.com/SCUT-DLVCLab/LongHisDoc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑