arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.01523cs.CVcs.CL
收藏

MMLongBench-Doc:通过可视化对长上下文文档理解进行基准测试

MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations

S-Lab, Nanyang Technological University(南洋理工大学S-Lab);Peking University(北京大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);National University of Singapore(新加坡国立大学);Wuhan University(武汉大学);Singapore Management University(新加坡管理大学);Shanghai AI Laboratory(上海人工智能实验室);University of Arizona(亚利桑那大学);Fudan University(复旦大学)
Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi … 展开作者

Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin Cao, Aixin Sun

AI总结:

本文提出长上下文多模态文档理解基准MMLongBench-Doc,包含1,062个基于130份长PDF的专家标注问题,评估14个LVLMs发现长上下文DU极具挑战,最佳模型GPT-4o仅获42.7% F1分数,凸显了研发更强大长上下文LVLMs的必要性。

摘要:

理解具有丰富布局和多模态组件的文档是一项长期且实用的任务。近期的大型视觉语言模型(LVLMs)在各种任务中取得了显著进展,特别是在单页文档理解(DU)方面。然而,它们在长上下文DU上的能力仍是一个未解决的问题。本工作提出了MMLongBench-Doc,这是一个长上下文、多模态基准,包含1,062个专家标注的问题。与以往的数据集不同,它基于130份冗长的PDF格式文档构建,平均页数为49.4页,文本词元数为20,971。为了进行全面评估,这些问题的答案依赖于来自(1)不同来源(文本、图像、图表、表格和布局结构)和(2)不同位置(即页码)的证据片段。此外,33.2%的问题为跨页问题,需要跨多页的证据。22.8%的问题被设计为无法回答,以检测潜在的幻觉。在14个LVLMs上的实验表明,长上下文DU对当前模型构成了巨大挑战。值得注意的是,表现最佳的模型GPT-4o仅获得42.7%的F1分数,而第二好的GPT-4V得分为31.4%。此外,12个LVLMs(除GPT-4o和GPT-4V外的所有模型)的表现甚至不如其输入了有损解析OCR文档的LLM对应模型。这些结果验证了未来研究更强大长上下文LVLMs的必要性。项目页面:https://mayubo2333.github.io/MMLongBench-Doc

英文摘要:

Understanding documents with rich layouts and multi-modal components is a long-standing and practical task. Recent Large Vision-Language Models (LVLMs) have made remarkable strides in various tasks, particularly in single-page document understanding (DU). However, their abilities on long-context DU remain an open problem. This work presents MMLongBench-Doc, a long-context, multi-modal benchmark comprising 1,062 expert-annotated questions. Distinct from previous datasets, it is constructed upon 130 lengthy PDF-formatted documents with an average of 49.4 pages and 20,971 textual tokens. Towards comprehensive evaluation, answers to these questions rely on pieces of evidence from (1) different sources (text, image, chart, table, and layout structure) and (2) various locations (i.e. page number). Moreover, 33.2% of the questions are cross-page questions requiring evidence across multiple pages. 22.8% of the questions are designed to be unanswerable for detecting potential hallucinations. Experiments on 14 LVLMs demonstrate that long-context DU greatly challenges current models. Notably, the best-performing model, GPT-4o, achieves an F1 score of only 42.7%, while the second-best, GPT-4V, scores 31.4%. Furthermore, 12 LVLMs (all except GPT-4o and GPT-4V) even present worse performance than their LLM counterparts which are fed with lossy-parsed OCR documents. These results validate the necessity of future research toward more capable long-context LVLMs. Project Page: https://mayubo2333.github.io/MMLongBench-Doc

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑