arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.11633cs.CV
收藏

DocGenome:一个用于训练和测试多模态大语言模型的开放大规模科学文档基准

DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models

Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);Shanghai Jiao Tong University(上海交通大学);Fudan University(复旦大学);Zhejiang University(浙江大学)
Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Bin Wang, Chao Xu, Cong… 展开作者

Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wenjie Wu, Hancheng Ye, Shiyang Feng, Bin Wang, Chao Xu, Conghui He, Pinlong Cai, Min Dou, Botian Shi, Sheng Zhou, Yongwei Wang, Bin Wang, Junchi Yan, Fei Wu, Yu Qiao

AI总结:

针对大模型在多页科学文档提取理解及图表公式处理上的不足,研究团队构建了涵盖153个学科50万份文档的多模态结构化基准DocGenome,具备完整、逻辑、多样、准确四大特征,可用于评估和训练大模型。

摘要:

科学文档记录了研究成果与宝贵的人类知识,构成了庞大的高质量数据语料库。因此,利用从这些文档中提取的多模态数据,并评估大模型处理面向科学文档的任务的能力,具有重要意义。尽管已有可观进展,但大模型在多页科学文档提取与理解任务上的表现仍然不佳,其处理图表、公式等文档内部数据格式的能力也仍待充分探索。为解决这些问题,我们提出了DocGenome,这是一个结构化文档基准,通过我们定制的自动标注流水线,对arXiv开放获取社区中153个学科的50万份科学文档进行标注构建而成。DocGenome具备四大核心特征:1)完整性:它是首个对所有模态数据进行结构化的数据集,包含13种布局属性及其LaTeX源代码。2)逻辑性:它提供了每份科学文档内不同实体之间的6种逻辑关系。3)多样性:它覆盖了多种面向文档的任务,包括文档分类、视觉定位、文档布局检测、文档转换、开放式单页问答与多页问答。4)正确性:它经过了专业团队开展的严格质量管控检查。我们开展了大量实验,以证明DocGenome的优势,并客观评估大模型在我们的基准上的性能。

英文摘要:

Scientific documents record research findings and valuable human knowledge, comprising a vast corpus of high-quality data. Leveraging multi-modality data extracted from these documents and assessing large models' abilities to handle scientific document-oriented tasks is therefore meaningful. Despite promising advancements, large models still perform poorly on multi-page scientific document extraction and understanding tasks, and their capacity to process within-document data formats such as charts and equations remains under-explored. To address these issues, we present DocGenome, a structured document benchmark constructed by annotating 500K scientific documents from 153 disciplines in the arXiv open-access community, using our custom auto-labeling pipeline. DocGenome features four key characteristics: 1) Completeness: It is the first dataset to structure data from all modalities including 13 layout attributes along with their LaTeX source codes. 2) Logicality: It provides 6 logical relationships between different entities within each scientific document. 3) Diversity: It covers various document-oriented tasks, including document classification, visual grounding, document layout detection, document transformation, open-ended single-page QA and multi-page QA. 4) Correctness: It undergoes rigorous quality control checks conducted by a specialized team. We conduct extensive experiments to demonstrate the advantages of DocGenome and objectively evaluate the performance of large models on our benchmark.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑