OmniCorpus：一个包含100亿级别图像与文本交错的统一多模态语料库
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text
AI总结：
针对现有图像-文本交错数据规模小、多样性不足的问题，研究团队构建了100亿规模的多模态语料库OmniCorpus，规模达同类数据集15倍且来源多样、格式灵活，为多模态大模型研究提供数据支撑。
摘要：
图像-文本交错数据由以自然文档格式排列的多张图像和文本构成，契合互联网数据的呈现范式，也与人类的阅读习惯高度相似。近期研究表明，这类数据有助于多模态上下文学习，且能在多模态微调过程中保留大语言模型的能力。然而，当前图像-文本交错数据的规模和多样性有限，制约了多模态大语言模型的发展。本文提出了OmniCorpus，一个100亿规模的图像-文本交错数据集。我们借助高效的数据引擎，过滤并提取了大规模高质量文档，其中包含86亿张图像和16960亿个文本token。与同类数据集（如MMC4、OBELICS）相比，我们的数据集具备以下特点：1）规模是前者的15倍，同时保持了良好的数据质量；2）来源更加多样，涵盖英文和非英文网站以及以视频为核心的网站；3）灵活性更强，可轻松从图像-文本交错格式降级为纯文本语料库和图像-文本对。通过全面的分析与实验，我们验证了所提出数据集的质量、可用性和有效性。我们希望这能为未来的多模态模型研究提供坚实的数据基础。代码和数据已发布于https://github.com/OpenGVLab/OmniCorpus。
英文摘要：
Image-text interleaved data, consisting of multiple images and texts arranged in a natural document format, aligns with the presentation paradigm of internet data and closely resembles human reading habits. Recent studies have shown that such data aids multimodal in-context learning and maintains the capabilities of large language models during multimodal fine-tuning. However, the limited scale and diversity of current image-text interleaved data restrict the development of multimodal large language models. In this paper, we introduce OmniCorpus, a 10 billion-scale image-text interleaved dataset. Using an efficient data engine, we filter and extract large-scale high-quality documents, which contain 8.6 billion images and 1,696 billion text tokens. Compared to counterparts (e.g., MMC4, OBELICS), our dataset 1) has 15 times larger scales while maintaining good data quality; 2) features more diverse sources, including both English and non-English websites as well as video-centric websites; 3) is more flexible, easily degradable from an image-text interleaved format to pure text corpus and image-text pairs. Through comprehensive analysis and experiments, we validate the quality, usability, and effectiveness of the proposed dataset. We hope this could provide a solid data foundation for future multimodal model research. Code and data are released at https://github.com/OpenGVLab/OmniCorpus.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
