arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.06706cs.CLcs.AI
收藏

探索大型多模态模型在密集文本上的能力

Exploring the Capabilities of Large Multimodal Models on Dense Text

Huazhong University of Science and Technology(华中科技大学)

Shuo Zhang, Biao Yang, Zhang Li, Zhiyin Ma, Yuliang Liu, Xiang Bai

AI总结:

本文构建含17万问答对的 DT-VQA 数据集,系统评估 GPT4V、Gemini 及开源 LMM 的密集文本理解能力,并验证提示工程与微调的效果。

摘要:

尽管大型多模态模型(LMM)在多模态任务中已取得显著进展,但它们在涉及密集文本内容的任务中的能力仍有待充分探索。密集文本承载着重要信息,常见于文档、表格和产品描述中。理解密集文本使我们能够获取更准确的信息,从而帮助做出更好的决策。为进一步探索 LMM 在复杂文本任务中的能力,我们提出了 DT-VQA 数据集,包含 17 万个问答对。在本文中,我们在该数据集上对 GPT4V、Gemini 以及多种开源 LMM 进行了全面评估,揭示了它们的优势与不足。此外,我们评估了面向 LMM 的两种策略的有效性:提示工程和下游微调。我们发现,即使使用自动标注的训练数据集,模型性能也可以获得显著提升。我们希望本研究能够推动 LMM 在密集文本任务中的研究。代码将发布于 https://github.com/Yuliang-Liu/MultimodalOCR。

英文摘要:

While large multi-modal models (LMM) have shown notable progress in multi-modal tasks, their capabilities in tasks involving dense textual content remains to be fully explored. Dense text, which carries important information, is often found in documents, tables, and product descriptions. Understanding dense text enables us to obtain more accurate information, assisting in making better decisions. To further explore the capabilities of LMM in complex text tasks, we propose the DT-VQA dataset, with 170k question-answer pairs. In this paper, we conduct a comprehensive evaluation of GPT4V, Gemini, and various open-source LMMs on our dataset, revealing their strengths and weaknesses. Furthermore, we evaluate the effectiveness of two strategies for LMM: prompt engineering and downstream fine-tuning. We find that even with automatically labeled training datasets, significant improvements in model performance can be achieved. We hope that this research will promote the study of LMM in dense text tasks. Code will be released at https://github.com/Yuliang-Liu/MultimodalOCR.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑