arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.15045cs.CV
收藏

DocLayLLM:面向富文本文档理解的大语言模型高效多模态扩展方案

DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding

South China University of Technology(华南理工大学);Alibaba Cloud Computing(阿里巴巴云计算);SCUT-Zhuhai Institute of Modern Industrial Innovation(华南理工大学珠海现代产业创新研究院)

Wenhui Liao, Jiapeng Wang, Hongliang Li, Chengyu Wang, Jun Huang, Lianwen Jin

AI总结:

针对富文本文档理解中现有多模态大模型算力需求高、多模态融合效果不佳的问题,提出DocLayLLM,通过轻量融合视觉与位置token、创新思维链预训练及退火技术,在轻量训练下性能超越现有依赖OCR和无OCR的同类方法。

摘要:

富文本文档理解(TDU)需要对包含大量文本内容与复杂布局的文档进行综合分析。尽管多模态大语言模型(MLLMs)已在该领域取得快速进展,但现有方法要么需要大量计算资源,要么难以实现有效的多模态融合。本文提出DocLayLLM,这是一款专为TDU设计的高效大语言模型多模态扩展方案。我们通过将视觉块token与2D位置token轻量集成到大语言模型的输入中,并利用大语言模型自身对文档内容进行编码,充分发挥了大语言模型的文档理解能力,同时增强了其对OCR信息的感知能力。我们还深入考量了思维链(CoT)的作用,创新性地提出了思维链预训练与思维链退火技术。DocLayLLM在轻量训练设置下即可取得优异性能,展现出其高效性与有效性。实验结果表明,DocLayLLM的性能优于现有依赖OCR的方法以及无OCR的同类方法。代码与模型可在https://github.com/whlscut/DocLayLLM获取。

英文摘要:

Text-rich document understanding (TDU) requires comprehensive analysis of documents containing substantial textual content and complex layouts. While Multimodal Large Language Models (MLLMs) have achieved fast progress in this domain, existing approaches either demand significant computational resources or struggle with effective multi-modal integration. In this paper, we introduce DocLayLLM, an efficient multi-modal extension of LLMs specifically designed for TDU. By lightly integrating visual patch tokens and 2D positional tokens into LLMs' input and encoding the document content using the LLMs themselves, we fully take advantage of the document comprehension capability of LLMs and enhance their perception of OCR information. We have also deeply considered the role of chain-of-thought (CoT) and innovatively proposed the techniques of CoT Pre-training and CoT Annealing. Our DocLayLLM can achieve remarkable performances with lightweight training settings, showcasing its efficiency and effectiveness. Experimental results demonstrate that our DocLayLLM outperforms existing OCR-dependent methods and OCR-free competitors. Code and model are available at https://github.com/whlscut/DocLayLLM.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑