一个边界框抵一个Token：面向文档理解的大语言模型中布局与文本的交错融合
A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding
AI总结：
针对现有文档理解方法融合布局与文本时的长序列、未充分利用LLM自回归特性的问题，提出LayTextLLM框架，将边界框投影为单嵌入与文本交错，在KIE和VQA任务上较此前SOTA分别提升15.2%和10.7%。
摘要：
近期，多项研究表明，将OCR生成的文本与空间布局和大语言模型（LLMs）结合，可高效完成文档理解任务。然而，现有将空间布局与文本融合的方法存在局限，如生成过长文本序列，或未能充分利用LLMs的自回归特性。本研究提出面向文档理解的大语言模型布局与文本交错融合框架（LayTextLLM）。LayTextLLM将每个边界框投影为单个嵌入，并与文本交错排列，在利用LLMs自回归特性的同时，有效避免了长序列问题。LayTextLLM不仅简化了布局与文本数据的交互，还在KIE（关键信息抽取）和VQA（视觉问答）任务上表现出性能提升。综合基准评估显示，相较于此前基于OCR的SOTA（最优水平）LLMs，LayTextLLM在KIE任务上提升15.2%，在VQA任务上提升10.7%。所有资源可访问https://github.com/LayTextLLM/LayTextLLM获取。
英文摘要：
Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in KIE and VQA. Comprehensive benchmark evaluations reveal significant improvements of LayTextLLM, with a 15.2% increase on KIE tasks and 10.7% on VQA tasks compared to previous SOTA OCR-based LLMs. All resources are available at https://github.com/LayTextLLM/LayTextLLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
