arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14073cs.CLcs.CVcs.LG
收藏

通过截图提升语言理解能力

Improving Language Understanding from Screenshots

Princeton University(普林斯顿大学)

Tianyu Gao, Zirui Wang, Adithya Bhaskar, Danqi Chen

AI总结:

本文提出Patch-and-Text Prediction目标,通过同时掩码恢复截图图像块与文本,使仅接收视觉输入的截图语言模型在GLUE任务上接近BERT性能,并验证其在自回归模型中的有效性。

摘要:

一类新兴的语言模型(LMs)能够在单一视觉视图内同时处理文本和图像,有望解锁图表理解和用户界面导航等复杂任务。我们将这类模型称为截图语言模型。尽管它们具有吸引力,但现有的截图语言模型在语言理解任务上仍大幅落后于纯文本模型。为了缩小这一差距,我们采用了一种简化设置,即模型输入为纯文本渲染的截图,并专注于提升截图语言模型的文本能力。我们提出了一种新颖的Patch-and-Text Prediction(PTP)目标,该目标同时掩码并恢复截图的图像块和截图内的文本。我们还对掩码率、图像块大小以及提升训练稳定性的设计进行了广泛的消融研究。我们的预训练模型在仅接收视觉输入的情况下,在8个GLUE任务中的6个上取得了与BERT相当的性能(差距在2%以内),并且相比先前工作提升最高达8%。此外,我们将PTP扩展到训练自回归截图语言模型,并证明了其有效性——我们的模型能够通过利用截图上下文显著降低困惑度。总之,我们希望我们的发现能够启发未来关于开发强大截图语言模型的研究,并将其应用扩展到更广泛的领域。

英文摘要:

An emerging family of language models (LMs), capable of processing both text and images within a single visual view, has the promise to unlock complex tasks such as chart understanding and UI navigation. We refer to these models as screenshot language models. Despite their appeal, existing screenshot LMs substantially lag behind text-only models on language understanding tasks. To close this gap, we adopt a simplified setting where the model inputs are plain-text-rendered screenshots, and we focus on improving the text ability of screenshot LMs. We propose a novel Patch-and-Text Prediction (PTP) objective, which masks and recovers both image patches of screenshots and text within screenshots. We also conduct extensive ablation studies on masking rates and patch sizes, as well as designs for improving training stability. Our pre-trained model, while solely taking visual inputs, achieves comparable performance with BERT on 6 out of 8 GLUE tasks (within 2%) and improves up to 8% over prior work. Additionally, we extend PTP to train autoregressive screenshot LMs and demonstrate its effectiveness--our models can significantly reduce perplexity by utilizing the screenshot context. Together, we hope our findings can inspire future research on developing powerful screenshot LMs and extending their reach to broader applications.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑