arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.00522cs.CL
收藏

像素语言模型的文本渲染策略

Text Rendering Strategies for Pixel Language Models

University of Copenhagen(哥本哈根大学);ROCKWOOL Foundation Research Unit(洛克伍德基金会研究单位);Johns Hopkins University(约翰斯·霍普金斯大学)

Jonas F. Lotz, Elizabeth Salesky, Phillip Rust, Desmond Elliott

AI总结:

本文针对像素语言模型提出字符二元组渲染策略,在句子级任务上提升性能且不损害词元级与多语言任务,并可用22M参数模型达到原86M参数模型的效果。

摘要:

基于像素的语言模型将文本渲染为图像进行处理,这使其能够处理任意文字系统,从而成为开放词汇语言建模的一种有前景的方法。然而,近期方法所使用的文本渲染器会产生大量几乎等价的输入图像块,由于输入表示中的冗余,这可能对下游任务并非最优。在本文中,我们研究了在 PIXEL 模型(Rust 等人,2023)中渲染文本的四种方法,并发现简单的字符二元组(character bigram)渲染在句子级任务上带来了性能提升,同时不损害词元级或多语言任务的性能。这种新的渲染策略还使得训练一个仅有 22M 参数的更紧凑模型成为可能,其性能与原始的 86M 参数模型相当。我们的分析表明,字符二元组渲染能够产生一致更好的模型,但其图像块嵌入空间是各向异性的,这是由图像块频率偏差所驱动的,从而揭示了基于图像块和基于分词的语言模型之间的联系。

英文摘要:

Pixel-based language models process text rendered as images, which allows them to handle any script, making them a promising approach to open vocabulary language modelling. However, recent approaches use text renderers that produce a large set of almost-equivalent input patches, which may prove sub-optimal for downstream tasks, due to redundancy in the input representations. In this paper, we investigate four approaches to rendering text in the PIXEL model (Rust et al., 2023), and find that simple character bigram rendering brings improved performance on sentence-level tasks without compromising performance on token-level or multilingual tasks. This new rendering strategy also makes it possible to train a more compact model with only 22M parameters that performs on par with the original 86M parameter model. Our analyses show that character bigram rendering leads to a consistently better model but with an anisotropic patch embedding space, driven by a patch frequency bias, highlighting the connections between image patch- and tokenization-based language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑