通过像素级回退克服词汇表约束
Overcoming Vocabulary Constraints with Pixel-level Fallback
AI总结：
提出一种无词汇表编码器，通过将文本渲染为像素生成输入嵌入来增强预训练语言模型，显著提升了机器翻译和跨语言迁移性能，无需大量重训即可增强多语能力并降低解码延迟。
摘要：
子词分词需要在计算效率与词汇表覆盖范围之间取得平衡，这通常会导致在训练期间未优先考虑的语言和文字上表现不佳。我们提出用一种无词汇表编码器来增强预训练语言模型，该编码器从渲染为像素的文本生成输入嵌入。通过在以英语为中心的语言模型上进行实验，我们证明该方法显著提升了机器翻译性能，促进了有效的跨语言迁移，并优于基于分词器的方法。此外，我们发现基于像素的表示优于字节级方法和标准词汇表扩展。该方法无需大量重训即可增强单语模型的多语能力，并通过输入压缩降低解码延迟。
英文摘要：
Subword tokenization requires balancing computational efficiency and vocabulary coverage, which often leads to suboptimal performance on languages and scripts not prioritized during training. We propose to augment pretrained language models with a vocabulary-free encoder that generates input embeddings from text rendered as pixels. Through experiments on English-centric language models, we demonstrate that our approach substantially improves machine translation performance and facilitates effective cross-lingual transfer, outperforming tokenizer-based methods. Furthermore, we find that pixel-based representations outperform byte-level approaches and standard vocabulary expansion. Our approach enhances the multilingual capabilities of monolingual language models without extensive retraining and reduces decoding latency via input compression.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
