MIXAR:将自回归像素基于语言模型扩展到多种语言和文字
MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
AI总结:
MIXAR是首个基于像素的多语言生成模型,通过八种不同语言和文字训练,提升了多语言任务的性能和鲁棒性。
摘要:
基于像素的语言模型正逐渐成为传统基于标记的方法的替代品,有望克服标记化挑战。然而,语言间的感知多样性在像素空间中多语言泛化方面构成了重大障碍。本文介绍了MIXAR,首个基于像素的生成模型,训练于八种不同语言和多种文字。我们通过实验证明MIXAR在判别和生成多语言任务中优于先前的像素模型及对比的标记化模型。此外,我们展示了MIXAR对训练中未见过的语言的鲁棒性。当模型扩展到0.5B参数时,不仅在生成任务如LAMBADA中表现更佳,而且在面对输入扰动如正字法攻击时也更具鲁棒性。
英文摘要:
Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges; however, existing models remain English-centric and are hard to train as generators. This paper introduces MIXAR, the first generative multilingual pixel-based language model, pretrained on eight high-resource languages spanning Latin and CJK scripts and scaled up to 0.5B parameters. Although pretrained on high-resource languages only, MIXAR generalizes to unseen low-resource languages better than a tokenizer-based model pretrained on the same corpus, with its largest gains on the scripts most distant from the pretraining mix, taking a step toward more equitable language technology. To make free generation work in pixel space, MIXAR replaces the notoriously unstable adversarial training stage of prior pixel-based models with a GRPO-based reinforcement-learning stage that rewards the model on its own free-running generations: it proves more stable and multiplies free-generation accuracy by up to 2.3x on multilingual benchmarks. Finally, MIXAR performs on par with comparable tokenizer-based models on discriminative benchmarks, improves consistently with scale, and withstands orthographic attacks more gracefully than tokenizer-based models. Code and models are available at https://github.com/april-tools/mixar.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。