RomanLens: 潜在罗曼化在LLMs多语言性中的作用
RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs
AI总结：
RomanLens研究了罗曼化在LLMs多语言处理中的潜在作用，发现中间层常以罗曼化形式表示目标词，并通过实验表明罗曼化有助于语言转换。
摘要：
大型语言模型（LLMs）尽管主要训练于以英语为中心的语料库，却表现出强大的多语言性能。这引发了根本性的问题：LLMs如何实现这种多语言能力？聚焦于使用非罗曼文字书写的语言，我们研究了罗曼化——用罗曼字符表示非罗曼文字——作为多语言处理潜在桥梁的作用。通过机制可解释性技术，我们分析了下一个token生成，并发现中间层经常以罗曼化形式表示目标词，然后过渡到本族文字，这种现象我们称为潜在罗曼化。进一步，通过激活补丁实验，我们证明LLMs在本族文字和罗曼化文字上编码语义概念的方式相似，表明存在共享的底层表示。此外，对于翻译成非罗曼文字语言的情况，我们的发现表明，当目标语言以罗曼化形式存在时，其表示在模型的层中出现得更早，相较于本族文字。这些见解有助于更深入理解LLMs中的多语言表示，并突显了罗曼化在促进语言转换中的隐含作用。
英文摘要：
Large Language Models (LLMs) exhibit strong multilingual performance despite being predominantly trained on English-centric corpora. This raises a fundamental question: How do LLMs achieve such multilingual capabilities? Focusing on languages written in non-Roman scripts, we investigate the role of Romanization - the representation of non-Roman scripts using Roman characters - as a potential bridge in multilingual processing. Using mechanistic interpretability techniques, we analyze next-token generation and find that intermediate layers frequently represent target words in Romanized form before transitioning to native script, a phenomenon we term Latent Romanization. Further, through activation patching experiments, we demonstrate that LLMs encode semantic concepts similarly across native and Romanized scripts, suggesting a shared underlying representation. Additionally, for translation into non-Roman script languages, our findings reveal that when the target language is in Romanized form, its representations emerge earlier in the model's layers compared to native script. These insights contribute to a deeper understanding of multilingual representation in LLMs and highlight the implicit role of Romanization in facilitating language transfer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
