RomanSetu：通过罗马化高效解锁大语言模型的多语言能力
RomanSetu: Efficiently unlocking multilingual capabilities of Large Language Models via Romanization
AI总结：
本研究提出RomanSetu方法，以罗马化文本为接口对英语LLM做持续预训练与指令微调，高效解锁其非罗马文字语言能力，性能媲美或优于原生文字方案。
摘要：
本研究解决了将大语言模型（LLMs）扩展到使用非罗马文字的非英语语言的挑战。我们提出了一种利用文本罗马化形式作为LLMs接口的方法，假设其频繁的非正式使用以及与英语共享的token可增强跨语言对齐。我们的方法包括对Llama 2这类英语LLM在非英语、非罗马文字语言的罗马化文本上进行持续预训练，随后在罗马化数据上进行指令微调。结果表明，罗马化文本不仅将token生成率降低了2-4倍，还在各类自然语言理解（NLU）、自然语言生成（NLG）和机器翻译（MT）任务上达到或超过原生文字表示的性能。此外，在罗马化文本上计算的嵌入与英语译文的对齐程度，比原生文字的嵌入更紧密。我们的方法为在自然语言处理（NLP）领域传统上代表性不足的语言利用英语LLMs的能力提供了一个有前景的方向。我们的代码可在https://github.com/AI4Bharat/romansetu获取。
英文摘要：
This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages that use non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involves the continual pretraining of an English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches or outperforms native script representation across various NLU, NLG, and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP. Our code is available on https://github.com/AI4Bharat/romansetu.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
