arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.20052cs.CL
收藏

理解与缓解LLMs中的语言混淆问题

Understanding and Mitigating Language Confusion in LLMs

Cohere(Cohere公司)

Kelly Marchisio, Wei-Yin Ko, Alexandre Bérard, Théo Dehaze, Sebastian Ruder

AI总结:

本文研究了LLMs无法一致生成用户期望语言的问题,创建了Language Confusion Benchmark评估15种语言,发现基础和英语中心模型更易混淆,并通过少样本提示、多语言SFT和偏好调整部分缓解此问题。

摘要:

我们研究了LLMs的一个令人惊讶的局限性:它们无法一致地以用户期望的语言生成文本。我们创建了Language Confusion Benchmark (LCB)来评估此类失败,涵盖15种类型学上多样化的语言,并使用现有的和新创建的英语及多语言提示词。我们在反映实际用例的单语和跨语言生成任务上评估了多种LLMs,发现Llama Instruct和Mistral模型表现出高度的语言混淆,即使是最强的模型也无法一致地以正确语言响应。我们观察到基础模型和以英语为中心的指令模型更容易出现语言混淆,复杂提示词和高采样温度会加剧这一问题。我们发现,通过少样本提示、多语言SFT和偏好调整可以部分缓解语言混淆。我们发布了我们的语言混淆基准,它作为高效、可扩展的多语言评估的第一层,网址为https://github.com/for-ai/language-confusion。

英文摘要:

We investigate a surprising limitation of LLMs: their inability to consistently generate text in a user's desired language. We create the Language Confusion Benchmark (LCB) to evaluate such failures, covering 15 typologically diverse languages with existing and newly-created English and multilingual prompts. We evaluate a range of LLMs on monolingual and cross-lingual generation reflecting practical use cases, finding that Llama Instruct and Mistral models exhibit high degrees of language confusion and even the strongest models fail to consistently respond in the correct language. We observe that base and English-centric instruct models are more prone to language confusion, which is aggravated by complex prompts and high sampling temperatures. We find that language confusion can be partially mitigated via few-shot prompting, multilingual SFT and preference tuning. We release our language confusion benchmark, which serves as a first layer of efficient, scalable multilingual evaluation at https://github.com/for-ai/language-confusion.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑