多语言大海捞针：探究多语言大语言模型的长上下文行为
Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models
AI总结：
该研究提出MLNeedle测试评估多语言LLM的长上下文信息检索能力，发现模型性能受语言类型、信息位置影响显著，且长上下文下跨语言检索表现不佳，为相关评估提供了参考。
摘要：
尽管近期的大语言模型（LLM）在回应多种语言的查询方面展现出卓越能力，但其处理多语言长上下文的能力仍未被探索。因此，系统评估LLM在多语言场景下的长上下文能力至关重要，尤其是在信息检索语境中。为填补这一空白，我们提出了多语言大海捞针（MLNeedle）测试，旨在评估模型从多语言干扰文本集合（干草堆）中检索相关信息（针）的能力。该测试是多语言问答任务的延伸，涵盖单语和跨语言检索两种场景。我们在MLNeedle上评估了四款最先进的LLM。研究结果表明，模型性能会随语言和“针”的位置发生显著变化。具体而言，我们观察到当“针”满足以下两种情况时模型性能最低：（i）使用英语语系以外的语言，以及（ii）位于输入上下文的中间位置。此外，尽管部分模型声称上下文长度达到8k token或更长，但随着上下文长度增加，没有一款模型能展现出令人满意的跨语言检索性能。我们的分析为理解多语言场景下LLM的长上下文行为提供了关键见解，可用于指导未来的评估协议制定。据我们所知，这是首项探究LLM多语言长上下文行为的研究。
英文摘要：
While recent large language models (LLMs) demonstrate remarkable abilities in responding to queries in diverse languages, their ability to handle long multilingual contexts is unexplored. As such, a systematic evaluation of the long-context capabilities of LLMs in multilingual settings is crucial, specifically in the context of information retrieval. To address this gap, we introduce the MultiLingual Needle-in-a-Haystack (MLNeedle) test, designed to assess a model's ability to retrieve relevant information (the needle) from a collection of multilingual distractor texts (the haystack). This test serves as an extension of the multilingual question-answering task, encompassing both monolingual and cross-lingual retrieval. We evaluate four state-of-the-art LLMs on MLNeedle. Our findings reveal that model performance can vary significantly with language and needle position. Specifically, we observe that model performance is the lowest when the needle is (i) in a language outside the English language family and (ii) located in the middle of the input context. Furthermore, although some models claim a context size of $8k$ tokens or greater, none demonstrate satisfactory cross-lingual retrieval performance as the context length increases. Our analysis provides key insights into the long-context behavior of LLMs in multilingual settings to guide future evaluation protocols. To our knowledge, this is the first study to investigate the multilingual long-context behavior of LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
