Thank You, Stingray：多语言大语言模型尚无法消歧跨语言词义
Thank You, Stingray: Multilingual Large Language Models Can Not (Yet) Disambiguate Cross-Lingual Word Sense
AI总结：
本研究针对多语言大语言模型跨语言语义评估的空白，推出跨语言词义消歧基准StingrayBench，用四组语言对的“假朋友”词测试模型，发现其偏向高资源语言，并提出相关量化新指标，助力构建更包容的语言模型。
摘要：
多语言大语言模型（LLMs）已备受关注，但人们对其在英语之外语言上的可靠性存在担忧。本研究通过引入全新的跨语言词义消歧基准StingrayBench，填补了跨语言语义评估领域的空白。本文提出，可利用“假朋友”（即两种语言中拼写相似但含义完全不同的词）来精准定位LLMs在跨语言词义消歧上的局限性。我们收集了印尼语-马来语、印尼语-他加禄语、中文-日语、英语-德语四组语言对的假朋友词，并测试LLMs在语境中区分这些词用法的能力。在对多款模型的分析中，我们观察到它们往往偏向资源更丰富的语言。我们还基于该基准提出了量化跨语言词义偏向与理解能力的新指标。本研究有助于开发更多样、更具包容性的语言模型，为更广泛的多语言群体提供更公平的使用渠道。
英文摘要：
Multilingual large language models (LLMs) have gained prominence, but concerns arise regarding their reliability beyond English. This study addresses the gap in cross-lingual semantic evaluation by introducing a novel benchmark for cross-lingual sense disambiguation, StingrayBench. In this paper, we demonstrate using false friends -- words that are orthographically similar but have completely different meanings in two languages -- as a possible approach to pinpoint the limitation of cross-lingual sense disambiguation in LLMs. We collect false friends in four language pairs, namely Indonesian-Malay, Indonesian-Tagalog, Chinese-Japanese, and English-German; and challenge LLMs to distinguish the use of them in context. In our analysis of various models, we observe they tend to be biased toward higher-resource languages. We also propose new metrics for quantifying the cross-lingual sense bias and comprehension based on our benchmark. Our work contributes to developing more diverse and inclusive language modeling, promoting fairer access for the wider multilingual community.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
