多项语言还是多群体？多语言大语言模型对价值导向的选择题的回答
Polyglots or Multitudes? Multilingual LLM Answers to Value-laden Multiple-Choice Questions
AI总结：
研究多语言LLM在价值导向选择题中的回答一致性，发现语言影响回答，需进一步探讨偏好微调的作用。
摘要：
选择题（MCQ）常用于评估知识、推理能力，甚至大型语言模型（LLM）中编码的价值。尽管多语言性对LLM事实回忆的影响已被研究，本文旨在探讨一个较少研究的问题：语言对价值导向选择题回答的差异。多语言LLM在不同语言下的回答是否一致，即像理论上的多语言者一样表现，还是根据问题语言的不同而回答，像多个单语言模型通过一个模型表达不同的价值？我们发布了一个新的语料库，即多语言欧洲价值观调查（MEVS），与先前依赖机器翻译或临时提示的工作不同，该语料库仅包含在8种欧洲语言中对齐的人类翻译调查问题。我们向超过三十种多语言LLM（各种大小、制造商和对齐微调状态）施加了部分问题，包括答案顺序、符号类型和尾字符的全面、受控提示变化。我们的结果表明，尽管较大、受指令微调的模型在整体一致性上更高，但其响应的稳健性在不同问题中变化很大，某些选择题在模型内和跨模型之间引发完全一致，而其他问题则使LLM回答分裂。语言特定的行为似乎在所有一致的、受指令微调的模型中出现，但只在某些问题上出现，这要求进一步研究偏好微调的选择效应。
英文摘要：
Multiple-Choice Questions (MCQs) are often used to assess knowledge, reasoning abilities, and even values encoded in large language models (LLMs). While the effect of multilingualism has been studied on LLM factual recall, this paper seeks to investigate the less explored question of language-induced variation in value-laden MCQ responses. Are multilingual LLMs consistent in their responses across languages, i.e. behave like theoretical polyglots, or do they answer value-laden MCQs depending on the language of the question, like a multitude of monolingual models expressing different values through a single model? We release a new corpus, the Multilingual European Value Survey (MEVS), which, unlike prior work relying on machine translation or ad hoc prompts, solely comprises human-translated survey questions aligned in 8 European languages. We administer a subset of those questions to over thirty multilingual LLMs of various sizes, manufacturers and alignment-fine-tuning status under comprehensive, controlled prompt variations including answer order, symbol type, and tail character. Our results show that while larger, instruction-tuned models display higher overall consistency, the robustness of their responses varies greatly across questions, with certain MCQs eliciting total agreement within and across models while others leave LLM answers split. Language-specific behavior seems to arise in all consistent, instruction-fine-tuned models, but only on certain questions, warranting a further study of the selective effect of preference fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
