全局思考，局部分组：使用多语言单词分组游戏评估大语言模型
Think Globally, Group Locally: Evaluating LLMs Using Multi-Lingual Word Grouping Games
AI总结：
本文提出受《纽约时报》Connections启发的GlobalGroup多语言单词分组游戏基准，评估大语言模型在抽象推理中的语言偏差，发现英语模态表现更优且开源与闭源模型存在性能差异。
摘要：
大语言模型（LLMs）可能因语言模态而在推理能力上表现出偏差，即使内容相似，在一种语言上的任务表现也可能优于另一种语言。以往的大多数工作通过依赖策略或知识即可确保成功的推理任务（如常识或数学任务）来评估这种偏差。然而，抽象推理对于日常生活中的推理至关重要，人们在此运用“跳出框框的思考”来识别和使用模式以解决问题，而不依赖公式化方法。相比之下，针对此类任务类型中的语言偏差进行评估的工作较少。在本文中，我们提出了一项受《纽约时报》Connections游戏启发的任务：GlobalGroup，该任务在多种语言中评估模型的抽象推理能力。我们构建了一个包含五种语言背景——英语、西班牙语、中文、印地语和阿拉伯语——的游戏基准，每种语言均提供母语版本和英文翻译版本以进行比较。我们还提出了游戏难度测量方法，以评估模型在难度相近的游戏上的表现，从而实现更受控的比较，这在推理评估中尤为重要。通过实验，我们发现英语模态在此抽象推理任务中总体上带来更好的表现，并且开源与闭源模型之间存在性能差异。
英文摘要：
Large language models (LLMs) can exhibit biases in reasoning capabilities due to linguistic modality, performing better on tasks in one language versus another, even with similar content. Most previous works evaluate this through reasoning tasks where reliance on strategies or knowledge can ensure success, such as in commonsense or math tasks. However, abstract reasoning is vital to reasoning for everyday life, where people apply "out-of-the-box thinking" to identify and use patterns for solutions, without a reliance on formulaic approaches. Comparatively, little work has evaluated linguistic biases in this task type. In this paper, we propose a task inspired by the New York Times Connections: GlobalGroup, that evaluates models in an abstract reasoning task across several languages. We constructed a game benchmark with five linguistic backgrounds -- English, Spanish, Chinese, Hindi, and Arabic -- in both the native language and an English translation for comparison. We also proposed game difficulty measurements to evaluate models on games with similar difficulty, enabling a more controlled comparison, which is particularly important in reasoning evaluations. Through experimentation, we find English modalities largely lead to better performance in this abstract reasoning task, and performance disparities between open- and closed-source models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
