文化翻译中迷失：大语言模型在不同文化背景下进行数学推理的能力如何？
Lost in Cultural Translation: Do LLMs Struggle with Math Across Cultural Contexts?
AI总结：
研究发现大语言模型在不同文化背景下处理数学问题时存在性能下降，文化适应影响推理模式，需更多多样化的训练数据以提升全球适用性。
摘要：
我们证明了大语言模型（LLMs）的数学推理具有文化敏感性：在六个文化适应的GSM8K基准测试变体中测试14种模型（来自Anthropic、OpenAI、Google、Meta、DeepSeek、Mistral和Microsoft），发现当数学问题嵌入不熟悉的文化背景时，准确性下降范围从0.3%（Claude 3.5 Sonnet）到5.9%（LLaMA 3.1-8B），即使数学逻辑未变。这些统计显著的性能下降（p < 0.01，通过McNemar检验确认）表明LLM的数学推理并非文化中性。为创建这些变体（针对海地、摩尔多瓦、巴基斯坦、所罗门群岛、索马里和苏里南），我们系统地替换了1,198个GSM8K问题中的文化实体（名称、食物、地点等），同时保持所有数学运算和数值。对18,887个实例的定量错误分析显示，文化适应影响更广泛的推理模式，数学推理错误占失败的54.7%，计算错误占34.5%。有趣的是，文化熟悉度可以提升性能：Mistral Saba在巴基斯坦适应的问题上优于一些大型模型，因为其训练数据暴露于中东和南亚地区。本研究强调需要更多多样化的训练数据以确保LLM在全球范围内表现稳健。
英文摘要：
We demonstrate that large language models' (LLMs) mathematical reasoning is culturally sensitive: testing 14 models from Anthropic, OpenAI, Google, Meta, DeepSeek, Mistral, and Microsoft across six culturally adapted variants of the GSM8K benchmark, we find accuracy drops ranging from 0.3% (Claude 3.5 Sonnet) to 5.9% (LLaMA 3.1-8B) when math problems are embedded in unfamiliar cultural contexts--even when the underlying mathematical logic remains unchanged. These statistically significant performance reductions (p < 0.01, confirmed through McNemar tests) reveal that mathematical reasoning in LLMs is not culturally neutral. To create these variants for Haiti, Moldova, Pakistan, Solomon Islands, Somalia, and Suriname, we systematically replaced cultural entities (names, foods, places, etc.) in 1,198 GSM8K questions while preserving all mathematical operations and numerical values. Our quantitative error analysis of 18,887 instances reveals that cultural adaptation affects broader reasoning patterns, with mathematical reasoning errors comprising 54.7% and calculation errors 34.5% of failures. Interestingly, cultural familiarity can enhance performance: Mistral Saba outperforms some larger models on Pakistan-adapted problems due to Middle Eastern and South Asian training data exposure. This study underscores the need for more diverse training data to ensure robust LLM performance across global contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
