Nunchi-Bench：以韩国迷信为重点的语言模型文化推理基准测试
Nunchi-Bench: Benchmarking Language Models on Cultural Reasoning with a Focus on Korean Superstition
AI总结：
本文提出Nunchi-Bench基准，通过247个韩国迷信相关问题评估多语言大模型的文化推理能力，发现模型虽能识别事实但难以实际应用，且明确文化框架比仅靠语言提示更有效。
摘要：
随着大语言模型（LLMs）成为各个领域的关键顾问，它们的文化敏感性和推理能力在多元文化环境中至关重要。我们引入了Nunchi-Bench，这是一个旨在评估LLMs文化理解能力的基准测试，重点关注韩国迷信。该基准测试包含247个问题，涵盖31个主题，评估事实知识、符合文化习惯的建议以及情境解读。我们在韩语和英语两种语言下评估多语言LLMs，以分析它们对韩国文化语境进行推理的能力，以及语言变化如何影响性能。为了系统地评估文化推理，我们提出了一种新颖的评估策略，采用定制化评分指标，捕捉模型识别文化细微差别并做出恰当回应的程度。我们的研究结果凸显了LLMs在文化推理方面面临的重大挑战。虽然模型通常能识别事实信息，但它们在将事实信息应用于实际场景时存在困难。此外，明确的文化框架比仅仅依赖提示语言更能有效提升性能。为了支持进一步的研究，我们公开发布了Nunchi-Bench以及一个排行榜。
英文摘要：
As large language models (LLMs) become key advisors in various domains, their cultural sensitivity and reasoning skills are crucial in multicultural environments. We introduce Nunchi-Bench, a benchmark designed to evaluate LLMs' cultural understanding, with a focus on Korean superstitions. The benchmark consists of 247 questions spanning 31 topics, assessing factual knowledge, culturally appropriate advice, and situational interpretation. We evaluate multilingual LLMs in both Korean and English to analyze their ability to reason about Korean cultural contexts and how language variations affect performance. To systematically assess cultural reasoning, we propose a novel evaluation strategy with customized scoring metrics that capture the extent to which models recognize cultural nuances and respond appropriately. Our findings highlight significant challenges in LLMs' cultural reasoning. While models generally recognize factual information, they struggle to apply it in practical scenarios. Furthermore, explicit cultural framing enhances performance more effectively than relying solely on the language of the prompt. To support further research, we publicly release Nunchi-Bench alongside a leaderboard.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
