CURE：文化理解与推理评估——面向大语言模型“厚”文化对齐评估的框架
CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs
AI总结：
针对现有大语言模型文化能力评估脱离语境、结果不稳定的问题，提出CURE厚文化对齐评估框架，通过真实场景与多维度指标实现更准确稳定的文化能力测评。
摘要：
大语言模型（LLMs）正越来越多地部署在多元文化环境中，但现有对其文化能力的评估仍存在局限。现有方法聚焦于脱离语境的正确性判断或强制选择评判，忽略了生成恰当响应所需的文化理解与推理能力要求。为填补这一空白，我们推出一套基准测试集，该测试集并非直接探查抽象规范或孤立陈述，而是为模型呈现需要基于文化推理的真实场景语境。除标准的精确匹配（Exact Match）指标外，我们引入四个补充指标（覆盖度Coverage、特异性Specificity、内涵Connotation、连贯性Coherence），以捕捉模型响应质量的不同维度。针对前沿模型的实证分析显示，薄评估会系统性高估模型的文化能力，且会产生高方差的不稳定评估结果。相比之下，厚评估能够暴露推理深度的差异，降低方差，为文化理解提供更稳定、可解释的信号。
英文摘要：
Large language models (LLMs) are increasingly deployed in culturally diverse environments, yet existing evaluations of cultural competence remain limited. Existing methods focus on de-contextualized correctness or forced-choice judgments, overlooking the need for cultural understanding and reasoning required for appropriate responses. To address this gap, we introduce a set of benchmarks that, instead of directly probing abstract norms or isolated statements, present models with realistic situational contexts that require culturally grounded reasoning. In addition to the standard Exact Match metric, we introduce four complementary metrics (Coverage, Specificity, Connotation, and Coherence) to capture different dimensions of model's response quality. Empirical analysis across frontier models reveals that thin evaluation systematically overestimates cultural competence and produces unstable assessments with high variance. In contrast, thick evaluation exposes differences in reasoning depth, reduces variance, and provides more stable, interpretable signals of cultural understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
