ExCAM：可解释的文化意识度量
ExCAM: Explainable Cultural Awareness Metrics
AI总结：
提出ExCAM，首个可识别、评分并解释指令-输出对中文化错误的专用评估度量，在平衡测试集上达到80%准确率。
摘要：
评估大型语言模型的文化意识对于确保生成文本的公平性和应用在全球范围内的泛化能力至关重要。最近的基准通过问答或文本生成任务探索食物等文化物品或压力情境下的行为等价值观。然而，创建这些基准需要耗时且昂贵的人工标注。此外，评估自由文本中文化意识的基准很少，且往往依赖过时的评估机制。为弥补这一空白，我们引入了ExCAM，一种可解释的文化意识度量，据我们所知，这是第一个专门用于识别、评分和解释指令-输出对中文化错误的评估度量。为了训练和评估ExCAM，我们引入了ExCAM40k，一个由九个现有基准组成的数据集，我们对其进行了重新格式化并增加了合成错误。与包括GPT-5在内的多个基线相比，ExCAM在平衡测试集上实现了高达80%的最高错误检测准确率。因此，ExCAM为自由文本的细粒度、可解释的文化评估开辟了道路。
英文摘要：
Evaluating the cultural awareness of large language models is crucial to ensure the fairness of generated text and the generalizability of applications across the world. Recent benchmarks explore cultural goods like food or values like behavior in stressful situations through the lens of question answering or text generation tasks. However, creating these benchmarks requires time-intensive and costly human annotations. Also, benchmarks that evaluate cultural awareness in free text are scarce and often rely on dated evaluation mechanisms. To address this gap, we introduce ExCAM, an Explainable Cultural Awareness Metric, which is, to our knowledge, the first dedicated evaluation metric that identifies, rates and explains cultural errors in instruction-output pairs. To train and evaluate ExCAM, we introduce ExCAM40k, a dataset comprised of nine existing benchmarks that we reformat and enhance with synthetic errors. Compared to several baselines, including GPT-5, ExCAM achieves the highest error detection rate with up to 80% accuracy on a balanced test set. Therefore, ExCAM opens the pathway towards fine-grained and explainable cultural evaluation of free text.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
