arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.00568cs.CL
收藏

一个用于评估基于归因理论的社会偏见的日本基准

A Japanese Benchmark for Evaluating Social Bias in Reasoning Based on Attribution Theory

Institute of Science Tokyo(东京科学大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);AIST(产业技术综合研究所);NII(国立信息学研究所)

Taihei Shiotani, Masahiro Kaneko, Naoaki Okazaki

AI总结:

本文基于归因理论,构建了新的日本基准JUBAKU-v2,用于评估推理中对内群体和外群体行为归因的偏见,通过固定结论来检测模型在文化偏见上的性能差异。

摘要:

在增强大型语言模型(LLMs)公平性方面,评估根植于特定语言地区文化背景的社会偏见至关重要。然而,大多数现有日本基准严重依赖将英语数据翻译成日语,这不一定能提供适合日本文化的评估。此外,它们只评估结论中的偏见,未能捕捉推理中的偏见。在本研究中,基于社会心理学中的归因理论,我们构建了一个新的数据集,'JUBAKU-v2',该数据集在固定结论的情况下评估推理中对内群体和外群体行为归因的偏见。该数据集包含216个反映日本特定文化偏见的例子。实验结果验证了该数据集比现有基准更能敏感地检测模型在文化偏见上的性能差异。

英文摘要:

In enhancing the fairness of Large Language Models (LLMs), evaluating social biases rooted in the cultural contexts of specific linguistic regions is essential. However, most existing Japanese benchmarks heavily rely on translating English data, which does not necessarily provide an evaluation suitable for Japanese culture. Furthermore, they only evaluate bias in the conclusion, failing to capture biases lurking in the reasoning. In this study, based on attribution theory in social psychology, we constructed a new dataset, ``JUBAKU-v2,'' which evaluates the bias in attributing behaviors to in-groups and out-groups within reasoning while fixing the conclusion. This dataset consists of 216 examples reflecting cultural biases specific to Japan. Experimental results verified that it can detect performance differences across models more sensitively than existing benchmarks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑