评估与改进用于LLM对齐的奖励模型的文化意识
Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignment
AI总结：
针对奖励模型文化意识评估缺失问题，提出CARB基准，并设计Think-as-Locals方法利用RLVR提升文化感知奖励建模，实验验证其有效性。
摘要：
奖励模型（RMs）对于使大型语言模型（LLMs）与多元文化对齐至关重要。因此，评估其文化意识对于进一步推进LLMs的全球对齐至关重要。然而，现有的奖励模型评估因缺乏文化相关的评估数据集，在评估文化意识方面存在不足。为填补这一空白，我们提出了文化意识奖励建模基准（CARB），涵盖4个文化领域的10种不同文化。我们对最先进的奖励模型进行的广泛评估揭示了它们在文化意识建模方面的缺陷，并展示了在CARB上的性能与下游多语言文化对齐任务之间的正相关性。进一步分析识别了文化感知奖励建模中的虚假相关性，即奖励模型的评分主要依赖于表面特征，而非真实的文化细微差别理解。为解决这些问题，我们提出了Think-as-Locals方法，通过基于可验证奖励的强化学习（RLVR）从生成式奖励模型中引出更深层的文化基础推理，并采用精心设计的奖励以确保准确的偏好判断和高质量的结构化评估标准生成。实验结果验证了其在减轻虚假特征干扰和推进文化感知奖励建模方面的有效性。
英文摘要：
Reward models (RMs) are crucial for aligning large language models (LLMs) with diverse cultures. Consequently, evaluating their cultural awareness is essential for further advancing global alignment of LLMs. However, existing RM evaluations fall short in assessing cultural awareness due to the scarcity of culturally relevant evaluation datasets. To fill this gap, we propose Cultural Awareness Reward modeling Benchmark (CARB), covering 10 distinct cultures across 4 cultural domains. Our extensive evaluation of state-of-the-art RMs reveals their deficiencies in modeling cultural awareness and demonstrates a positive correlation between performance on CARB and downstream multilingual cultural alignment tasks. Further analysis identifies the spurious correlations within culture-aware reward modeling, wherein RM's scoring relies predominantly on surface-level features rather than authentic cultural nuance understanding. To address these, we propose Think-as-Locals to elicit deeper culturally grounded reasoning from generative RMs via reinforcement learning from verifiable rewards (RLVR) and employ well-designed rewards to ensure accurate preference judgments and high-quality structured evaluation criteria generation. Experimental results validate its efficacy in mitigating spurious features interference and advancing culture-aware reward modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
