RATE：基于不完美反事实的奖励模型因果可解释性
RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
AI总结：
针对奖励模型黑箱难以明确奖励依据的问题，本文提出RATE方法，利用大语言模型生成不完美反事实并通过两次改写校正偏差，以衡量回复属性对奖励的因果效应，验证了其有效性。
摘要：
在对齐或评估大语言模型（LLM）时，奖励模型被广泛用作人类偏好的代理指标。然而，奖励模型属于黑箱，人们通常并不清楚它们实际奖励的究竟是什么。本文提出了基于改写的属性处理估计器（Rewrite-based Attribute Treatment Estimator，RATE），这是一种用于衡量奖励模型对回复高层属性（如情感、有用性或复杂度）敏感度的有效方法。重要的是，RATE衡量的是属性对奖励的因果效应。RATE利用LLM改写回复，生成可用于衡量因果效应的不完美反事实样本。一个关键挑战在于，这些改写存在缺陷，可能会导致奖励模型对该属性敏感度的估计出现显著偏差。RATE的核心思路是通过两次改写来校正这种不完美改写带来的影响。我们验证了RATE流程的有效性，并通过实验证明它是一种有效的估计器。
英文摘要：
Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and it is often unclear what, exactly, they are actually rewarding. In this paper we develop Rewrite-based Attribute Treatment Estimator (RATE) as an effective method for measuring the sensitivity of a reward model to high-level attributes of responses, such as sentiment, helpfulness, or complexity. Importantly, RATE measures the causal effect of an attribute on the reward. RATE uses LLMs to rewrite responses to produce imperfect counterfactuals examples that can be used to measure causal effects. A key challenge is that these rewrites are imperfect in a manner that can induce substantial bias in the estimated sensitivity of the reward model to the attribute. The core idea of RATE is to adjust for this imperfect-rewrite effect by rewriting twice. We establish the validity of the RATE procedure and show empirically that it is an effective estimator.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
