从软目标到奖励信号：分配与奖励目标如何交互

中文摘要 AI 辅助

软偏好目标指定监督强度，奖励目标将该强度转换为学习到的奖励信号。一个核心设计问题仍然存在：将一组固定的偏好强度分配给不同的响应对，会如何改变不同目标产生的奖励？我们引入分配几何来研究这种交互。均值匹配平滑控制目标离散度，而层内重分配改变对应关系并保持完整的目标分布。在五种奖励目标中，在共同的精度等价预算内，完整对应关系在比较的软目标中保留了最大的干净偏好边际。衰减顺序随奖励目标而变化，揭示了对相同目标分配的不同响应。独立重分配和相关的源构造再现了保留方向。衰减-保留概况通过边际幅度、编辑响应和精度来比较这些组合。与独立校准的缩放相比，APLOT均匀目标在聚合编辑和呈现编辑上均提供额外的衰减。这些发现建立了一个联合设计空间，其中目标放置和奖励目标塑造超出偏好精度的奖励属性。

英文摘要

Soft preference targets specify supervision strength, and reward objectives convert that strength into learned reward signals. A central design question remains: how does assigning a fixed set of preference strengths to different response pairs change the rewards produced by different objectives? We introduce assignment geometry to study this interaction. Mean-matched smoothing controls target dispersion, while within-stratum reassignment changes correspondence and preserves the complete target distribution. Across five reward objectives, intact correspondence retains the largest clean preference margins among the compared soft targets within a common accuracy-equivalence budget. Attenuation orderings change with the reward objective, revealing different responses to the same target assignments. Independent reassignments and a related source construction reproduce the retention direction. An attenuation-retention profile compares these combinations through margin magnitude, edit response, and accuracy. Against independently calibrated scaling, APLOT uniform targets deliver additional attenuation on both aggregate and presentation edits. These findings establish a joint design space in which target placement and reward objective shape reward properties beyond preference accuracy.

发表机构

• Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）
• School of Vehicle and Mobility, Tsinghua University（清华大学车辆与运载学院）
• Tencent Holdings Limited（腾讯控股有限公司）
• SZ DJI Technology Co., Ltd.（深圳市大疆创新科技有限公司）

机构由 AI 辅助整理，请以论文原文为准。