arXiv 2606.105692026-06-09cs.CLcs.AI

隐藏共识：人类反馈中的偏好有效性压缩

Hidden Consensus:Preference-Validity Compression in Human Feedback

YTL AI Labs；Universiti Malaya（马来亚大学）；Monash University Malaysia（莫纳什大学马来西亚校区）；Universiti Malaysia Sarawak（马来西亚沙捞越大学）

Dorcas Chia Ern Chua, Karen Myn Hui Lee, Jia Yue Tan, Zhen Xue Gue, Norzalena Abdul Hamid, Azima Binti Azmi, Keat Mei Yeong, Aizat Izyani binti Mujab, Hafsah No… 展开作者

Dorcas Chia Ern Chua, Karen Myn Hui Lee, Jia Yue Tan, Zhen Xue Gue, Norzalena Abdul Hamid, Azima Binti Azmi, Keat Mei Yeong, Aizat Izyani binti Mujab, Hafsah Noor Azam, Chee Guo Khoo, Han Ying Lim, Chee Seng Chan

备注：: 28 pages. When AI learns from human feedback, it forces a single "correct" answer, but sometimes multiple answers are all genuinely valid, and that nuance gets thrown away

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出偏好有效性压缩问题，即RLHF将多元有效反馈压缩为单一奖励目标，导致对齐测量偏差。通过马来西亚语料分析，79%的提示存在多个多数支持响应，表明多数聚合测量的是argmax可接受性而非多元对齐。

摘要：

标准的RLHF流程通常将异质的人类判断简化为单一的标量奖励目标。我们认为这种简化在结构多元的社会中可能错误地衡量对齐，在这些社会中，分歧可能反映文化、历史、语言、区域或规范性的解释，而非标注噪声。我们将这种失败称为偏好有效性压缩，即多个多元有效的响应选项被压缩成一个优化目标。以马来西亚为诊断场景，我们通过偏好事件分析RLHF风格的反馈聚合，这些事件将提示、响应和跨解释框架的可接受性判断联系起来。在来自20名参与者和107个三人标注提示的321个偏好事件中，79%的提示包含多个多数支持的响应，而单一赢家聚合会丢弃这些响应，并且当考虑所有多数支持的选项时，顶部响应之间的明显优势差距会消失。参与者经常选择多个可接受的响应，而被丢弃的响应明显反映了连贯的本地、实践或文化框架。这些发现表明，该语料中的多数聚合测量的是argmax可接受性而非多元对齐。我们将此视为测量有效性问题，并认为未来的对齐方法应满足有效性保持一致性，即在多元有效的解释框架中保持稳定，而不是将它们压缩为单一的奖励目标。

英文摘要：

Standard RLHF pipelines often reduce heterogeneous human judgments into a single scalar reward target. We argue that this reduction can mis-measure alignment in structurally plural societies, where disagreement may reflect culturally, historically, linguistically, regionally, or normatively grounded interpretations rather than annotation noise. We call this failure Preference-Validity Compression, the collapse of multiple plural-valid response options into a single optimization target. Using Malaysia as a diagnostic setting, we analyze RLHF-style feedback aggregation through preference events linking prompts, responses, and acceptability judgments across interpretive frames. Across 321 preference events from 20 participants and 107 trio-annotated prompts, 79% of prompts contain more than one majority-supported response that single-winner aggregation would discard, and apparent dominance gaps between top responses diminish when all majority-supported options are considered. Participants frequently select multiple acceptable responses, and discarded responses demonstrably reflect coherent local, practical, or cultural frames. These findings show that majority aggregation in this corpus measures argmax acceptability rather than plural alignment. We treat this as a measurement-validity issue and argue that future alignment methods should satisfy Validity-Preserving Consistency, remaining stable across plural-valid interpretive frames rather than collapsing them into a single reward target.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。