arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.24124cs.LGcs.AIcs.CL
收藏

对齐税:对齐语言模型中的响应同质化及其对不确定性估计的影响

The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation

Mingyi Liu

AI总结:

研究发现对齐语言模型存在响应同质化现象,影响不确定性估计方法的效果,提出通过叠加不确定性信号提升模型性能。

摘要:

经过强化学习与人类反馈对齐的语言模型表现出响应同质化:在TruthfulQA(n=790)中,40-79%的问题在10个独立样本中产生单一语义簇。在受影响的问题上,基于采样的不确定性方法无判别能力(AUROC=0.500),而自由令牌熵保留信号(0.603)。这种对齐税是任务依赖的:在GSM8K(n=500)中,令牌熵达到0.724(Cohen's d=0.81)。基线与指令模型的消融分析确认了对齐的因果作用:基线模型单簇率1.0%,指令模型28.5%(p < 10^{-6})。训练阶段消融(Base 0.0% -> SFT 1.5% -> DPO 4.0% SCR)将原因定位在DPO而非SFT。四个模型家族的跨家族复制显示对齐税的严重性因家族和规模而异。在22个实验、5个基准、4个模型家族和3个模型规模(3B-14B)上验证,使用Jaccard、嵌入和NLI基线在三个DeBERTa规模(全部约0.51 AUROC)上验证。跨嵌入器验证使用两个独立嵌入家族排除耦合偏差。在WebQuestions(58.0% SCR)上的跨数据集验证证实了超越TruthfulQA的泛化。核心发现——响应同质化——是实现无关且标签无关的。受此诊断启发,我们探索了一个最便宜的优先级级联(UCBD)以叠加正交的不确定性信号。选择性预测将GSM8K准确性从84.4%提升到93.2%在50%覆盖率下;弱相关边界(|r| <= 0.12)实现57%的成本节省。

英文摘要:

RLHF-aligned language models exhibit response homogenization: on TruthfulQA (n=790), 40-79% of questions produce a single semantic cluster across 10 i.i.d. samples. On affected questions, sampling-based uncertainty methods have zero discriminative power (AUROC=0.500), while free token entropy retains signal (0.603). This alignment tax is task-dependent: on GSM8K (n=500), token entropy achieves 0.724 (Cohen's d=0.81). A base-vs-instruct ablation confirms the causal role of alignment: the base model shows 1.0% single-cluster rate vs. 28.5% for the instruct model (p < 10^{-6}). A training stage ablation (Base 0.0% -> SFT 1.5% -> DPO 4.0% SCR) localizes the cause to DPO, not SFT. Cross-family replication on four model families reveals alignment tax severity varies by family and scale. We validate across 22 experiments, 5 benchmarks, 4 model families, and 3 model scales (3B-14B), with Jaccard, embedding, and NLI-based baselines at three DeBERTa scales (all ~0.51 AUROC). Cross-embedder validation with two independent embedding families rules out coupling bias. Cross-dataset validation on WebQuestions (58.0% SCR) confirms generalization beyond TruthfulQA. The central finding -- response homogenization -- is implementation-independent and label-free. Motivated by this diagnosis, we explore a cheapest-first cascade (UCBD) over orthogonal uncertainty signals. Selective prediction raises GSM8K accuracy from 84.4% to 93.2% at 50% coverage; weakly dependent boundaries (|r| <= 0.12) enable 57% cost savings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑