符合反馈对齐：量化答案级可靠性以实现鲁棒的大语言模型对齐
Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
AI总结：
Conformal Feedback Alignment通过量化答案级可靠性提升大语言模型对齐的鲁棒性和数据效率。
摘要：
基于偏好的对齐方法，如人类反馈强化学习（RLHF），通过成对偏好学习，但标签往往嘈杂且不一致。现有的不确定性意识方法对偏好进行加权，但忽略了更根本的因素：被比较的答案的可靠性。为了解决这个问题，我们提出了符合反馈对齐（CFA），一种框架，该框架将偏好加权建立在统计保证的符合预测（CP）之上。CFA通过构建具有可控覆盖的符合预测集来量化答案级可靠性，并将这些可靠性汇总为DPO和PPO风格训练的原理性权重。在不同数据集上的实验表明，CFA提高了对齐的鲁棒性和数据效率，突显了建模答案侧不确定性可以补充偏好层面的加权，并产生更鲁棒、更高效的数据对齐。代码已提供。
英文摘要：
Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the \emph{answers} being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling \emph{answer-side} uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
