通过基于代理的不确定性估计改进语言模型中的指令遵循
Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation
AI总结：
针对语言模型指令响应质量评估困难的问题，提出基于贝叶斯近似的URM模型，通过评估响应不确定性优化数据筛选与策略目标，显著提升了模型指令遵循能力。
摘要：
评估语言模型对指令的响应质量至关重要，但由于人类语言在不同语境下的复杂性，这极具挑战。这种复杂性常导致模糊或不一致的解释，使得准确评估变得困难。为解决此问题，我们提出了一种新颖的不确定性感知奖励模型（URM），该模型基于贝叶斯近似，为配对响应的质量引入了稳健的不确定性估计。我们的支持不确定性的代理使用偏好数据集进行训练，它不仅对响应进行奖励评分，还评估其固有的不确定性。实证结果表明，将所提出的代理纳入语言模型训练具有显著益处。我们的方法通过优化训练的数据筛选和改进策略优化目标，提升了语言模型的指令遵循能力，从而在 Vicuna 和 MT-bench 等基准上大幅超越了现有方法。这些发现凸显了我们提出的方法实质性地推进了语言模型训练，并为在语言模型中利用不确定性开辟了新途径。
英文摘要：
Assessing response quality to instructions in language models is vital but challenging due to the complexity of human language across different contexts. This complexity often results in ambiguous or inconsistent interpretations, making accurate assessment difficult. To address this issue, we propose a novel Uncertainty-aware Reward Model (URM) that introduces a robust uncertainty estimation for the quality of paired responses based on Bayesian approximation. Trained with preference datasets, our uncertainty-enabled proxy not only scores rewards for responses but also evaluates their inherent uncertainty. Empirical results demonstrate significant benefits of incorporating the proposed proxy into language model training. Our method boosts the instruction following capability of language models by refining data curation for training and improving policy optimization objectives, thereby surpassing existing methods by a large margin on benchmarks such as Vicuna and MT-bench. These findings highlight that our proposed approach substantially advances language model training and paves a new way of harnessing uncertainty within language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
