arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.07401cs.AI
收藏

大语言模型对齐中的多样化偏好研究

On Diversified Preferences of Large Language Model Alignment

Tencent AI Lab(腾讯AI实验室);HiThink Research(海思思考研究院);Alibaba Group(阿里巴巴集团);Peng Cheng Lab(鹏城实验室)

Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, Zenglin Xu

AI总结:

针对人类偏好多样性阻碍大语言模型对齐效果的问题,本文定量分析了偏好多样性对不同规模奖励模型的影响规律,提出ECE评估指标与MORE多目标奖励学习方法,实验验证其能有效提升对齐性能。

摘要:

使大语言模型(LLM)与人类偏好对齐,已被公认为提升LLM交互质量的关键。然而在这个多元化的世界中,标注者的不同品味会导致人类偏好存在多样性,这会阻碍LLM对齐方法的效果。本文首次对不同规模(参数从13亿到70亿)、在具备多样化偏好的人类反馈数据上训练的奖励模型,开展了实验缩放定律的定量分析。分析表明,人类偏好多样化带来的影响同时取决于模型规模和数据规模:容量充足的更大模型能缓解多样化偏好的负面影响,而较小的模型难以适配这类偏好。为减轻多样化偏好的影响,我们引入了期望校准误差(Expected Calibration Error, ECE)这一新指标来评估奖励模型(RM),并证明该指标与LLM的对齐性能存在显著正相关。此外,我们提出了多目标奖励学习方法(Multi-Objective Reward learning, MORE),以提升奖励模型在共享偏好上的校准性能。通过在4个模型、5个人类偏好数据集上开展实验,我们发现校准误差可作为评估奖励模型的关键指标,且MORE方法能取得更优的对齐性能。

英文摘要:

Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotators' different tastes, which hinders the effectiveness of LLM alignment methods. This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse preferences. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them. To mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of LLMs. Furthermore, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared preferences. Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑