arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.11191cs.CL
收藏

面向大语言模型的人类偏好学习综述

A Survey on Human Preference Learning for Large Language Models

Harbin Institute of Technology(哈尔滨工业大学);Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Soochow University(苏州大学)

Ruili Jiang, Kehai Chen, Xuefeng Bai, Zhixuan He, Juntao Li, Muyun Yang, Tiejun Zhao, Liqiang Nie, Min Zhang

AI总结:

该综述从以偏好为中心的视角,系统梳理了大语言模型人类偏好学习领域在反馈来源格式、偏好建模使用、对齐效果评估三方面的研究进展,并对未来研究方向进行了展望。

摘要:

近期多功能大语言模型(LLMs)的快速发展,很大程度上依赖于通过偏好学习让能力不断增强的基础模型与人类意图对齐,从而提升LLMs在各类场景下的适用性与有效性。尽管已有大量相关研究,但针对人类偏好如何被引入LLMs的研究视角仍较为有限,这可能阻碍人们更深入地理解人类偏好与LLMs之间的关系,也不利于认清其局限性。本综述从以偏好为中心的视角,梳理了LLMs人类偏好学习领域的研究进展,涵盖偏好反馈的来源与格式、偏好信号的建模与使用,以及对齐后LLMs的评估。我们首先根据数据来源与格式对人类反馈进行分类;随后总结人类偏好建模技术,并对比不同模型流派的优缺点;此外,我们还按照利用人类偏好信号的目标,对各类偏好使用方法进行分类梳理;最后,我们总结了当前主流的从人类意图对齐角度评估LLMs的方法,并对LLMs的人类意图对齐研究进行了展望。

英文摘要:

The recent surge of versatile large language models (LLMs) largely depends on aligning increasingly capable foundation models with human intentions by preference learning, enhancing LLMs with excellent applicability and effectiveness in a wide range of contexts. Despite the numerous related studies conducted, a perspective on how human preferences are introduced into LLMs remains limited, which may prevent a deeper comprehension of the relationships between human preferences and LLMs as well as the realization of their limitations. In this survey, we review the progress in exploring human preference learning for LLMs from a preference-centered perspective, covering the sources and formats of preference feedback, the modeling and usage of preference signals, as well as the evaluation of the aligned LLMs. We first categorize the human feedback according to data sources and formats. We then summarize techniques for human preferences modeling and compare the advantages and disadvantages of different schools of models. Moreover, we present various preference usage methods sorted by the objectives to utilize human preference signals. Finally, we summarize some prevailing approaches to evaluate LLMs in terms of alignment with human intentions and discuss our outlooks on the human intention alignment for LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑