arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2406.15178cs.CL
收藏

面向大语言模型的混合对齐训练

Hybrid Alignment Training for Large Language Models

School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院);NiuTrans Research(NiuTrans研究所);Institute of Psychology, CAS(中国科学院心理研究所)

Chenglong Wang, Hang Zhou, Kaiyan Chang, Bei Li, Yongyu Mu, Tong Xiao, Tongran Liu, Jingbo Zhu

AI总结:

针对大语言模型两阶段对齐训练中指令遵循与人类偏好目标易冲突的问题,提出基于交替优化与改进弹性权重巩固的混合对齐训练方法Hbat,在摘要、对话任务上显著优于传统两阶段方法及各类基线。

摘要:

对齐训练对大语言模型(LLM)贴合人类意图与偏好至关重要,通常分为目标不同的两个阶段:指令遵循对齐与人类偏好对齐。但按顺序执行这两类对齐存在固有问题:两类目标可能冲突,LLM无法保证同时良好对齐指令与人类偏好。针对该问题,本文提出混合对齐训练方法Hbat,基于交替对齐与改进的弹性权重巩固方法实现,核心思路是在对齐训练中交替优化不同目标,使两类对齐任务更好协同。我们在摘要生成与对话任务上开展Hbat实验,结果表明所提Hbat显著优于所有基线;值得注意的是,无论采用近端策略优化还是直接偏好优化,Hbat相比传统两阶段对齐训练均能带来稳定的性能提升。

英文摘要:

Alignment training is crucial for enabling large language models (LLMs) to cater to human intentions and preferences. It is typically performed based on two stages with different objectives: instruction-following alignment and human-preference alignment. However, aligning LLMs with these objectives in sequence suffers from an inherent problem: the objectives may conflict, and the LLMs cannot guarantee to simultaneously align with the instructions and human preferences well. To response to these, in this work, we propose a Hybrid Alignment Training (Hbat) approach, based on alternating alignment and modified elastic weight consolidation methods. The basic idea is to alternate between different objectives during alignment training, so that better collaboration can be achieved between the two alignment tasks.We experiment with Hbat on summarization and dialogue tasks. Experimental results show that the proposed \textsc{Hbat} can significantly outperform all baselines. Notably, Hbat yields consistent performance gains over the traditional two-stage alignment training when using both proximal policy optimization and direct preference optimization.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑