真实即更好：将大语言模型与在线人类行为对齐
The Real, the Better: Aligning Large Language Models with Online Human Behaviors
AI总结：
本文提出RLHB框架，通过生成对抗方式直接利用真实在线人类行为对齐大语言模型，以自然语言行为建模和多模型联合训练实现可持续在线对齐，并经实验验证有效。
摘要：
大语言模型对齐被广泛使用和研究，以避免大语言模型产生无用和有害的回复。然而，漫长的训练过程和预定义的偏好偏差阻碍了对在线多样化人类偏好的适应。为此，本文提出了一种对齐框架，称为基于人类行为的强化学习（RLHB），通过直接利用真实的在线人类行为来对齐大语言模型。采用生成对抗框架，生成器被训练以按照预期的人类行为进行回复；而判别器则试图验证查询、回复和人类行为三元组是否来自真实的在线环境。以自然语言形式进行的行为建模以及多模型联合训练机制实现了主动且可持续的在线对齐。实验结果通过人工评估和自动评估均证实了所提方法的有效性。
英文摘要：
Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
