面向长文问答的公理化偏好建模
Axiomatic Preference Modeling for Longform Question Answering
AI总结：
本文提出公理化偏好建模框架，通过原则驱动生成偏好信号训练约220M参数模型，使其在长文答案评分上与人类标签一致性超过 GPT-4。
摘要：
GPT-4 等大型语言模型（LLM）的卓越能力，部分源于诸如基于人类反馈的强化学习（RLHF）等后训练过程，该过程涉及在奖励模型中编码的人类偏好。然而，这些奖励模型（RM）通常并不直接了解偏好标注是依据何种原因或原则作出的。在本研究中，我们识别出可引导 RM 更好地与人类偏好保持一致的原则，并随后开发了一个公理化框架，用于生成丰富多样的偏好信号以维护这些原则。我们利用这些公理化信号训练一个模型，为长文问题的答案打分。我们的方法产生了一个仅有约 220M 参数的 Preference Model（偏好模型），其与黄金人工标注偏好标签的一致性高于 GPT-4。本工作的贡献包括：训练了一个独立的偏好模型，能够在同一尺度上对人类生成和 LLM 生成的答案进行评分；开发了一个公理化框架，用于生成针对特定原则定制的训练数据对；并表明少量公理化信号即可帮助小型模型在偏好评分方面超越 GPT-4。我们在 huggingface 上发布了模型：https://huggingface.co/corbyrosset/axiomatic_preference_model
英文摘要：
The remarkable abilities of large language models (LLMs) like GPT-4 partially stem from post-training processes like Reinforcement Learning from Human Feedback (RLHF) involving human preferences encoded in a reward model. However, these reward models (RMs) often lack direct knowledge of why, or under what principles, the preferences annotations were made. In this study, we identify principles that guide RMs to better align with human preferences, and then develop an axiomatic framework to generate a rich variety of preference signals to uphold them. We use these axiomatic signals to train a model for scoring answers to longform questions. Our approach yields a Preference Model with only about 220M parameters that agrees with gold human-annotated preference labels more often than GPT-4. The contributions of this work include: training a standalone preference model that can score human- and LLM-generated answers on the same scale; developing an axiomatic framework for generating training data pairs tailored to certain principles; and showing that a small amount of axiomatic signals can help small models outperform GPT-4 in preference scoring. We release our model on huggingface: https://huggingface.co/corbyrosset/axiomatic_preference_model
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
