大型语言模型在偏好标注中的冗长性偏差
Verbosity Bias in Preference Labeling by Large Language Models
AI总结：
本文研究大型语言模型在偏好标注中的冗长性偏差，发现GPT-4比人类更偏好长答案，并提出一种度量该偏差的指标。
摘要：
近年来，大型语言模型（LLMs）的普及程度显著激增，改变了自然语言处理和机器学习的格局。提升LLMs性能的一个关键因素是通过人类反馈的强化学习（RLHF）实现与人类的对齐，例如GPT-4、Bard等许多LLMs都采用了这一方法。此外，最近的研究正在探索用其他LLMs的反馈替代人类反馈，称为人工智能反馈的强化学习（RLAIF）。我们考察了使用其他LLMs评估LLMs时带来的偏差，并更深入地研究了冗长性偏差——即LLMs有时偏好更冗长的答案，即使这些答案质量相似。我们发现，在我们的问题设置中，GPT-4比人类更偏好较长的答案。我们还提出了一种度量这种偏差的指标。
英文摘要：
In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
