在人类-模型偏好对齐中最大化信号
Maximizing Signal in Human-Model Preference Alignment
AI总结：
本文探讨如何通过分离标注噪声与信号、遵循方法学最佳实践，并利用人类判断评估护栏分类器，从而在模型训练与评估中最大化人类反馈信号，使模型行为与用户偏好对齐。
摘要：
强大的大语言模型（LLMs）的出现引发了自然语言理解和自然语言生成领域的范式转变。使LLMs在这些任务中极具价值的特性——创造力、生成流畅语言的能力以及从大规模语料库中快速有效地抽取信息的能力——也给评估其输出带来了新的挑战。急于将产品推向市场导致团队退而依赖快速、成本效益高的自动评估，这些评估虽有价值，但并不能免除在模型训练和评估中对人类判断的需求。本文主张，在最终用户需要认同机器学习模型所做决策的情况下——例如毒性检测或为摘要提取要点——模型应当基于代表这些用户偏好的数据进行训练和评估。我们通过阐明人类反馈在模型训练和评估的标注与判断任务中的作用来支持这一论点。首先，我们提出了在标注任务中分离噪声与信号的方法。然后，我们表明，通过遵循经过验证的方法学最佳实践，可以将标注分歧中的噪声降至最低，同时可以将信号最大化，使其在模型训练和评估任务中发挥不可或缺的作用。最后，我们通过一个案例研究来说明最佳实践，在该案例中，使用人类判断对两个护栏分类器进行评估，以使最终模型行为与用户偏好对齐。我们希望本文能为研究人员和专业人士提供将人类判断整合到其机器学习和生成式AI评估工具包中的指导方针，尤其是在致力于实现与用户需求和期望对齐的准确且无偏的特征时。
英文摘要：
The emergence of powerful LLMs has led to a paradigm shift in Natural Language Understanding and Natural Language Generation. The properties that make LLMs so valuable for these tasks -- creativity, ability to produce fluent speech, and ability to quickly and effectively abstract information from large corpora -- also present new challenges to evaluating their outputs. The rush to market has led teams to fall back on quick, cost-effective automatic evaluations which offer value, but do not obviate the need for human judgments in model training and evaluation. This paper argues that in cases in which end users need to agree with the decisions made by ML models -- e.g. in toxicity detection or extraction of main points for summarization -- models should be trained and evaluated on data that represent the preferences of those users. We support this argument by explicating the role of human feedback in labeling and judgment tasks for model training and evaluation. First, we propose methods for disentangling noise from signal in labeling tasks. Then we show that noise in labeling disagreement can be minimized by adhering to proven methodological best practices, while signal can be maximized to play an integral role in model training and evaluation tasks. Finally, we illustrate best practices by providing a case study in which two guardrails classifiers are evaluated using human judgments to align final model behavior to user preferences. We aim for this paper to provide researchers and professionals with guidelines to integrating human judgments into their ML and generative AI evaluation toolkit, particularly when working toward achieving accurate and unbiased features that align with users' needs and expectations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
