Style Outweighs Substance: LLM Judges在Alignment Benchmarking中的Failure Modes
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
AI总结：
本文发现LLM-judges的偏好与alignment的具体指标不相关，且优先考虑style而非factuality和safety，同时指出post-training中的SFT阶段对alignment影响最大。
摘要：
2022年11月ChatGPT的发布引发了人们对post-training的极大兴趣，以及大量新的preference optimization (PO)方法。这些方法声称通过与人类成对偏好更好的对应来实现更好的alignment，这通常由LLM-judges来衡量。在这项工作中，我们试图回答以下问题——LLM-judge的偏好是否能转化为在alignment的其他更具体的指标上的进展，如果不能，为什么不能？我们定义了一个具体的alignment指标，并引入了SOS-Bench (Substance Outweighs Style Benchmark)，据我们所知，这是迄今为止最大的标准化、可复现的LLM meta-benchmark。我们发现：(1) LLM-judge的偏好与safety、world knowledge和instruction following的具体衡量标准不相关；(2) LLM-judges具有强大的隐式偏见，优先考虑style而非factuality和safety；以及(3) post-training的supervised fine-tuning (SFT)阶段，而不是PO阶段，对alignment的影响最大，其中data scaling和prompt diversity是驱动因素。我们的代码库和完整结果可在https://github.com/penfever/sos-bench找到。
英文摘要：
The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior alignment by virtue of better correspondence with human pairwise preferences, often measured by LLM-judges. In this work, we attempt to answer the following question -- do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? We define a concrete metric for alignment, and introduce SOS-Bench (Substance Outweighs Style Benchmark), which is to the best of our knowledge the largest standardized, reproducible LLM meta-benchmark to date. We find that (1) LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following; (2) LLM-judges have powerful implicit biases, prioritizing style over factuality and safety; and (3) the supervised fine-tuning (SFT) stage of post-training, and not the PO stage, has the greatest impact on alignment, with data scaling and prompt diversity as the driving factors. Our codebase and complete results can be found at https://github.com/penfever/sos-bench.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
