超越正确性:跨文化的主观写作偏好评估
Beyond Correctness: Evaluating Subjective Writing Preferences Across Cultures
AI总结:
本文提出WritingPreferenceBench数据集,发现基于序列的奖励模型在评估主观写作偏好时表现不佳,而生成奖励模型通过推理链达到更高准确率,揭示了当前RLHF方法在捕捉主观偏好方面的局限性。
摘要:
当前的偏好学习方法在标准基准上实现高精度,但在去除客观质量信号时表现出显著的性能下降。我们引入了WritingPreferenceBench,这是一个包含1800对人工标注的偏好对(1200个英文,600个中文)的数据集,涵盖8种创意写作类型,其中响应在客观正确性、事实准确性和长度上进行了匹配。在该基准上,基于序列的奖励模型——RLHF的标准架构——仅达到52.7%的平均准确率,而零样本语言模型判断者达到53.9%。相比之下,生成奖励模型,它们会产生明确的推理链,达到81.8%的准确率。我们观察到在不同写作类型中模型内方差较高:个体模型在不同写作类别中的准确率从18.2%到81.8%不等,标准差平均为10.1%。这种方差在模型规模上持续存在,27B参数模型在8B变体上没有一致的改进。我们的结果表明,当前的RLHF方法主要学会检测客观错误,而不是捕捉主观质量偏好(例如创造力、风格 flair 和情感共鸣),并且成功的偏好建模可能需要中间推理表示,而不是直接分类。
英文摘要:
Current preference learning methods achieve high accuracy on standard benchmarks but exhibit significant performance degradation when objective quality signals are removed. We introduce WritingPreferenceBench, a dataset of 1,800 human-annotated preference pairs (1,200 English, 600 Chinese) across 8 creative writing genres, where responses are matched for objective correctness, factual accuracy, and length. On this benchmark, sequence-based reward models--the standard architecture for RLHF--achieve only 52.7% mean accuracy, while zero-shot language model judges perform at 53.9%. In contrast, generative reward models that produce explicit reasoning chains achieve 81.8% accuracy. We observe high within-model variance across genres: individual models range from 18.2% to 81.8% accuracy across different writing categories, with standard deviations averaging 10.1%. This variance persists regardless of model scale, with 27B parameter models showing no consistent improvement over 8B variants. Our results suggest that current RLHF methods primarily learn to detect objective errors rather than capture subjective quality preferences (e.g., creativity, stylistic flair, and emotional resonance), and that successful preference modeling may require intermediate reasoning representations rather than direct classification.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。