发表机构
Stanford University; University of Toronto（斯坦福大学; 多伦多大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出CreativePreferences数据集（2.8M文本、317M判断、42任务），区分偏好中可验证、可表述与隐性成分，发现普遍存在表述性和可验证性差距，并指出表述偏好会偏离隐性维度。
AI 中文摘要
是什么让一个短篇故事引人入胜？一篇新闻文章具有新闻价值？或一个数学证明优雅？这些概念难以表述或验证；其含义至少部分是隐性的。然而，现代AI模型主要通过表述性的章程、评分标准和验证器（即在RLAIF和RLVR中）来改进；偏好的隐性组成部分通常研究不足。我们引入了一个大型、带标签的偏好数据集CreativePreferences，包含2.8M条文本，由317M条人类偏好判断标注，覆盖7个创意领域，共42个基准任务。我们分别使用可执行程序、评分标准库和密集训练模型（分别为V、A和VAT）对这些标签进行建模。我们观察到稳健的表述性差距（VAT-VA）和可验证性差距（VAT-V）；我们通过一种新颖的测量方法估计每个差距的上下界，该方法发现可表述和可验证的度量，识别虚假变量，并使用捕获-再捕获方法估计未发现度量的价值。这些差距出现在所有领域，即使在传统上被视为完全可验证的领域：以正确性为中心的领域（即数学和软件工程）以及以主张和新颖性为中心的领域（即新闻、专利、同行评审）。差距的大小因领域而异（例如，同行评审和创意写作具有最大的表述性差距），并随着更多人参与判断而扩大，这与Collins的集体隐性知识一致。我们展示了两个后果：（1）在人类生成内容上，完整模型更接近人类偏好，通常与表述性标准不一致；（2）类比古德哈特定律，表述偏好会使其偏离隐性维度。表述性和可验证性差距具有重要影响；我们给出了关于任务何时可以被提示、学习机制如何改进以及何时应将判断留给人类的建议。
英文摘要
What makes a short story gripping; a news article newsworthy; or a math proof elegant? These constructs resist articulation or verification; their meaning is at least partially tacit. However, modern AI models are improved primarily via articulated constitutions, rubrics and verifiers (i.e. in RLAIF and RLVR); tacit components of preferences are typically understudied. We introduce a large, labeled preference dataset CreativePreferences, containing 2.8M texts labeled by 317M human preference judgments across 7 creative domains, with 42 benchmark tasks. We model these labels with executable programs, rubric banks and densely trained models (V, A and VAT, respectively). We observe robust articulability gaps, VAT-VA; and verifiability gaps, VAT-V; we estimate upper and lower bounds for each gap with a novel measurement approach that discovers articulable and verifiable metrics, identifies spurious variables and estimates the value of undiscovered metrics using capture-recapture. These gaps occur across all domains, even in domains traditionally treated as fully verifiable: correctness-centered domains (i.e. mathematics and software engineering) and claim- and novelty-centric domains (i.e. news, patents, peer review). The size of the gap varies based on domain (e.g. peer review and creative writing have the largest articulability gaps) and widens as more people take part in the judgment, consistent with Collins' collective tacit knowledge. We show two consequences: (1) on human generations, the full model more closely matches human preferences, often in disagreement with articulated criteria, and (2) in an analogy to Goodhart's law, articulating preference shifts it away from the tacit dimension. Articulability and verifiability gaps are consequential; we give recommendations on when tasks can be prompted; how learning mechanisms might improve; and when to leave judgments with humans.
Comments15 pages main text, 14 pages of references, 107-page appendix (136 pages total); 15 figures, 48 tables; 213 references
