需要多少人判断？人类偏好评估的可行性极限
How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation
AI总结：
研究发现人类偏好评估中，当偏好信号分散时，比例分配是最佳策略，而精心设计的基准测试能通过减少方差提高检测性。
摘要：
人类偏好评估被广泛用于比较生成模型，但仍然不清楚需要多少判断才能可靠地检测到微小改进。我们证明，当偏好信号在提示（即所有提示类型信息相似）中分散时，比例分配是极小大最优的：没有分配策略能显著提高可检测性。对大规模人类偏好数据集的实证分析显示，大多数比较都处于这种分散模式，表现出需要远多于通常收集的判断的小偏好边际，即使在良好采样比较中也是如此。这些限制在评估协议和模态中持续存在，包括聊天、图像生成和带有执行反馈的代码生成。相比之下，经过精心设计的基准测试通过系统地减少提示诱导的变异性，诱导出更大的边际并通过减少提示级方差的1.5倍来提高可检测性。我们的结果表明，不确定或负面的人类评估结果经常反映的是评估不足而非模型等价，强调了必须明确考虑效应大小、预算和协议设计的必要性。
英文摘要：
Human preference evaluations are widely used to compare generative models, yet it remains unclear how many judgments are required to reliably detect small improvements. We show that when preference signal is diffuse across prompts (i.e., all prompt types are similarly informative), proportional allocation is minimax-optimal: no allocation strategy substantially improves detectability. Empirical analysis of large-scale human preference datasets shows that most comparisons fall into this diffuse regime, exhibiting small preference margins that require far more judgments than typically collected, even in well-sampled comparisons. These limits persist across evaluation protocols and modalities, including chat, image generation, and code generation with execution feedback. In contrast, curated benchmarks that reduce prompt induced variability systematically induce larger margins and improve detectability through a $1.5\times$ reduction in prompt-level variance. Our results show that inconclusive or negative human evaluation outcomes frequently reflect underpowered evaluation rather than model equivalence, underscoring the need to account explicitly for effect size, budget, and protocol design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
