对数评分，幂律发现：在基于代理的评估中区分测量与覆盖
Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
AI总结：
本文通过960次实验展示基于代理的评估在Turing式验证中与人类评分无差异，发现评分与独特问题发现之间存在对数与幂律的分离，揭示了发现空间的幂律分布特性。
摘要：
基于大语言模型的代理评判者是评估对话AI的新方法，但其评估可信度及所需数量仍存疑问。通过15个任务中两组模型对的960次会话，研究发现基于角色的代理评判者在Turing式验证中与人类评分无差异。随后识别出评分-覆盖度分离现象：质量评分随面板规模对数增长，而独特问题发现遵循亚线性幂律，两者均表现出递减回报，但评分饱和速度约为发现的两倍。假设这反映了发现空间的幂律分布：关键问题由小面板首先发现，而边缘案例需逐步更大面板，类似生态学中的物种积累曲线。机制源于群体多样性——Big Five性格条件使代理探索不同质量维度，专家评判者作为对抗性探针推动发现进入分布尾部。受控消融实验证实，结构化角色条件而非简单提示是产生这种缩放特性的必要条件。
英文摘要：
LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
