BenchBrowser:用于评估基准有效性证据的检索
BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
AI总结:
BenchBrowser通过检索20个基准测试集中的相关评估项目,帮助评估者诊断基准测试在内容有效性与收敛有效性上的不足,从而弥补实践者意图与实际测试内容之间的差距。
摘要:
语言模型基准测试是否真正测量了从业者所期望的内容?高层元数据过于粗略,无法传达基准测试的细致现实:一个“诗歌”基准可能从不测试俳句,而“遵循指令”基准通常测试任意混合的技能。这种不透明性使得验证与从业者目标的一致性变得费力,即使模型在未测试的用户兴趣方面表现不佳,也可能产生能力的幻觉。我们引入了BenchBrowser,一个检索器,能够在20个基准测试集上检索与自然语言使用案例相关的评估项目。通过一项由人类研究验证的高检索精度研究,BenchBrowser生成证据,帮助从业者诊断低内容有效性(能力方面覆盖范围狭窄)和低收敛有效性(测量相同能力时缺乏稳定排名)。BenchBrowser因此有助于量化实践者意图与基准测试实际测试内容之间的关键差距。
英文摘要:
Do language model benchmarks actually measure what practitioners intend them to ? High-level metadata is too coarse to convey the granular reality of benchmarks: a "poetry" benchmark may never test for haikus, while "instruction-following" benchmarks will often test for an arbitrary mix of skills. This opacity makes verifying alignment with practitioner goals a laborious process, risking an illusion of competence even when models fail on untested facets of user interests. We introduce BenchBrowser, a retriever that surfaces evaluation items relevant to natural language use cases over 20 benchmark suites. Validated by a human study confirming high retrieval precision, BenchBrowser generates evidence to help practitioners diagnose low content validity (narrow coverage of a capability's facets) and low convergent validity (lack of stable rankings when measuring the same capability). BenchBrowser, thus, helps quantify a critical gap between practitioner intent and what benchmarks actually test.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。