arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.12974cs.CL
收藏

BenchmarkCards:面向大语言模型基准的标准化文档

BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks

University of Notre Dame(诺特尔大学);IBM Research Ireland(IBM爱尔兰研究院);IBM Research NY, USA(IBM纽约研究院)

Anna Sokol, Elizabeth Daly, Michael Hind, David Piorkowski, Xiangliang Zhang, Nuno Moniz, Nitesh Chawla

AI总结:

针对大语言模型基准难以选择且易被误用的问题,本文提出BenchmarkCards文档框架,通过标准化基准的目标、方法、数据来源和局限性等属性,简化基准选择并提升评估透明度。

摘要:

大语言模型(LLMs)是能够处理多种任务的强大工具。由于模型在不同领域表现出各异的能力,针对特定任务比较和选择合适的大语言模型需要系统化的评估方法。然而,面对众多可用选项,找到合适的基准十分困难。这种复杂性不仅增加了基准被误用和误解的风险,还要求大语言模型用户付出大量精力,以寻找最符合其特定需求的基准。为解决这些问题,我们引入了BenchmarkCards,这是一个直观且经过验证的文档框架,用于标准化关键基准属性,例如目标、方法、数据来源和局限性。通过涉及基准创建者和用户的用户研究,我们表明BenchmarkCards能够简化基准选择并提升透明度,从而促进在评估大语言模型时做出明智决策。数据与代码:https://github.com/SokolAnn/BenchmarkCards

英文摘要:

Large language models (LLMs) are powerful tools capable of handling diverse tasks. Comparing and selecting appropriate LLMs for specific tasks requires systematic evaluation methods, as models exhibit varying capabilities across different domains. However, finding suitable benchmarks is difficult given the many available options. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce \texttt{BenchmarkCards}, an intuitive and validated documentation framework that standardizes critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies involving benchmark creators and users, we show that \texttt{BenchmarkCards} can simplify benchmark selection and enhance transparency, facilitating informed decision-making in evaluating LLMs. Data & Code: https://github.com/SokolAnn/BenchmarkCards

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑