arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.04062cs.LG
收藏

评估事实表:一种用于记录AI评估的结构化框架

Eval Factsheets: A Structured Framework for Documenting AI Evaluations

FAIR at Meta(Meta的FAIR)

Florian Bordes, Candace Ross, Justine T Kao, Evangelia Spiliopoulou, Adina Williams

AI总结:

本文提出Eval Factsheets,一种结构化框架,用于系统记录AI评估方法,通过分类法和问卷方法提升评估的透明度和可重复性。

摘要:

人工智能评估方法的迅速普及带来了可重复性、透明性和知情决策方面的重大挑战。然而,与受益于结构化文档框架(如Datasheets和Model Cards)的数据集和模型不同,评估方法缺乏系统化的文档标准。我们介绍了Eval Factsheets,一种结构化的描述性框架,通过全面的分类法和问卷方法来记录AI系统评估。我们的框架将评估特征分为五个基本维度:情境(谁进行了评估以及何时?)、范围(评估什么?)、结构(评估是如何构建的?)、方法(它如何运作?)和对齐(以何种方式它是可靠/有效/稳健的?)。我们将这一分类法实现为一个实用的问卷,涵盖五个部分,包含必填和推荐的文档元素。通过在多个基准上的案例研究,我们证明Eval Factsheets能够有效捕捉多样化的评估范式——从传统基准到LLM-as-judge方法论——同时保持一致性和可比性。我们希望Eval Factsheets能被纳入现有的和新发布的评估框架中,从而促进更多的透明度和可重复性。

英文摘要:

The rapid proliferation of benchmarks has created significant challenges in reproducibility, transparency, and informed decision-making. However, unlike datasets and models -- which benefit from structured documentation frameworks like Datasheets and Model Cards -- evaluation methodologies lack systematic documentation standards. We introduce Eval Factsheets, a structured, descriptive framework for documenting AI system evaluations through a comprehensive taxonomy and questionnaire-based approach. Our framework organizes evaluation characteristics across five fundamental dimensions: Context (Who made the evaluation and when?), Scope (What does it evaluate?), Structure (With what the evaluation is built?), Method (How does it work?) and Alignment (In what ways is it reliable/valid/robust?). We implement this taxonomy as a practical questionnaire spanning five sections with mandatory and recommended documentation elements. Through case studies on multiple benchmarks, we demonstrate that Eval Factsheets effectively captures diverse evaluation paradigms -- from traditional benchmarks to LLM-as-judge methodologies -- while maintaining consistency and comparability. We hope Eval Factsheets are incorporated into both existing and newly released evaluation frameworks and lead to more transparency and reproducibility.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑