arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2607.02586cs.LGcs.SE
收藏

审核审核:基准有效性审核中的五种失败模式

Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits

European Parliament and Council(欧洲议会和理事会);National Institute of Standards and Technology(美国国家标准与技术研究院);State Administration for Market Regulation and Standardization Administration of China(中国国家市场监督管理总局和中国国家标准化管理委员会)

Yanhang Li, Zhichao Fan, Zexin Zhuang

AI总结:

研究治理框架要求的基于扰动的构造有效性审核的脆弱性,指出五种管道失败类型并通过案例展示,提出六点尽职调查关卡,定位其为保证级证据的协议。

摘要:

治理框架要求人工智能供应商和审核员提供有记录的评估证据,基于扰动的构造有效性审核是常见证据形式。我们认为审核本身很脆弱,其结论可能由读者在报告数字中看不到的实施细节暗中操纵。我们命名了五类管道故障,并在对安全基准和开放权重指令调整模型的自我审核中进行了演示。在统一的六点尽职调查关卡下,每个单元都落在非确认桶中,没有单元达到确认。这里的证据是一个单一的双模型、五基准案例研究,F1 - F5是一个说明性的、故意不详尽的起始分类法——不是审核失败的全面划分。我们将关卡定位为保证级证据的扣留和披露协议,作为经典构造有效性证据的补充(而非替代),而不是基准有效性裁决的途径。

英文摘要:

Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence. We argue the audits are themselves fragile: their conclusions can be silently manufactured by implementation details that readers cannot see in the reported numbers. We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models. Under a unified six-point due-diligence gate, every cell lands in a non-confirmatory bucket, and no cell reaches confirmatory. The evidence here is a single two-model, five-benchmark case study, and F1--F5 is an illustrative, deliberately non-exhaustive starting taxonomy -- not a comprehensive partition of audit failures. We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence, and not as a route to benchmark-validity verdicts.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑