arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.07002cs.AImath.STstat.MLstat.TH
收藏

具有任何时间有效保证的AI系统自适应审计

Adaptive auditing of AI systems with anytime-valid guarantees

University of California, San Francisco(加州大学旧金山分校);Stanford University(斯坦福大学)

Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman, Yifan Mai, Jean Feng

AI总结:

本文提出一种自适应审计框架,通过双视角假设检验方法,在有限样本下实现统计严谨的AI系统鲁棒性验证,证明在严格审计下系统可被认证为全局鲁棒。

摘要:

在生成AI系统故障模式表征中,标注和评估的成本和时间是一个主要瓶颈。因此,自适应测试范式受到青睐,其中根据过往结果机会性决定标注案例和数量。尽管该框架高度实用,但其极端灵活性使得难以得出统计严谨的结论,因为它违反了经典假设:观测数量通常有限(通常10到50个案例),采样和停止决策是在数据收集过程中而非基于预定义规则。为表征从高度自适应审计中可以得出什么统计推论,我们引入了两种'对立'视角的假设检验框架:(i)模型的零假设,即不存在性能低于目标阈值的故障模式;(ii)审计员的零假设,即审计员有采样策略能发现故障模式。利用Safe Anytime-Valid Inference(SAVI),我们将审计员视为进行'测试赌局',这转化为同时进行e-processes以检验对立的零假设。此外,如果审计员足够强大,我们证明这两个假设在渐近意义上是互为逆的,即通过严格审计确实可认证AI系统为全局鲁棒。实证表明,我们提出的方法在有限样本下维持任何时间有效的一类错误控制,优于预定义测试方法,并且有时在仅20个观测下即可得出统计严谨的结论。

英文摘要:

A major bottleneck in characterizing the failure modes of generative AI systems is the cost and time of annotation and evaluation. Consequently, adaptive testing paradigms have gained popularity, where one opportunistically decides which cases and how many to annotate based on past results. While this framework is highly practical, its extreme flexibility makes it difficult to draw statistically rigorous conclusions, as it violates classical assumptions: the number of observations is typically limited (often 10 to 50 cases) and decisions regarding sampling and stopping are made in the midst of data collection rather than based a pre-specified rule. To characterize what statistical inferences can be drawn from highly adaptive audits, we introduce a hypothesis testing framework from two "dueling" perspectives: (i) the model's null that asserts there is no failure mode with performance below a target threshold versus (ii) the auditor's null that asserts they have a sampling strategy that will uncover a failure mode. Leveraging Safe Anytime-Valid Inference (SAVI), we formalize the auditor as conducting "testing by betting," which translates into simultaneous e-processes for testing the dueling null hypotheses. Furthermore, if the auditor is sufficiently powerful, we prove that these two hypotheses are asymptotically inverses of each other, in that passage of such an audit certifies the AI system as being globally robust. Empirically, we demonstrate that our proposed testing procedures maintain anytime-valid Type-I error control, outperform pre-specified testing methods, and can reach statistically rigorous conclusions sometimes with as few as 20 observations.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑