arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.11944cs.LGcs.CY
收藏

利用预测多重性衡量AI法案中的个体表现

Using predictive multiplicity to measure individual performance within the AI Act

University of Tübingen and Tübingen AI Center(图宾根大学和图宾根人工智能中心);University of Tübingen and CZS Institute for Artificial Intelligence and Law(图宾根大学和图宾根人工智能与法律研究所);University of Tübingen, Tübingen AI Center and CZS Institute for Artificial Intelligence and Law(图宾根大学、图宾根人工智能中心和图宾根人工智能与法律研究所)

Karolin Frohnapfel, Mara Seyfert, Sebastian Bordt, Ulrike von Luxburg, Kristof Meding

AI总结:

本文探讨AI决策中的预测多重性现象,分析其与欧盟AI法案的冲突,并提出基于个体冲突比率和δ-模糊性的量化方法及实践建议。

摘要:

在构建用于决策支持的AI系统时,常会遇到预测多重性现象:不存在单一最佳模型;相反,可以构建许多总体准确率相似但在个别案例预测上不同的模型。尤其是当决策直接影响人类时,这可能会非常令人不满。对于模型间高度不一致的个人,本可以选择另一个总体准确率相似的模型,该模型会对该个人的案例做出不同决定。我们认为这种任意性与欧盟AI法案相冲突,该法案要求高风险AI系统的提供者不仅要在数据集层面报告性能,还要针对特定个人报告性能。本文旨在将预测多重性与欧盟AI法案关于准确性的规定联系起来,并随后提出关于如何在实践中评估和报告预测多重性的具体建议。具体来说:(1) 我们介绍AI法案的准确性规定,并论证纳入预测多重性信息可以有助于提供者遵守特定规定。(2) 基于这一法律严谨的分析,我们提出个体冲突比率和δ-模糊性作为量化模型间在个别案例上不一致性的工具,并帮助检测受冲突预测影响的个体。(3) 基于计算洞察,我们推导出易于实施的规则,说明模型提供者如何在实践中评估预测多重性。(4) 最终,我们建议在AI法案下,应向部署者提供关于预测多重性的信息,使他们能够判断系统对特定个体的输出是否足够可靠以用于其用例。

英文摘要:

When building AI systems for decision support, one often encounters the phenomenon of predictive multiplicity: a single best model does not exist; instead, one can construct many models with similar overall accuracy that differ in their predictions for individual cases. Especially when decisions have a direct impact on humans, this can be highly unsatisfactory. For a person subject to high disagreement between models, one could as well have chosen a different model of similar overall accuracy that would have decided the person's case differently. We argue that this arbitrariness conflicts with the EU AI Act, which requires providers of high-risk AI systems to report performance not only at the dataset level but also for specific persons. The goal of this paper is to put predictive multiplicity in context with the EU AI Act's provisions on accuracy and to subsequently derive concrete suggestions on how to evaluate and report predictive multiplicity in practice. Specifically: (1) We introduce the AI Act's accuracy provisions and argue that incorporating information about predictive multiplicity could serve compliance with specific provisions for providers. (2) Based on this legally rigorous analysis, we suggest individual conflict ratios and $δ$-ambiguity as tools to quantify the disagreement between models on individual cases and to help detect individuals subject to conflicting predictions. (3) Based on computational insights, we derive easy-to-implement rules on how model providers could evaluate predictive multiplicity in practice. (4) Ultimately, we suggest that information about predictive multiplicity should be made available to deployers under the AI Act, enabling them to judge whether system outputs for specific individuals are reliable enough for their use case.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑