arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.18357cs.SE
收藏

人工智能功能正确性的统计置信度:一种AI产品功能正确性评估的方法

Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation

Pontifical Catholic University of Rio de Janeiro(里约热内卢天主教大学)

Wallace Albertini, Marina Condé Araújo, Júlia Condé Araújo, Antonio Pedro Santos Alves, Marcos Kalinowski

AI总结:

本文提出了一种基于统计置信度的方法,用于评估AI产品功能正确性,通过连接业务需求与模型性能的统计置信度,提供更稳健的评估手段。

摘要:

人工智能系统质量评估是一个根本性挑战,由于其本质上具有概率性。标准如ISO/IEC 25059提供了质量模型,但缺乏实际且统计上稳健的方法来评估功能正确性。本文提出并评估了统计置信度在功能正确性(SCFC)方法,旨在通过将业务需求连接到考虑模型平均性能和变异性的统计置信度度量来填补这一空白。该方法包含四个步骤:定义定量规范限,执行分层和概率抽样,应用自助法估计性能度量的置信区间,以及计算能力指数作为最终指标。该方法通过在工业中涉及与AI专家访谈的两个真实世界AI系统案例研究进行了评估。专家们收集了关于该方法的有用性、易用性和在实际场景中采用的意图的有价值的见解。我们得出结论,所提出的方法是将功能正确性评估从点估计转化为统计置信度陈述的一种可行且有价值的方法。

英文摘要:

The quality assessment of Artificial Intelligence (AI) systems is a fundamental challenge due to their inherently probabilistic nature. Standards such as ISO/IEC 25059 provide a quality model, but they lack practical and statistically robust methods for assessing functional correctness. This paper proposes and evaluates the Statistical Confidence in Functional Correctness (SCFC) approach, which seeks to fill this gap by connecting business requirements to a measure of statistical confidence that considers both the model's average performance and its variability. The approach consists of four steps: defining quantitative specification limits, performing stratified and probabilistic sampling, applying bootstrapping to estimate a confidence interval for the performance metric, and calculating a capability index as a final indicator. The approach was evaluated through a case study on two real-world AI systems in industry involving interviews with AI experts. Valuable insights were collected from the experts regarding the utility, ease of use, and intention to adopt the methodology in practical scenarios. We conclude that the proposed approach is a feasible and valuable way to operationalize the assessment of functional correctness, moving the evaluation from a point estimate to a statement of statistical confidence.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑