arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.24055cs.AIcs.SE
收藏

CIRCLE:从现实视角评估AI的框架

CIRCLE: A Framework for Evaluating AI from a Real-World Lens

Civitaas Insights;Bournemouth University(伯恩茅斯大学);Independent Researcher(独立研究者);Cohere Labs(Cohere实验室);National Research Council Canada(加拿大国家研究理事会);Intellect Frontier;University of Sydney(悉尼大学);National Physical Laboratory, UK(英国国家物理实验室);Université de Sherbrooke(谢布罗克大学);University of Washington(华盛顿大学);Principled AI;Humane Intelligence Non-profit(人智非营利组织);Virginia State University(弗吉尼亚州立大学);Humane Intelligence Public Benefit Corporation(人智公共利益公司)
Reva Schwartz, Carina Westling, Morgan Briggs, Marzieh Fadaee, Isar Nejadgholi, Matthew Holmes, Fariza Rashid, Maya Carlyle, Afaf Taïk, Kyra Wilson, Peter Dougl… 展开作者

Reva Schwartz, Carina Westling, Morgan Briggs, Marzieh Fadaee, Isar Nejadgholi, Matthew Holmes, Fariza Rashid, Maya Carlyle, Afaf Taïk, Kyra Wilson, Peter Douglas, Theodora Skeadas, Gabriella Waters, Rumman Chowdhury, Thiago Lacerda

AI总结:

本文提出CIRCLE框架,通过六个阶段生命周期方法,弥合模型性能指标与部署中AI实际结果之间的现实差距,提供系统性的实证证据。

摘要:

本文提出CIRCLE框架,通过六个阶段生命周期方法,弥合模型性能指标与部署中AI实际结果之间的现实差距,提供系统性的实证证据。当前方法如MLOps框架和AI模型基准提供详细的系统稳定性及模型能力洞察,但无法为AI堆栈外的决策者提供系统性的证据,说明这些系统在现实情境中的实际表现及其对组织的长期影响。CIRCLE通过正式化将堆栈外的 Stakeholder 关心转化为可测量信号,不同于局限于本地的参与式设计或往往回顾性的算法审计,CIRCLE提供了一种结构化的前瞻性协议,将情境敏感的定性洞察与可扩展的定量指标联系起来。通过整合现场测试、红队行动和纵向研究等方法,CIRCLE产生系统性的知识:在不同地点可比 yet 敏感于本地情境的证据。这反过来可以基于实际下游影响而非理论能力来实现治理。

英文摘要:

This study proposes CIRCLE, a six-stage, lifecycle-based framework to bridge the reality gap between model-centric performance metrics and AI system outcomes in deployment. Current approaches such as MLOps frameworks and AI model benchmarks offer detailed insights into system stability and model capabilities, but they do not provide decision makers outside the AI stack with systematic evidence of how these systems actually behave in real world contexts or affect their organizations over time. CIRCLE operationalizes the Validation phase of TEVV (Test, Evaluation, Verification, and Validation) by translating priorities of stakeholders outside the stack into measurable signals. Unlike participatory design which often remains localized, or algorithmic audits which are often retrospective, CIRCLE provides a structured, prospective protocol for linking context sensitive qualitative insights to scalable quantitative metrics. By integrating methods such as field testing, red teaming, and longitudinal studies into a coordinated pipeline, CIRCLE produces systematic knowledge; evidence that is comparable across sites yet sensitive to local context. This can enable governance based on materialized downstream effects rather than theoretical capabilities.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑