发表机构
Amazon Web Services（亚马逊网络服务公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
研究提出云原生评估即服务EaaS架构，将人工智能评估方法化为六个微服务。验证了方法的关键问题，如覆盖率、插补影响、漂移检测能力和公平性监测等。给出了相关实验结果，且与开源工具比较，表明该架构有创新性。
AI 中文摘要
我们展示了EaaS，一种云原生参考架构，它将人工智能评估方法作为六个无状态Kubernetes微服务来运行：具有有限样本校正自适应预测集的共形预测、校准评估、通过随机傅里叶特征近似最大均值差异进行漂移检测、使用自助置信区间进行公平性监测、基于有向无环图的管道编排器以及结果存储API。我们验证了四个关键方法问题。首先，经验覆盖率与K = 50个随机校准/测试分割中的边际共形保证一致，平均覆盖率在名义目标的1.4个百分点内。其次，所有四个MMLU答案令牌出现在前20个对数概率中，无需插补，10%的模拟插补产生的覆盖率影响小于1.5%。第三，RFF - MMD在中位数启发式带宽下对轻度和重度漂移实现了100%的检测能力，第一类错误在5 - 8.5%之间。第四，对UCI成人收入数据集的公平性监测揭示了按种族划分的显著人口统计学平等差异（DP差距 = 0.33），且跨连续批次的警报稳定。共形预测和校准服务在批量大小为100时达到了亚2毫秒的p99延迟；RFF - MMD需要约500毫秒，适用于定期批量监测。与四个开源工具的比较表明，据我们所知，目前没有平台结合了共形预测即服务、微服务分解和基于有向无环图的编排。
英文摘要
We present EaaS, a cloud-native reference architecture that operationalizes AI evaluation methods as six stateless Kubernetes microservices: conformal prediction with finite-sample-corrected Adaptive Prediction Sets, calibration assessment, drift detection via RFF-approximated Maximum Mean Discrepancy, fairness monitoring with bootstrap confidence intervals, a DAG-based pipeline orchestrator, and a result storage API. We validate four key methodological concerns. First, empirical coverage is consistent with the marginal conformal guarantee across K=50 random calibration/test splits, with mean coverage within 1.4 percentage points of the nominal target. Second, all four MMLU answer tokens appear in the top-20 logprobs with 0% imputation needed, and simulated imputation at 10% produces less than 1.5% coverage impact. Third, RFF-MMD achieves 100% detection power for mild and severe drift at the median heuristic bandwidth, with Type I error between 5-8.5%. Fourth, fairness monitoring on the UCI Adult Income dataset reveals significant demographic parity disparities by race (DP gap=0.33) with stable alerts across sequential batches. Conformal prediction and calibration services achieve sub-2ms p99 latency at batch size 100; RFF-MMD requires ~500ms suited for periodic batch monitoring. A comparison with four open-source tools suggests that, to the best of our knowledge, no current platform combines conformal-prediction-as-a-service, microservice decomposition, and DAG-based orchestration.
Comments23 pages, 15 figures, 12 tables
