发表机构
Singapore University of Technology and Design（新加坡科技设计大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
RankCert提出一种鲁棒决策认证方法，在结构不确定性下仅当多个指标一致时认证AI导师选择策略，否则弃权，实验表明其降低决策损失但选择性风险未显著改善。
AI 中文摘要
当预测能力充分的学习者模型暗示不同的策略排名时，基于模拟的导师选择可能不稳定。RankCert仅在模型平均效用、概率最优、后验遗憾、跨域排名、家族覆盖以及留一域外和留一可见家族外平均值支持同一候选者时，认证八个等预算辅导策略之一；否则它弃权（不执行）。我们在1,280个冻结的留出设置中评估了RankCert，这些设置涵盖五个轮换的留出预言机家族，每个家族64个场景，以及四种队列规模。校准使用了获得许可的去标识化EdNet-KT1衍生数据，包含5,000名学习者和590,056条保留响应；所有五个家族代表均通过了冻结充分性门槛。最小域平均成对top-1一致性为0.272917（95%置信区间[0.253646, 0.293229]），显示出显著的结构性分歧。队列噪声方差从n=30到n=300减小，而结构家族份额保持非零。相对于全覆盖点选择，RankCert将总留出决策损失减少了0.006605个归一化结果单位（95%置信区间[0.004859, 0.008407]）。然而，在可比较的覆盖率下，相对于置信门控点证书，它并未降低选择性风险（差异-0.000213；95%置信区间[-0.003238, 0.002384]；Holm p=0.929654）。认证发生在3.75%的设置中，且仅在稳定场景中；RankCert在每个模糊、错误指定和结构冲突设置中均弃权（不执行）。“安全”仅指在声明效用和不确定性集合下的基准范围决策认证；不提出任何关于人类学习、因果性、部署有效性或一般安全性的主张。
英文摘要
Simulation-based tutor selection can be unstable when predictively adequate learner models imply different policy rankings. RankCert certifies one of eight equal-budget tutoring policies only when model-averaged utility, probability-best, posterior regret, cross-domain rank, family coverage, and leave-one-domain-out and leave-one-visible-family-out averages support the same candidate; otherwise it abstains. We evaluated RankCert in 1,280 frozen held-out settings spanning five rotating held-out oracle families, 64 scenarios per family, and four cohort sizes. Calibration used a licensed, de-identified EdNet-KT1 derivative with 5,000 learners and 590,056 retained responses; all five family representatives passed the frozen adequacy gate. Minimum-domain mean pairwise top-1 agreement was 0.272917 (95% CI [0.253646, 0.293229]), showing substantial structural disagreement. Cohort-noise variance decreased from n = 30 to n = 300, while the structural family share remained nonzero. RankCert reduced total held-out decision loss relative to full-coverage point selection by 0.006605 normalized-outcome units (95% CI [0.004859, 0.008407]). At comparable coverage, however, it did not reduce selective risk relative to a confidence-gated point certificate (difference -0.000213; 95% CI [-0.003238, 0.002384]; Holm p = 0.929654). Certification occurred in 3.75% of settings and only in stable scenarios; RankCert abstained in every ambiguous, misspecified, and structural-conflict setting. "Safe" denotes only benchmark-scoped decision certification under the declared utility and uncertainty set; no human-learning, causal, deployment-effectiveness, or general-safety claim is made.
Comments19 pages, 5 figures, 7 tables. Submitted to ACM Transactions on Intelligent Systems and Technology (TIST)
