发表机构
Heinz College of Information Systems and Public Policy, Carnegie Mellon University; Harvard Business School（卡内基梅隆大学海因茨信息系统与公共政策学院; 哈佛商学院）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出一种离线元策略设计方法，在预期风险更新次数预算下最大化累积价值，通过动态规划选择更新调度，并揭示信噪比驱动更新频率与风险分配，实验验证了性能-风险权衡。
AI 中文摘要
随着新数据的到来，模型可以被重新训练，但部署每一个新版本都面临着用更差的策略替换好策略的风险。我们研究如何在未来候选策略训练之前规划策略更新（即元策略），以平衡改进的收益与性能回退的风险。我们的离线元策略在预期执行次数（即比被替换策略表现更差的更新次数）的预算约束下，最大化期望累积价值。我们从历史学习轨迹中估计可能切换的价值和风险，将更新调度表示为有向无环图中的一条路径，并使用动态规划选择调度。一项主导阶分析确定了策略改进的信噪比是更新频率、等待时间和风险分配的关键驱动因素：更清晰的改进支持更早、更频繁的更新，而噪声更大的改进则需要更长的等待或更大的风险支出。它们的渐近速率也揭示了随着时间推移，实现更高安全性的边际成本递减。在合成数据和临床试验数据上的实验展示了性能-风险权衡，并将我们的方法与替代基线进行了比较。
英文摘要
Models can be retrained as new data arrive, but deploying every new version risks replacing a good policy with a worse one. We study how to plan policy updates (i.e., meta-policy) before future candidates are trained, balancing the benefits of improvement against the risk of performance regression. Our offline meta-policy maximizes expected cumulative value subject to a budget on the expected number of updates that perform worse than the policies they replace. We estimate the value and risk of possible switches from historical learning trajectories, represent an update schedule as a path in a directed acyclic graph, and select a schedule using dynamic programming. A leading-order analysis identifies the signal-to-noise ratio of policy improvement as a key driver of update frequency, waiting times, and risk allocation: clearer improvements support earlier, more frequent updates, while noisier improvements call for longer waits or greater risk expenditure. Their asymptotic rates also reveal a diminishing marginal cost of achieving greater safety over time. Experiments on synthetic and clinical trial data illustrate the performance--risk tradeoff and compare our method with alternative baselines.
