注意仿真到现实的差距并像科学家一样思考
Exploration and Experimentation with a Misspecified Simulator
AI总结:
本文研究了在仿真和现实之间如何补充实验以减少价值差距,提出了Fisher-SEP方法,并通过两个案例研究展示了其应用。
摘要:
假设有规划者拥有一个预先训练的序列决策问题的仿真器,并有机会在现实中进行实验。仿真器查询成本低,但继承了校准数据中的混杂因素和漂移。实验是无偏的,但每次试验消耗一个现实单位。我们研究了规划者何时以及如何补充仿真器进行实验。我们给出了三个结果。首先,扩展的仿真引理将仿真器的价值误差分解为校准-部署偏移,该偏移可以随机化识别,以及一个参数残差,无法通过进一步交互减少。第二,仿真器最优策略与最优解之间的价值差距分为局部部分,这部分在部署策略已访问的状态上,以及可达性部分,这部分在部署策略未访问的状态上。在纯被动学习下,可达性部分在任何时间范围内都保持远离零。第三,我们提出了Fisher-SEP,一种辅助仿真的实验策略(SEP),该策略最小化目标策略价值的后验预测方差,具有仅奖励和仅转换的特殊化版本。两个案例研究展示了这些制度。在自动售货机供应链中,前端实验在时间范围足够长以抵消试点成本后超过后验更新。在HIV移动测试示例中,有一个走廊将一个受监控区域与一个受监控较差的区域分开,只有设计的探索才能到达受监控较差的区域。
英文摘要:
Randomized experiments are the standard for measuring causal effects that guide decisions, but are costly. Simulators calibrated to historical data offer a cheaper alternative, but their policies can become suboptimal under drift, a change in dynamics after calibration. We study if, when, and how to collect data for recalibrating the simulator. We consider a system with multiple units running in parallel and managed by a single operator. The real-world system is a partially observed process with a latent variable, while the simulator models it as a fully observed Markovian process, and thus is misspecified. Two data collection strategies exploit the current policy: (i) never recalibrating and (ii) recalibrating from on-policy data (passive adaptation). Two others collect off-policy data by trying actions outside the current policy: (iii) continuous exploration across all units and (iv) randomized experimentation on a sample of units over a predefined window. We investigate these strategies theoretically and in a vending-machine simulation. Experiments can improve on passive adaptation when drift is invisible in on-policy data or recalibration lags behind continuing drift. Experimentation is preferable to exploration when trying a worse action is expensive and the dynamics may not have changed.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。