arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.06603cs.LG
收藏

临床强化学习中时间重采样的隐藏风险

The impact of temporal resampling on the safety of clinical reinforcement learning: A simulated trial

University College London(伦敦大学学院);University College London Hospitals NHS Foundation Trust(伦敦大学学院医院国家健康服务基金会信托)

Thomas Frost, Hrisheekesh Vaidya, Steve Harris

AI总结:

本文研究了时间重采样对临床强化学习性能的影响,发现重采样会降低模型表现,建议未来研究应优先使用自然临床决策时间的 dataset。

摘要:

强化学习(RL)是一种用于做出最优选择的人工智能。在医疗领域,研究人员通常使用离线RL(ORL),其中模型从回顾性观察数据中训练和评估。为了适应本质上不规则的临床记录,研究人员经常将数据重新采样到均匀的时间间隔内进行训练(称为分箱)。然而,离散化数据使模型面临临床场景的虚构表示,尤其是在决策时间不可预测的情况下。由于这些模型缺乏稳健的试验证据,我们选择通过使用30个虚拟1型糖尿病患者进行计算机仿真临床试验来进一步探索这一影响。该模拟器被修改以包含决策之间的随机间隔,并用于生成离线RL的训练数据集。我们训练了三种ORL算法,分别在未处理数据集和等效的10分钟、2小时和4小时间隔重采样数据集上进行训练。当部署回模拟环境时,发现时间重采样会使模型性能降低高达60%,4小时分箱导致所有代理的表现均劣于数据集的基线。对重采样数据的回顾性评估积极掩盖了这一影响,预测1.5-3倍更好的回报,而实际中代理的表现则较差。我们建议未来在该领域的研究应优先考虑具有自然临床决策时间的数据集,这可能是这些模型安全应用于患者护理之前必要的步骤。

英文摘要:

Reinforcement learning (RL) is a type of artificial intelligence for making optimal choices. In healthcare, researchers generally use offline RL (ORL), where models are trained and evaluated from retrospective observational data. To accommodate naturally irregular clinical records, researchers often resample the data into uniform time intervals before training (known as binning). However, binning may create counterfactual trajectories that do not represent clinical practice. We first quantified these distortions in patients receiving continuous insulin infusions in the MIMIC-IV intensive care database. Binning reversed the apparent causal order for over 70% of paired glucose measurements and insulin adjustments within the same interval, and frequently masked significant dysglycaemic events and merged distinct treatment decisions together. We then conducted an in silico clinical trial using 30 virtual patients with type 1 diabetes from the FDA-approved UVA/Padova simulator. The simulator was modified to include stochastic intervals between decisions and used to generate a training dataset for offline RL. We trained three ORL algorithms on both the unprocessed dataset and equivalent datasets resampled at 10-minute, 2-hour, and 4-hour intervals. When deployed back into the simulated environment, temporal resampling was found to reduce model performance by up to 60% relative to unprocessed data, with 4-hour binning causing all agents to perform worse than the dataset's baseline. Retrospective evaluation on resampled data actively obscured this effect, predicting 1.5-3x better returns than agents achieved in practice. We recommend that future research in this area prioritises datasets with natural clinical timings between decisions, which may be a necessary step before these models can be safely deployed into patient care.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑