arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2008.12333cs.LGcs.AIstat.ML
收藏

深度强化学习控制丙泊酚的意识丧失水平

Controlling Level of Unconsciousness by Titrating Propofol with Deep Reinforcement Learning

Picower Institute for Learning and Memory, Massachusetts Institute of Technology(麻省理工学院皮考尔学习与记忆研究所);Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology(麻省理工学院脑与认知科学系);Department of Anesthesiology, Critical Care and Pain Medicine, Massachusetts General Hospital(麻省总医院麻醉、重症监护与疼痛医学科)

Gabe Schamberg, Marcus Badgeley, Emery N. Brown

摘要:

强化学习(RL)可用于拟合从患者状态到药物方案的映射。先前的研究已经使用确定性和基于值的表格学习来从观察到的麻醉状态学习丙泊酚剂量。深度RL用深度神经网络取代了表格,并已用于从注册数据库中学习药物方案。在这里,我们执行深度RL的第一个应用程序在模拟环境中的麻醉剂剂量的闭环控制。我们使用交叉熵方法来训练深度神经网络,以将观察到的麻醉状态映射到输注固定剂量丙泊酚的概率。在测试过程中,我们实施了一个确定性的政策,将输液的概率转换为连续的输液速率。该模型在具有随机参数的模拟药代动力学/药效学模型上进行训练和测试,以确保对患者变异性的稳健性。深度RL代理的性能明显优于比例-积分-微分控制器(中位绝对性能误差为1.7% +/- 0.6和3.4% +/- 1.2)。对连续输入变量而不是表格建模提供了更鲁棒的模式识别,并利用了我们的先验领域知识。Deep RL学习了一个平滑的策略,对数据科学家和麻醉护理提供者都有自然的解释。

英文摘要:

Reinforcement Learning (RL) can be used to fit a mapping from patient state to a medication regimen. Prior studies have used deterministic and value-based tabular learning to learn a propofol dose from an observed anesthetic state. Deep RL replaces the table with a deep neural network and has been used to learn medication regimens from registry databases. Here we perform the first application of deep RL to closed-loop control of anesthetic dosing in a simulated environment. We use the cross-entropy method to train a deep neural network to map an observed anesthetic state to a probability of infusing a fixed propofol dosage. During testing, we implement a deterministic policy that transforms the probability of infusion to a continuous infusion rate. The model is trained and tested on simulated pharmacokinetic/pharmacodynamic models with randomized parameters to ensure robustness to patient variability. The deep RL agent significantly outperformed a proportional-integral-derivative controller (median absolute performance error 1.7% +/- 0.6 and 3.4% +/- 1.2). Modeling continuous input variables instead of a table affords more robust pattern recognition and utilizes our prior domain knowledge. Deep RL learned a smooth policy with a natural interpretation to data scientists and anesthesia care providers alike.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑