arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2602.02283cs.LGstat.ML
收藏

基于选择模型的Q学习用于延迟反馈收益管理

Choice-Model-Assisted Q-learning for Delayed-Feedback Revenue Management

Massachusetts Institute of Technology(麻省理工学院)

Owen Shen, Patrick Jaillet

AI总结:

本文提出基于选择模型的Q学习方法,用于解决延迟反馈下的收益管理问题,通过部分世界模型提升决策鲁棒性并减少偏差。

摘要:

我们研究了具有延迟反馈的强化学习在收益管理中的应用,其中相当大的价值由在预订后几天内观察到的客户取消和修改决定。我们提出了选择模型辅助的强化学习:一个经过校准的离散选择模型被用作固定的部分世界模型,用于在决策时间推断学习目标的延迟部分。在固定模型部署模式下,我们证明了使用模型推断目标的表格Q学习会收敛到最优Q函数的O(ε/(1-γ))邻域,其中ε总结了部分模型误差,此外还有一个O(t^{-1/2})的采样项。在基于61619个酒店预订数据校准的模拟器中的实验显示:(i)在稳定状态下,与成熟度缓冲DQN基线没有统计上可检测的差异;(ii)在家庭参数变化下有积极效果,在10个变化场景中经过Holm-Bonferroni校正后,有5个场景显著收益,最高达12.4%;(iii)在结构性规格错误下持续退化,其中选择模型假设被违反(收益降低1.4-2.6%)。这些结果表明了部分行为模型在移位下提高鲁棒性的时机以及引入有害偏差的时机。

英文摘要:

We study reinforcement learning for revenue management with delayed feedback, where a substantial fraction of value is determined by customer cancellations and modifications observed days after booking. We propose \emph{choice-model-assisted RL}: a calibrated discrete choice model is used as a fixed partial world model to impute the delayed component of the learning target at decision time. In the fixed-model deployment regime, we prove that tabular Q-learning with model-imputed targets converges to an $O(\varepsilon/(1-γ))$ neighborhood of the optimal Q-function, where $\varepsilon$ summarizes partial-model error, with an additional $O(t^{-1/2})$ sampling term. Experiments in a simulator calibrated from 61{,}619 hotel bookings (1{,}088 independent runs) show: (i) no statistically detectable difference from a maturity-buffer DQN baseline in stationary settings; (ii) positive effects under in-family parameter shifts, with significant gains in 5 of 10 shift scenarios after Holm--Bonferroni correction (up to 12.4\%); and (iii) consistent degradation under structural misspecification, where the choice model assumptions are violated (1.4--2.6\% lower revenue). These results characterize when partial behavioral models improve robustness under shift and when they introduce harmful bias.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑