基于交互的 contextual Markov 决策过程学习与个性化反馈
Interaction-Grounded Learning for Contextual Markov Decision Processes with Personalized Feedback
AI总结：
本文提出了一种计算高效的算法，用于在具有个性化反馈的上下文性 Markov 决策过程中实现次线性遗憾保证，通过扩展奖励估计器构造并设计逆间隙加权算法，有效提升了多轮交互中学习个性化目标的能力。
摘要：
本文研究了交互导向学习（IGL）[Xie 等，2021]，一种为现实场景设计的范式，其中学习者接收由未知机制生成的间接反馈，而不是显式的数值奖励。尽管先前关于 IGL 的工作提供了高效的算法并具有可证明的保证，但这些结果仅限于单步设置，限制了其在现代顺序决策系统如多轮大型语言模型（LLM）部署中的应用。为了弥合这一差距，我们提出了一种计算高效的算法，该算法在具有个性化反馈的上下文性 episodic Markov 决策过程（MDPs）中实现了次线性遗憾保证。技术上，我们扩展了 Zhang 等[2024a]的奖励估计器构造，从单步扩展到多步设置，解决了在 MDPs 下解码潜在奖励的特殊挑战。基于此估计器，我们设计了逆间隙加权（IGW）算法用于策略优化。最后，我们通过在合成 episodic MDP 和现实世界用户预订数据集上的实验，展示了我们的方法在从多轮交互中学习个性化目标方面的有效性。
英文摘要：
In this paper, we study Interaction-Grounded Learning (IGL) [Xie et al., 2021], a paradigm designed for realistic scenarios where the learner receives indirect feedback generated by an unknown mechanism, rather than explicit numerical rewards. While prior work on IGL provides efficient algorithms with provable guarantees, those results are confined to single-step settings, restricting their applicability to modern sequential decision-making systems such as multi-turn Large Language Model (LLM) deployments. To bridge this gap, we propose a computationally efficient algorithm that achieves a sublinear regret guarantee for contextual episodic Markov Decision Processes (MDPs) with personalized feedback. Technically, we extend the reward-estimator construction of Zhang et al. [2024a] from the single-step to the multi-step setting, addressing the unique challenges of decoding latent rewards under MDPs. Building on this estimator, we design an Inverse-Gap-Weighting (IGW) algorithm for policy optimization. Finally, we demonstrate the effectiveness of our method in learning personalized objectives from multi-turn interactions through experiments on both a synthetic episodic MDP and a real-world user booking dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
