arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.17409cs.IR
收藏

Markov推荐过程中的价值函数分解

Value Function Decomposition in Markov Recommendation Process

Xiaobei Wang, Shuchang Liu, Qingpeng Cai, Xiang Li, Lantao Hu, Han li, Guangming Xie

AI总结:

针对推荐系统中TD学习未分别建模策略与环境随机性导致的长期奖励估计次优问题,提出通过分解TD损失解耦两类随机因子的方法,实现更准确快速的估计并提升探索鲁棒性。

摘要:

推荐系统的最新进展表明,用户与系统的交互本质上构成了长期优化问题,并且可以采用在线强化学习来提升推荐性能。一般的解决方案框架包含一个价值函数,该函数估计用户未来的期望累积奖励并指导推荐策略的训练。为了避免局部最优,策略可以在推理期间探索潜在的高质量动作,以增加找到更好未来奖励的机会。为了适应逐步推荐过程,学习价值函数的一种广泛采用的方法是从用户两个连续状态值之间的差异中进行学习。然而,我们认为这种范式涉及混合随机因子的挑战:来自随机策略和不确定用户环境的两个随机因子在标准时序差分(TD)学习中没有被单独建模,这可能导致对长期奖励的次优估计和较低效的动作探索。作为解决方案,我们展示了通过分解原始时序差分损失可以分别近似这两个因子。这种解耦学习框架能够实现更准确的估计,具有更快的学习速度,并提升了对动作探索的鲁棒性。作为对我们提出方法的经验验证,我们在基于公共数据集构建的模拟在线环境中进行了离线实验。

英文摘要:

Recent advances in recommender systems have shown that user-system interaction essentially formulates long-term optimization problems, and online reinforcement learning can be adopted to improve recommendation performance. The general solution framework incorporates a value function that estimates the user's expected cumulative rewards in the future and guides the training of the recommendation policy. To avoid local maxima, the policy may explore potential high-quality actions during inference to increase the chance of finding better future rewards. To accommodate the stepwise recommendation process, one widely adopted approach to learning the value function is learning from the difference between the values of two consecutive states of a user. However, we argue that this paradigm involves a challenge of Mixing Random Factors: there exist two random factors from the stochastic policy and the uncertain user environment, but they are not separately modeled in the standard temporal difference (TD) learning, which may result in a suboptimal estimation of the long-term rewards and less effective action exploration. As a solution, we show that these two factors can be separately approximated by decomposing the original temporal difference loss. The disentangled learning framework can achieve a more accurate estimation with faster learning and improved robustness against action exploration. As an empirical verification of our proposed method, we conduct offline experiments with simulated online environments built on the basis of public datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑