缓解交互式推荐中离线强化学习的马太效应
Alleviating Matthew Effect of Offline Reinforcement Learning in Interactive Recommendation
AI总结：
针对离线强化学习推荐中保守策略导致的马太效应，本文提出DORL方法，通过在高熵状态上放宽悲观限制并惩罚降低多样性的动作，在捕捉用户兴趣的同时缓解热门偏差。
摘要：
离线强化学习（RL）是一种无需与在线环境交互、直接从日志数据中离线学习策略的技术，已成为交互式推荐等决策过程中的优选方案。离线强化学习面临价值高估问题。为解决该问题，现有方法采用保守策略，例如将学习到的策略约束在行为策略附近，或惩罚很少被访问的状态-动作对。然而，将此类离线强化学习应用于推荐时，会引发严重的马太效应，即富者愈富、贫者愈贫，具体表现为推广热门物品或类别，同时压制不太受欢迎的物品或类别。这是实际推荐系统中一个必须解决的突出问题。本文旨在缓解基于离线强化学习的推荐中的马太效应。通过理论分析，我们发现现有方法的保守性无法有效追求用户的长期满意度。这启发我们添加一个惩罚项，以放宽对日志策略高熵状态的悲观限制，并间接惩罚导致状态多样性降低的动作。由此形成了本文的主要技术贡献：去偏的基于模型的离线强化学习（DORL）方法。实验表明，DORL不仅能很好地捕捉用户兴趣，还能缓解马太效应。代码实现见 https://github.com/chongminggao/DORL-codes。
英文摘要：
Offline reinforcement learning (RL), a technology that offline learns a policy from logged data without the need to interact with online environments, has become a favorable choice in decision-making processes like interactive recommendation. Offline RL faces the value overestimation problem. To address it, existing methods employ conservatism, e.g., by constraining the learned policy to be close to behavior policies or punishing the rarely visited state-action pairs. However, when applying such offline RL to recommendation, it will cause a severe Matthew effect, i.e., the rich get richer and the poor get poorer, by promoting popular items or categories while suppressing the less popular ones. It is a notorious issue that needs to be addressed in practical recommender systems. In this paper, we aim to alleviate the Matthew effect in offline RL-based recommendation. Through theoretical analyses, we find that the conservatism of existing methods fails in pursuing users' long-term satisfaction. It inspires us to add a penalty term to relax the pessimism on states with high entropy of the logging policy and indirectly penalizes actions leading to less diverse states. This leads to the main technical contribution of the work: Debiased model-based Offline RL (DORL) method. Experiments show that DORL not only captures user interests well but also alleviates the Matthew effect. The implementation is available via https://github.com/chongminggao/DORL-codes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
