arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2205.04797cs.IR
收藏

推荐强化学习中的状态编码器:一个可复制性研究

State Encoders in Reinforcement Learning for Recommendation: A Reproducibility Study

Jin Huang, Harrie Oosterhuis, Bunyamin Cetinkaya, Thijs Rood, Maarten de Rijke

摘要:

用于推荐的强化学习(RL 4 Rec)方法越来越受到关注,因为它们可以快速适应用户反馈。一个典型的RL 4 Rec框架包括(1)一个状态编码器,用于对存储用户历史交互的状态进行编码,以及(2)一个RL方法,用于采取行动并观察奖励。先前的工作比较了四个状态编码器的环境中,用户反馈是基于真实世界的记录用户数据模拟。基于注意力的状态编码器被认为是最佳的选择,因为它达到了最高的性能。然而,这一发现是有限的行动者-评论家的方法,四个状态编码器,和评估模拟器,不去偏见记录的用户数据。针对这些缺点,我们复制并扩展了现有的基于注意力的状态编码器的比较(1)在公开可用的去偏RL 4 Rec SOFA模拟器中(2)不同的RL方法,(3)更多的状态编码器,(4)不同的数据集。重要的是,我们的实验结果表明,与更多的状态编码器相比,现有的研究结果并不能推广到从不同的数据集和基于深度Q网络(DQN)的方法生成的去偏SOFA模拟器。

英文摘要:

Methods for reinforcement learning for recommendation (RL4Rec) are increasingly receiving attention as they can quickly adapt to user feedback. A typical RL4Rec framework consists of (1) a state encoder to encode the state that stores the users' historical interactions, and (2) an RL method to take actions and observe rewards. Prior work compared four state encoders in an environment where user feedback is simulated based on real-world logged user data. An attention-based state encoder was found to be the optimal choice as it reached the highest performance. However, this finding is limited to the actor-critic method, four state encoders, and evaluation-simulators that do not debias logged user data. In response to these shortcomings, we reproduce and expand on the existing comparison of attention-based state encoders (1) in the publicly available debiased RL4Rec SOFA simulator with (2) a different RL method, (3) more state encoders, and (4) a different dataset. Importantly, our experimental results indicate that existing findings do not generalize to the debiased SOFA simulator generated from a different dataset and a Deep Q-Network (DQN)-based method when compared with more state encoders.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑