arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.18702cs.LG
收藏

有限供应下的离策略学习

Off-Policy Learning with Limited Supply

Keio University(Keio大学);Institute of Science Tokyo(东京科学研究所);Meiji University(Meiji大学);Yale University(Yale大学);LY Corporation(LY公司);Hanjuku-kaso, Co., Ltd.

Koichi Tanaka, Ren Kishimoto, Bushun Kawagishi, Yusuke Narita, Yasuo Yamamoto, Nobuyuki Shimizu, Yuta Saito

AI总结:

本文研究了在情境老虎机中受限供应下的离策略学习问题,提出了一种新的OPLS方法,通过考虑用户间的相对预期奖励来更高效地分配有限供应的物品,实验证明其在有限供应情境下的优越性。

摘要:

我们研究了情境老虎机中的离策略学习(OPL),这在推荐系统和在线广告等广泛的实际应用中起着关键作用。典型的OPL在情境老虎机中假设一个无约束环境,其中策略可以无限次选择同一物品。然而,在许多实际应用中,包括优惠券分配和电子商务,有限供应通过分布式优惠券的预算限制或产品库存限制来限制物品。在这些设置中,贪心地选择当前用户预期奖励最高的物品可能导致该物品的早期耗尽,使其无法为未来可能生成更高预期奖励的用户使用。因此,最优的无约束设置中的OPL方法在有限供应设置中可能变得次优。为了解决这个问题,我们提供了一个理论分析,显示传统贪心OPL方法可能无法最大化策略性能,并证明在有限供应设置中必须存在性能更优的策略。基于这一见解,我们引入了一种新的方法,称为有限供应下的离策略学习(OPLS)。与简单选择预期奖励最高的物品不同,OPLS关注相对预期奖励较高的物品,从而更有效地分配有限供应的物品。我们在合成和现实数据集上的实验证明,OPLS在具有有限供应的情境老虎机问题中优于现有的OPL方法。

英文摘要:

We study off-policy learning (OPL) in contextual bandits, which plays a key role in a wide range of real-world applications such as recommendation systems and online advertising. Typical OPL in contextual bandits assumes an unconstrained environment where a policy can select the same item infinitely. However, in many practical applications, including coupon allocation and e-commerce, limited supply constrains items through budget limits on distributed coupons or inventory restrictions on products. In these settings, greedily selecting the item with the highest expected reward for the current user may lead to early depletion of that item, making it unavailable for future users who could potentially generate higher expected rewards. As a result, OPL methods that are optimal in unconstrained settings may become suboptimal in limited supply settings. To address the issue, we provide a theoretical analysis showing that conventional greedy OPL approaches may fail to maximize the policy performance, and demonstrate that policies with superior performance must exist in limited supply settings. Based on this insight, we introduce a novel method called Off-Policy learning with Limited Supply (OPLS). Rather than simply selecting the item with the highest expected reward, OPLS focuses on items with relatively higher expected rewards compared to the other users, enabling more efficient allocation of items with limited supply. Our empirical results on both synthetic and real-world datasets show that OPLS outperforms existing OPL methods in contextual bandit problems with limited supply.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑