arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2501.05591cs.LG
收藏

使用离线鲁棒强化学习进行会话级动态广告负载优化

Session-Level Dynamic Ad Load Optimization using Offline Robust Reinforcement Learning

Meta

Tao Liu, Qi Xu, Wei Shi, Zhigang Hua, Shuang Yang

AI总结:

针对会话级广告负载优化中的混杂偏差与分布偏移,论文提出离线鲁棒对决 DQN 框架,在数据集、真实广告数据及在线 A/B 测试中均取得显著收益。

摘要:

会话级动态广告负载优化旨在通过动态平衡用户体验质量与广告变现,在用户在线会话期间实时个性化所投放广告的密度和类型。传统的基于因果学习的方法面临关键技术挑战,尤其是在处理混杂偏差和分布偏移方面。本文中,我们开发了一种基于离线深度 Q 网络(DQN)的框架,可有效缓解动态系统中的混杂偏差,并且与最佳的基于因果学习的生产基线相比,实现了超过 80% 的离线收益。此外,为提高该框架针对意外分布偏移的鲁棒性,我们进一步通过一种新颖的离线鲁棒对决 DQN 方法增强该框架。随着扰动增强,该方法在多个 OpenAI-Gym 数据集上取得了更稳定的奖励,并在真实广告投放数据上带来额外 5% 的离线收益。该方法已部署到多个生产系统中,取得了显著的顶层收益。上线后的在线 A/B 测试显示,互动度与广告评分之间的权衡效率实现了两位数提升,显著增强了我们平台同时服务消费者和广告主的能力。

英文摘要:

Session-level dynamic ad load optimization aims to personalize the density and types of delivered advertisements in real time during a user's online session by dynamically balancing user experience quality and ad monetization. Traditional causal learning-based approaches struggle with key technical challenges, especially in handling confounding bias and distribution shifts. In this paper, we develop an offline deep Q-network (DQN)-based framework that effectively mitigates confounding bias in dynamic systems and demonstrates more than 80% offline gains compared to the best causal learning-based production baseline. Moreover, to improve the framework's robustness against unanticipated distribution shifts, we further enhance our framework with a novel offline robust dueling DQN approach. This approach achieves more stable rewards on multiple OpenAI-Gym datasets as perturbations increase, and provides an additional 5% offline gains on real-world ad delivery data. Deployed across multiple production systems, our approach has achieved outsized topline gains. Post-launch online A/B tests have shown double-digit improvements in the engagement-ad score trade-off efficiency, significantly enhancing our platform's capability to serve both consumers and advertisers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑