面向自动出价的轨迹级迭代强化学习框架
Trajectory-wise Iterative Reinforcement Learning Framework for Auto-bidding
AI总结：
针对自动出价中仿真到在线部署的性能差距，本文提出迭代离线RL框架下的TEE轨迹级数据收集利用方法及SEAS安全探索机制，并在阿里巴巴展示广告平台验证有效。
摘要：
在在线广告中，广告主参与广告拍卖以获取广告机会，并且通常利用需求方平台（DSP）提供的自动出价工具。当前的自动出价算法通常采用强化学习（RL）。然而，出于安全考虑，大多数基于RL的自动出价策略都是在仿真环境中训练的，这导致其部署到在线环境时性能下降。为缩小这一差距，我们可以并行部署多个自动出价智能体，以收集大规模交互数据集。随后可利用离线RL算法训练新策略。训练后的策略又可被部署以进一步收集数据，从而形成一个迭代训练框架，我们称之为迭代离线RL。在本文中，我们识别出该迭代离线RL框架的性能瓶颈，其根源在于离线RL算法内在的保守性所导致的无效探索与利用。为克服这一瓶颈，我们提出了轨迹级探索与利用（TEE），它从轨迹视角为迭代离线RL引入了一种新颖的数据收集与数据利用方法。此外，为在保证TEE在线探索安全性的同时保持数据集质量，我们提出了基于自适应动作选择的安全探索（SEAS）。在阿里巴巴展示广告平台上进行的离线实验和真实世界实验均证明了所提方法的有效性。
英文摘要：
In online advertising, advertisers participate in ad auctions to acquire ad opportunities, often by utilizing auto-bidding tools provided by demand-side platforms (DSPs). The current auto-bidding algorithms typically employ reinforcement learning (RL). However, due to safety concerns, most RL-based auto-bidding policies are trained in simulation, leading to a performance degradation when deployed in online environments. To narrow this gap, we can deploy multiple auto-bidding agents in parallel to collect a large interaction dataset. Offline RL algorithms can then be utilized to train a new policy. The trained policy can subsequently be deployed for further data collection, resulting in an iterative training framework, which we refer to as iterative offline RL. In this work, we identify the performance bottleneck of this iterative offline RL framework, which originates from the ineffective exploration and exploitation caused by the inherent conservatism of offline RL algorithms. To overcome this bottleneck, we propose Trajectory-wise Exploration and Exploitation (TEE), which introduces a novel data collecting and data utilization method for iterative offline RL from a trajectory perspective. Furthermore, to ensure the safety of online exploration while preserving the dataset quality for TEE, we propose Safe Exploration by Adaptive Action Selection (SEAS). Both offline experiments and real-world experiments on Alibaba display advertising platform demonstrate the effectiveness of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
