发表机构
Microsoft（微软公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究提出AdsWorldEngine对话式广告框架，通过协调器与工具协同演化的自改进循环，结合机会门、评估器及基于标签的判断建模，在离线与在线实验中均显著提升广告性能。
AI 中文摘要
对话式广告旨在在多轮助手交互中推送有用的广告。与用户意图通常以简短独立查询表达的传统基于查询的广告不同，对话式广告必须从当前用户查询、助手响应及对话历史中推断潜在商业意图，同时判断广告是否有帮助而非具有侵入性。我们提出AdsWorldEngine，一种用于对话式广告的智能体框架。AdsWorldEngine包含机会门（Opportunity Gate，用于确定是否应展示广告）、协调器（Orchestrator，用于生成商业意图、调用广告工具并构建排名前三的广告列表）、评估器（Evaluator，用于对推送的广告评分以进行离线优化）。核心贡献是迭代式智能体-工具训练流程：我们首先通过监督微调与智能体强化学习训练协调器，再利用高、低奖励的rollout结果构建偏好数据以训练工具，由此形成自改进循环，使系统不仅学习如何使用广告工具，还能从受奖励的行为中学习如何改进工具。为支持主观生产决策，我们引入基于标签的判断建模，该模型从明确指南下收集的人工标签训练判断模型，通过思考轨迹丰富标签，通过反思过滤不一致的理由，并进一步优化二元判断，采用保留非对称奖励差距的成本敏感型GRPO变体。离线实验显示，AdsWorldEngine与当前生产环境中的广告推送系统相比，多样性提升60%，相关性提升80%；在线A/B测试中，其提升RPM达22%，广告覆盖率提升74%。
英文摘要
Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user's intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad delivery system. In an online A/B test, it increases RPM by 22% and ads coverage by 74%.
