RLRF：基于排序器反馈强化学习的竞争性搜索智能体设计
RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback
AI总结：
针对竞争性搜索场景下LLM生成修改内容的需求，提出基于排序竞赛偏好数据训练的RLRF框架，其智能体在分布外场景和策略对手下均优于现有方法，展现强化学习在该领域的潜力。
摘要：
竞争性搜索是指文档发布者修改文档以提升其在查询结果中排名的场景。近年来，发布者越来越多地利用大语言模型（LLM）生成和修改竞争性内容。我们提出了排序器反馈强化学习（Reinforcement Learning from Ranker Feedback, RLRF）框架，该框架使用从排名竞赛中得到的偏好数据集训练LLM。基于LLM的发布者智能体的目标是优化内容以提升排名，同时考虑竞争智能体的策略。我们采用不依赖人工撰写数据的方法生成数据集。实验表明，我们提出的智能体在基于LLM的竞争性文档修改任务中，始终显著优于此前提出的方法。我们进一步证明，这些智能体在未训练过的排序函数（即分布外场景）下依然有效，且能适应策略性对手。这些发现证实了强化学习在竞争性搜索中具有巨大潜力。
英文摘要：
Competitive search is a setting where document publishers modify them to improve their ranking in response to a query. Recently, publishers have increasingly leveraged LLMs to generate and modify competitive content. We introduce Reinforcement Learning from Ranker Feedback (RLRF), a framework that trains LLMs using preference datasets derived from ranking competitions. The goal of a publisher (LLM-based) agent is to optimize content for improved ranking while accounting for the strategies of competing agents. We generate the datasets using approaches that do not rely on human-authored data. We show that our proposed agents consistently and substantially outperform previously suggested approaches for LLM-based competitive document modification. We further show that our agents are effective with ranking functions they were not trained for (i.e., out of distribution) and they adapt to strategic opponents. These findings provide support to the significant potential of using reinforcement learning in competitive search.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
