通过可验证奖励的强化学习指导大语言模型进行谈判
Instructing LLMs to Negotiate using Reinforcement Learning with Verifiable Rewards
AI总结：
本文探讨了通过可验证奖励的强化学习训练大语言模型进行谈判的有效性，揭示了四阶段战略演化，展示了训练后的模型在谈判中提取剩余价值的能力。
摘要：
最近大语言模型（LLMs）的发展确立了其作为自主交互代理的潜力。然而，它们在不完全信息战略游戏中，如双边价格谈判中常表现不佳。本文研究了通过可验证奖励的强化学习（RLVR）是否能有效训练LLMs进行谈判。具体而言，我们探索了学习过程中出现的战略行为。我们引入了一个框架，训练了一个中等规模的买方代理与受监管的LLM卖方在广泛现实产品分布中进行博弈。通过将奖励信号直接与经济剩余最大化和严格的私人预算约束相结合，我们揭示了一种新的四阶段战略演化。代理从幼稚的讨价还价发展到使用激进的起始价格，经历僵局阶段，最终发展出复杂的说服技能。我们的结果表明，这种可验证训练使30B模型在提取剩余价值方面显著优于十倍于其规模的前沿模型。此外，训练后的代理在面对训练中未见过的更强对手时仍能保持稳健的泛化能力，并在面对敌对、对抗性卖方身份时仍有效。
英文摘要：
The recent advancement of Large Language Models (LLMs) has established their potential as autonomous interactive agents. However, they often struggle in strategic games of incomplete information, such as bilateral price negotiation. In this paper, we investigate if Reinforcement Learning from Verifiable Rewards (RLVR) can effectively teach LLMs to negotiate. Specifically, we explore the strategic behaviors that emerge during the learning process. We introduce a framework that trains a mid-sized buyer agent against a regulated LLM seller across a wide distribution of real-world products. By grounding reward signals directly in the maximization of economic surplus and strict adherence to private budget constraints, we reveal a novel four-phase strategic evolution. The agent progresses from naive bargaining to using aggressive starting prices, moves through a phase of deadlock, and ultimately develops sophisticated persuasive skills. Our results demonstrate that this verifiable training allows a 30B agent to significantly outperform frontier models over ten times its size in extracting surplus. Furthermore, the trained agent generalizes robustly to stronger counterparties unseen during training and remains effective even when facing hostile, adversarial seller personas.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
