arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.18982cs.AI
收藏

SAVOIR:通过基于Shapley的奖励归因学习社交能力

SAVOIR: Learning Social Savoir-Faire via Shapley-based Reward Attribution

The University of Hong Kong(香港大学);Harbin Institute of Technology(哈尔滨工业大学);Harbin Institute of Technology, Shenzhen(哈尔滨工业大学(深圳))

Xiachong Feng, Yi Jiang, Xiaocheng Feng, Deyi Yin, Libo Qin, Yangfan Ye, Lei Huang, Weitao Ma, Yuxuan Gu, Chonghan Qin, Bing Qin, Lingpeng Kong

AI总结:

本文提出SAVOIR框架,基于合作博弈理论解决语言代理训练中的信用分配问题,通过预期效用转移和Shapley值实现公平奖励分配,实验显示其在SOTOPIA基准上达到新状态-of-the-art性能。

摘要:

社交智能,即在复杂人际互动中导航的能力,是语言代理面临的基本挑战。通过强化学习训练此类代理需要解决信用分配问题:确定个体发言对多轮对话结果的贡献。现有方法直接使用语言模型分配回合级奖励,产生具有回顾性且缺乏理论基础的归因。我们提出SAVOIR(ShApley Value fOr SocIal RL),一种基于合作博弈理论的新型原则性框架。我们的方法结合了两个互补原则:预期效用转移将归因从回顾性归因转向前瞻性估值,捕捉发言在促进有利未来轨迹中的战略潜力;Shapley值确保公平的信用分配,具有效率、对称性和边际性的公理保证。在SOTOPIA基准上的实验表明,SAVOIR在所有评估设置中均达到新的state-of-the-art性能,我们的7B模型匹配或超过包括GPT-4o和Claude-3.5-Sonnet在内的专有模型。值得注意的是,即使大型推理模型也持续表现不佳,表明社交智能需要不同于分析推理的定性能力。

英文摘要:

Social intelligence, the ability to navigate complex interpersonal interactions, presents a fundamental challenge for language agents. Training such agents via reinforcement learning requires solving the credit assignment problem: determining how individual utterances contribute to multi-turn dialogue outcomes. Existing approaches directly employ language models to distribute episode-level rewards, yielding attributions that are retrospective and lack theoretical grounding. We propose SAVOIR (ShApley Value fOr SocIal RL), a novel principled framework grounded in cooperative game theory. Our approach combines two complementary principles: expected utility shifts evaluation from retrospective attribution to prospective valuation, capturing an utterance's strategic potential for enabling favorable future trajectories; Shapley values ensure fair credit distribution with axiomatic guarantees of efficiency, symmetry, and marginality. Experiments on the SOTOPIA benchmark demonstrate that SAVOIR achieves new state-of-the-art performance across all evaluation settings, with our 7B model matching or exceeding proprietary models including GPT-4o and Claude-3.5-Sonnet. Notably, even large reasoning models consistently underperform, suggesting social intelligence requires qualitatively different capabilities than analytical reasoning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑