arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.14596cs.CLcs.AI
收藏

教授模型平衡抵制与接受说服

Teaching Models to Balance Resisting and Accepting Persuasion

UNC Chapel Hill(北卡罗来纳大学教堂山分校);Anthropic(Anthropic公司)

Elias Stengel-Eskin, Peter Hase, Mohit Bansal

AI总结:

针对大型语言模型易受对抗性说服且难接受有益说服的问题,提出利用多智能体递归对话树生成数据、通过偏好优化训练的PBT方法,平衡正负说服表现,提升多智能体辩论稳定性与综合性能。

摘要:

大型语言模型(LLMs)易受说服影响,当面对对抗性对话者时可能带来风险。我们迈出了防御模型免受说服的第一步,同时认为防御对抗性(即负面)说服只是问题的一半:模型还应能够接受有益(即正面)的说服以改进答案。我们证明,仅针对某一方面优化模型会导致在另一方面表现不佳。为平衡正面和负面说服,我们引入说服训练(Persuasion-Training,简称PBT),它利用多智能体递归对话树创建数据,并通过偏好优化训练模型在适当时候接受说服。PBT允许我们使用较小的7-8B模型之间对话生成的数据来训练更大的70B模型。此外,PBT持续提高对错误信息的抵抗力和对质疑的恢复力,同时在包含正面和负面说服的整体数据上取得最佳综合性能。关键是,我们表明PBT模型在两个领域( trivia 和常识问答)的多智能体辩论中是更好的队友。我们发现,没有PBT时,强弱模型对的性能不稳定,模型呈现答案的顺序决定团队获得强模型还是弱模型的性能;PBT带来更好、更稳定的结果和更少的顺序依赖性,强模型持续带动弱模型提升。

英文摘要:

Large language models (LLMs) are susceptible to persuasion, which can pose risks when models are faced with an adversarial interlocutor. We take a first step towards defending models against persuasion while also arguing that defense against adversarial (i.e. negative) persuasion is only half of the equation: models should also be able to accept beneficial (i.e. positive) persuasion to improve their answers. We show that optimizing models for only one side results in poor performance on the other. In order to balance positive and negative persuasion, we introduce Persuasion-Training (or PBT), which leverages multi-agent recursive dialogue trees to create data and trains models via preference optimization to accept persuasion when appropriate. PBT allows us to use data generated from dialogues between smaller 7-8B models for training much larger 70B models. Moreover, PBT consistently improves resistance to misinformation and resilience to being challenged while also resulting in the best overall performance on holistic data containing both positive and negative persuasion. Crucially, we show that PBT models are better teammates in multi-agent debates across two domains (trivia and commonsense QA). We find that without PBT, pairs of stronger and weaker models have unstable performance, with the order in which the models present their answers determining whether the team obtains the stronger or weaker model's performance. PBT leads to better and more stable results and less order dependence, with the stronger model consistently pulling the weaker one up.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑