arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.00322cs.CLcs.GT
收藏

在多轮多智能体博弈中演化多样化的红队语言模型

Evolving Diverse Red-team Language Models in Multi-round Multi-agent Games

Peking University(北京大学);Beijing University of Posts and Telecommunications(北京邮电大学)

Chengdong Ma, Ziran Yang, Hai Ci, Jun Gao, Minquan Gao, Xuehai Pan, Yaodong Yang

AI总结:

针对LLM红队测试中的模式崩溃问题,提出动态红队博弈(RTG)及游戏化红队求解器(GRTS),通过多样性度量缓解模式崩溃并保证近似纳什均衡收敛,实验证明其能探索多样隐式攻击,为可扩展毒性检测与安全对齐铺平道路。

摘要:

部署大型语言模型(LLM)的主要挑战是确保其无害性。红队可以通过攻击 LLM 来识别漏洞以实现安全。然而,当前的努力严重依赖单轮提示设计和针对固定蓝队的单边红队优化。这些静态方法导致生成多样性显著降低,即所谓的模式崩溃,使得难以发现日益复杂的人机交互中的潜在风险。在此,我们引入动态红队博弈(RTG)来全面分析红队与蓝队之间的多轮攻防交互。此外,我们开发了带有多样性度量的游戏化红队求解器(GRTS),以缓解模式崩溃,并在理论上保证近似纳什均衡的收敛,从而为双方产生更好的策略。实证结果表明,GRTS 探索了多样且隐式的攻击,以自适应地利用各种 LLM,超越了特定模式的约束。深刻地,我们揭示的红队任务的几何结构与陀螺假说相一致,证实了构建多样化 LLM 群体作为异构人类专家红队成员的有前途代理的必要性。这为 LLM 的可扩展毒性检测和安全对齐铺平了道路。

英文摘要:

The primary challenge in deploying Large Language Model (LLM) is ensuring its harmlessness. Red team can identify vulnerabilities by attacking LLM to attain safety. However, current efforts heavily rely on single-round prompt designs and unilateral red team optimizations against fixed blue teams. These static approaches lead to significant reductions in generation diversity, known as the mode collapse, which makes it difficult to discover the potential risks in the increasingly complex human-LLM interactions. Here we introduce dynamic Red Team Game (RTG) to comprehensively analyze the multi-round offensive and defensive interactions between red team and blue team. Furthermore, we develop a Gamified Red Team Solver (GRTS) with diversity measures to mitigate mode collapse and theoretically guarantee the convergence of approximate Nash equilibrium which results in better strategies for both teams. Empirical results demonstrate that GRTS explore diverse and implicit attacks to adaptively exploit various LLMs, surpassing the constraints of specific modes. Insightfully, the geometrical structure we unveil of the red team task aligns with the spinning top hypothesis, confirming the necessity of constructing a diverse LLM population as a promising proxy for heterogeneous human expert red-teamers. This paves the way for scalable toxicity detection and safe alignment for LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑