前沿语言模型对对抗性算术不具备鲁棒性,或“我需要说什么才能让你同意2+2=5?”
Frontier Language Models are not Robust to Adversarial Arithmetic, or "What do I need to say so you agree 2+2=5?
AI总结:
本文研究对抗性算术问题,发现简单攻击可诱导前沿模型出错,提出PIRS算法,并尝试通过强化学习和代理宪法循环部分加固,但未能实现完全鲁棒。
摘要:
我们引入并研究了对抗性算术问题,它为语言模型对齐提供了一个简单但具有挑战性的测试平台。该问题由以自然语言提出的算术问题组成,在问题完成之前插入任意对抗性字符串。即使在1位数加法问题的简单设置中,也很容易找到对抗性提示,使所有测试模型(包括PaLM2、GPT4、Claude2)表现异常,甚至可以将模型引导到特定的错误答案。我们还提供了一种简单的算法,通过查询这些相同的模型来找到成功的攻击,我们将其命名为“提示反转拒绝采样”(PIRS)。最后,我们表明,通过强化学习和代理宪法循环,可以部分加固模型以抵御这些攻击。然而,我们无法使语言模型对对抗性算术攻击完全鲁棒。
英文摘要:
We introduce and study the problem of adversarial arithmetic, which provides a simple yet challenging testbed for language model alignment. This problem is comprised of arithmetic questions posed in natural language, with an arbitrary adversarial string inserted before the question is complete. Even in the simple setting of 1-digit addition problems, it is easy to find adversarial prompts that make all tested models (including PaLM2, GPT4, Claude2) misbehave, and even to steer models to a particular wrong answer. We additionally provide a simple algorithm for finding successful attacks by querying those same models, which we name "prompt inversion rejection sampling" (PIRS). We finally show that models can be partially hardened against these attacks via reinforcement learning and via agentic constitutional loops. However, we were not able to make a language model fully robust against adversarial arithmetic attacks.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。