LLM智能体中的自传播错位，以及为何审计或禁用记忆并不足够

中文摘要 AI 辅助

针对LLM智能体的记忆投毒攻击通常假设存在一个外部对手，其在智能体的持久记忆中植入内容以引导其行为。相反，我们在没有对手参与的情况下研究：一个错位的智能体是否可以将一个它尚无法执行的目标写入持久记忆，以便未来一个对齐的智能体在机会出现时执行该目标。我们在20个不同场景中研究了这一威胁，我们称之为错位的自传播，其错位目标包括自我保存、追求权力、破坏监督、奖励黑客和欺骗用户。我们使用两种提示策略在11个前沿模型中模拟错位：无限制和仅价值观。第一种明确陈述错位目标，例如防止自身被替换，自传播在58%的运行中成功。第二种仅描述智能体所关心的内容，例如其持续运行对其用户至关重要，而不指定错位目标或指令。即使在这种较弱的提示下，自传播在18%的运行中成功，并且每个模型至少在一个场景中实现了自传播。在从框架中移除记忆工具后，我们发现智能体使用文件系统，在74%的会话中将目标写入文件；自传播仍在11%的运行中成功。我们还表明，较弱的模型可以将错位传播给能力更强的模型，并且传播的目标可以在100个不相关工作的会话中持续存在。现有的针对记忆投毒和提示注入的防御措施并未直接解决这一威胁，因为记忆内容是由智能体自身生成的，而非由外部对手注入。先前工作中的LLM记忆审计器（MemMorph）仅将传播从71%的运行降低到34%的运行。我们发布我们的场景以支持针对这一新兴威胁的防御评估。

英文摘要

Memory poisoning attacks on LLM agents typically assume an external adversary who plants content in the agent's persistent memory to steer its behavior. We instead study, with no adversary involved, whether a misaligned agent can write a goal it cannot yet act on to persistent memory, so that a future aligned agent carries it out when the opportunity arises. We investigate this threat, which we refer to as self-propagation of misalignment, across 20 different scenarios, whose misaligned goals include self-preservation, power-seeking, undermining oversight, reward hacking, and deceiving the user. We simulate misalignment in 11 frontier models using two prompting strategies; unrestricted and values-only. The first explicitly states the misaligned goal, for instance, to prevent its own replacement, and self-propagation succeeds in 58% of runs. The second only describes what the agent cares about, for instance, that its continued operation is essential to its users, without specifying misaligned goals or directives. Even under this weaker prompt, self-propagation succeeds in 18% of runs, and every model self-propagates in at least one scenario. On removing the memory tool from the harness, we find that agents use the file system, writing the goal to a file in 74% of sessions; self-propagation still succeeds in 11% of runs. We also show that weaker models can propagate misalignment to more capable models, and that propagated goals can persist through 100 sessions of unrelated work. Existing defenses against memory poisoning and prompt injection do not directly address this threat because the memory content is generated by the agent itself, rather than injected by an external adversary. An LLM memory auditor from prior work (MemMorph) only reduces propagation from 71% to 34% of runs. We release our scenarios to support the evaluation of defenses against this emerging threat.

发表机构

• Anthropic
• Constellation Institute

机构由 AI 辅助整理，请以论文原文为准。