Nautilus Compass:生产LLM代理的黑盒人格漂移检测
Nautilus Compass: Black-box Persona Drift Detection for Production LLM Agents
AI总结:
Nautilus Compass通过提示文本层的余弦相似度检测生产编码代理的人格漂移,无需调用LLM提取事实,实现高效且低成本的代理记忆层。
摘要:
生产LLM编码代理在长时间会话中会发生漂移:它们忘记用户指定的约束,陷入用户已标记的错误,并编造先前协议。白盒方法如人格向量需要模型权重,无法应用于封闭API(如Claude、GPT-4)。我们提出了Nautilus Compass,一种针对生产编码代理的黑盒人格漂移检测器和代理记忆层。该方法完全在提示文本层操作:通过BGE-m3嵌入的加权top-k均值计算用户提示与行为锚文本之间的余弦相似度。Compass是目前已知的唯一公开代理记忆层(在Mem0、Letta、Cognee、Zep、MemOS、smrti验证至2026年5月)不调用LLM在索引时间提取事实或构建图的层;直接嵌入原始对话文本。系统作为Claude Code插件、MCP 2024-11-05 A2A服务器(Cursor、Cline、Hermes)、CLI和REST API在一个守护进程中提供,配备Merkle链审计日志用于防篡改的锚点更新。在由真实Claude Code会话跟踪构建并由独立LLM法官标注的保留测试集上,Compass达到漂移检测的ROC AUC 0.83。嵌入检索管道在LongMemEval-S v0.8上得分为56.6%,在EverMemBench-Dynamic(n=500)上得分为44.4%,超越了四个已发表的EverMemBench Table 4基线。LongMemEval-S 56.6%比最近的白盒领导者(90%+)低约30个百分点,我们将其视为无提取设计的架构上限。端到端重现成本为3.50美元(比GPT-4o-judged堆栈便宜约14倍)。这些数字配对的跨供应商行为A/B测试作为初步系统级证据。
英文摘要:
Production LLM coding agents drift over long sessions: they forget user-specified constraints, slip into mistakes the user already flagged, and confabulate prior agreements. White-box approaches such as persona vectors require model weights and so cannot be applied to closed APIs (Claude, GPT-4) that most users actually interact with. We present Nautilus Compass, a black-box persona drift detector and agent memory layer for production coding agents. The method operates entirely at the prompt-text layer: cosine similarity between user prompts and behavioral anchor texts, aggregated by a weighted top-k mean using BGE-m3 embeddings. Compass is, to our knowledge, the only public agent memory layer (among Mem0, Letta, Cognee, Zep, MemOS, smrti verified May 2026) that does not call an LLM at index time to extract facts or build a graph; raw conversation text is embedded directly. The system ships as a Claude Code plugin, an MCP 2024-11-05 A2A server (Cursor, Cline, Hermes), a CLI, and a REST API on one daemon, with a Merkle-chained audit log for tamper-evident anchor updates. On a held-out test set built from real Claude Code session traces and labeled by an independent LLM judge, Compass reaches ROC AUC 0.83 for drift detection. The embedded retrieval pipeline scores 56.6% on LongMemEval-S v0.8 and 44.4% on EverMemBench-Dynamic (n=500), topping the four published EverMemBench Table 4 baselines. LongMemEval-S 56.6% is ~30 points below recent white-box leaders (90+%); we treat that as the architectural ceiling of the no-extraction design. End-to-end reproduction cost is $3.50 (~14x cheaper than GPT-4o-judged stacks). A paired cross-vendor behavior A/B accompanies these numbers as preliminary system-level evidence. Code, anchors, frozen test data, and audit-log tooling are MIT-licensed at github.com/chunxiaoxx/nautilus-compass.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。