Arbiter：检测LLM代理系统提示中的干扰
Arbiter: Detecting Interference in LLM Agent System Prompts
AI总结：
Arbiter通过结合形式评估规则和多模型LLM扫描，检测LLM代理系统提示中的干扰模式，发现152个问题并揭示漏洞类别差异。
摘要：
LLM基于的编码代理系统的系统提示是软件构件，用于指导代理行为，但缺乏传统软件所应用的测试基础设施。我们提出了Arbiter，一个结合形式评估规则和多模型LLM扫描的框架，用于检测系统提示中的干扰模式。应用于三个主要的编码代理系统提示：Claude Code（Anthropic）、Codex CLI（OpenAI）和Gemini CLI（Google），我们发现了152个发现，涵盖无向扫描阶段，以及在单个供应商的定向分析中发现的21个手动标记的干扰模式。我们显示，提示架构（单体、扁平、模块化）与观察到的故障类别强烈相关，但与严重性无关，且多模型评估发现的漏洞类别与单模型分析发现的完全不同。一个扫描发现指出Gemini CLI的内存系统中存在结构数据丢失的问题，该问题与Google已提交并修复的漏洞一致，该修复解决了症状，但未解决扫描者识别的架构级根本原因。跨供应商分析的总成本为0.27美元。
英文摘要：
System prompts for LLM-based coding agents are software artifacts that govern agent behavior, yet lack the testing infrastructure applied to conventional software. We present Arbiter, a framework combining formal evaluation rules with multi-model LLM scouring to detect interference patterns in system prompts. Applied to three major coding agent system prompts: Claude Code (Anthropic), Codex CLI (OpenAI), and Gemini CLI (Google), we identify 152 findings across the undirected scouring phase and 21 hand-labeled interference patterns in directed analysis of one vendor. We show that prompt architecture (monolithic, flat, modular) strongly correlates with observed failure class but not with severity, and that multi-model evaluation discovers categorically different vulnerability classes than single-model analysis. One scourer finding was structural data loss in Gemini CLI's memory system was consistent with an issue filed and patched by Google, which addressed the symptom without addressing the schema-level root cause identified by the scourer. Total cost of cross-vendor analysis: \$0.27 USD.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
