Chronicle：用于LLM智能体回归测试的切点重放

中文摘要 AI 辅助

大语言模型的响应是非确定性的，因此LLM智能体中的故障难以复现：故障取决于无法按位复现的推理、读取变化状态的工具，以及重跑时很少重复的多步轨迹。记录与重放使运行可复现，但现有的智能体工具仅记录运行以进行追踪或评分，而非针对代码变更进行测试。我们提出Chronicle，它在非确定性边界处将智能体运行记录为不可变信封，并从记录中重放。其核心操作——切点重放——从记录中选取边界的子集，并用新代码实时执行互补子集，将记录的事件转化为在持续集成中运行的回归测试。在包含6个模拟模型边界的记录故障基准上，记录每次跨越增加23微秒（占假设300毫秒模型调用的0.008%），完整重放发出零次模型调用，并在20次重复中保持位稳定，切点测试在故障代码上失败，在所有6个事件中对受保护和良性变更通过。在对受保护工具进行的变异研究中，切点测试捕获了所有允许记录的不安全动作通过的变异体，而使用相同断言的基线（对所有边界进行桩替换）则未捕获任何变异体。Chronicle和基准可在以下URL公开获取：https://this https URL。

英文摘要

Large language model responses are non-deterministic, so failures in LLM agents are hard to reproduce: a failure depends on inference that is not bitwise reproducible, on tools that read changing state, and on a multi-step trajectory that a re-run rarely repeats. Record-and-replay makes a run reproducible, but existing agent tooling records runs only to trace or score them, not to test a code change against them. We present Chronicle, which records an agent run at its non-deterministic boundaries as immutable envelopes and replays it from the record. Its central operation, cut-point replay, serves a chosen subset of boundaries from the record and executes the complementary subset live with new code, turning a recorded incident into a regression test that runs in continuous integration. On a benchmark of 6 recorded failures with simulated model boundaries, recording adds 23 μs per crossing (0.008% of an assumed 300 ms model call), full replay issues zero model calls and is bit-stable across 20 repetitions, and cut-point tests fail on faulty code and pass on guarded and benign changes for all 6 incidents. In a mutation study of the guarded tools, cut-point tests catch every mutant that lets the recorded unsafe action through, while a baseline that stubs every boundary, using the same assertion, catches none. Chronicle and the benchmark are publicly available at https://github.com/theagentplane/chronicle.

发表机构

• Microsoft（微软）

机构由 AI 辅助整理，请以论文原文为准。