超越困惑度：面向LLM测试时训练中部署记忆声称的行为评估框架
Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training
AI总结：
提出行为评估框架，通过证据阶梯和显式记忆基线校准LLM测试时训练的记忆声称，揭示代理指标与部署行为之间的差距。
摘要：
大型语言模型测试时训练（TTT）通常通过局部代理指标进行评估：模型在最近token、检索上下文、目标域数据或可验证任务尝试上进行更新，然后通过困惑度、未来token损失、长上下文性能或奖励来判断。这些指标与流适应、域适应、上下文压缩和奖励支持的测试时改进声称相匹配。然而，对于TTT结果越来越多地被用来推动的能力——部署助手记忆、个性化或稀疏部署后学习——这些指标提供的证据较弱，这些能力反而需要行为证据，如后续回忆、释义鲁棒性、保留、局部性、冲突处理以及在原始支持上下文被移除后的下游行动中使用。我们引入了一个行为评估框架，将TTT记忆声称校准到支持它们的证据上。它包含两个组成部分：一个声称校准的证据阶梯，将流/域适应、桥接内化和部署时行为学习分开；以及一个评估协议，配有匹配的显式记忆基线和互斥的失败类别。我们通过审计最近的TTT和记忆相关工作，并将其实例化为一个受控诊断来验证该框架，在该诊断中，在稀疏的一次性事实设置下，单步LoRA更新降低了三个Qwen3模型规模上的支持和答案损失，而生成的自由形式回忆保持为零，暴露了代理改进与部署行为之间的可测量差距。该框架为作者和评估者提供了将TTT记忆声称与实际报告的证据对齐的具体标准。
英文摘要：
Large language model test-time training (TTT) is often evaluated through local proxy metrics: models are updated on recent tokens, retrieved context, target-domain data, or verifiable task attempts, and then judged by perplexity, future-token loss, long-context performance, or reward. These metrics are well matched to claims about stream adaptation, domain adaptation, context compression, and reward-backed test-time improvement. They are weaker evidence, however, for a capability that TTT results are increasingly used to motivate: deployed assistant memory, personalization, or sparse post-deployment learning, which instead requires behavioral evidence such as later recall, paraphrase robustness, retention, locality, conflict handling, and use in downstream actions after the original support context is removed. We introduce a behavioral evaluation framework that calibrates TTT memory claims to the evidence that supports them. It has two components: a claim-calibrated evidence ladder that separates stream/domain adaptation, bridge internalization, and deployment-time behavioral learning; and an evaluation protocol with matched explicit-memory baselines and mutually exclusive failure categories. We validate the framework by auditing recent TTT and memory-adjacent work and by instantiating it as a controlled diagnostic in which, in a sparse nonce-fact setting, one-step LoRA updates lower support and answer loss across three Qwen3 model scales while generated free-form recall stays at zero, exposing a measurable gap between proxy improvement and deployment behavior. The framework gives authors and evaluators a concrete standard for aligning TTT memory claims with the evidence actually reported.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
