一种用于语言代理中程序记忆检索的基准测试
A Benchmark for Procedural Memory Retrieval in Language Agents
AI总结：
本文提出了一种用于评估语言代理程序记忆检索能力的基准测试，通过构建双语料库和系统分层查询，揭示了基于嵌入的方法在新上下文中的泛化问题，并展示了LLM生成的程序抽象在跨上下文转移中的优势。
摘要：
当前人工智能代理在熟悉环境中表现优异，但面对涉及未见过词汇的新任务时却表现急剧下降--这是程序记忆系统的核心限制。我们提出了第一个基准测试，它将程序记忆检索与任务执行分离，评估代理是否能够识别跨越不同对象实例的函数等效过程。使用ALFWorld，我们构建了专家和LLM生成轨迹的双语料库，并使用系统分层查询评估了六种检索方法。我们的结果揭示了一个明显的泛化悬崖：基于嵌入的方法在熟悉上下文中表现强劲，但在新上下文中却显著下降，而LLM生成的程序抽象在跨上下文转移中表现出可靠的稳定性。受控消融显示，尽管嵌入捕获了一些词汇层面的抽象，但它们本质上将过程视为无序的词袋，丢弃了跨上下文转移所需的时序结构。语料库规模带来的收益远大于表示丰富度，揭示了当前编码器在架构上的天花板。我们的基准测试提供了第一个诊断框架，将真正的程序理解与表层记忆区分开来，并提供了开发能够可靠泛化的检索系统的工具。资源可在我们的GitHub仓库（https://github.com/qpiai/Proced_mem_bench）中获得。
英文摘要：
Current AI agents excel in familiar settings, but fail sharply when faced with novel tasks with unseen vocabularies -- a core limitation of procedural memory systems. We present the first benchmark that isolates procedural memory retrieval from task execution, evaluating whether agents can recognize functionally equivalent procedures that span different object instantiations. Using ALFWorld, we construct dual corpora of expert and LLM-generated trajectories and evaluate six retrieval methods using systematically stratified queries. Our results expose a clear generalization cliff: embedding-based methods perform strongly on familiar contexts, yet degrade considerably on novel ones, while LLM-generated procedural abstractions demonstrate reliable cross-context transfer. Controlled ablations show that although embeddings capture some lexical-level abstraction, they fundamentally treat procedures as unordered bags of words, discarding temporal structure necessary for cross-context transfer. Corpus scale delivers far larger gains than representation enrichment, revealing an architectural ceiling in current encoders. Our benchmark offers the first diagnostic framework separating genuine procedural understanding from surface-level memorization and gives tools for developing retrieval systems capable of dependable generalization. Resources available at our GitHub repository (https://github.com/qpiai/Proced_mem_bench).
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
