arXivDaily arXiv每日学术速递 周一至周五更新
北美退伍军人节（Veterans Day），12 日无更新
arXiv 2602.11243cs.LGcs.CL
收藏

评估LLM智能体中的记忆结构

Evaluating Memory Structure in LLM Agents

HSE University（莫斯科国立高等经济学院）；Yandex；YSDA；New Economic School（新经济学院）

Alina Shutova, Alexandra Olenina, Ivan Vinogradov, Anton Sinitsin

备注：
Preprint, work in progress
阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出StructMemEval基准测试，通过结构化记忆任务（如交易账本、待办事项列表、树结构等）评估LLM智能体组织长期记忆的能力，发现简单检索增强LLM难以胜任，而记忆智能体在提示下可解决，但现代LLM在无提示时无法自主识别记忆结构。

摘要：

现代基于LLM的智能体和聊天助手依赖长期记忆框架来存储可重用知识、回忆用户偏好并增强推理。随着研究人员创建更复杂的记忆架构，分析其能力并指导未来记忆设计变得越来越困难。大多数长期记忆基准侧重于简单事实保留、多跳回忆和基于时间的变化。虽然这些能力无疑很重要，但通常可以通过简单的检索增强LLM实现，并且不测试复杂的记忆层次。为了弥补这一差距，我们提出了StructMemEval——一个测试智能体组织其长期记忆能力（而不仅仅是事实回忆）的基准。我们收集了一系列人类通过以特定结构组织知识来解决的任务：交易账本、待办事项列表、树结构等。我们的初步实验表明，简单的检索增强LLM在这些任务上表现困难，而记忆智能体在提示如何组织记忆时可以可靠地解决它们。然而，我们还发现，现代LLM在未被提示时并不总是能识别记忆结构。这突显了未来在LLM训练和记忆框架改进方面的一个重要方向。

英文摘要：

Modern LLM-based agents and chat assistants rely on long-term memory frameworks to store reusable knowledge, recall user preferences, and augment reasoning. As researchers create more complex memory architectures, it becomes increasingly difficult to analyze their capabilities and guide future memory designs. Most long-term memory benchmarks focus on simple fact retention, multi-hop recall, and time-based changes. While undoubtedly important, these capabilities can often be achieved with simple retrieval-augmented LLMs and do not test complex memory hierarchies. To bridge this gap, we propose StructMemEval - a benchmark that tests the agent's ability to organize its long-term memory, not just factual recall. We gather a suite of tasks that humans solve by organizing their knowledge in a specific structure: transaction ledgers, to-do lists, trees and others. Our initial experiments show that simple retrieval-augmented LLMs struggle with these tasks, whereas memory agents can reliably solve them if prompted how to organize their memory. However, we also find that modern LLMs do not always recognize the memory structure when not prompted to do so. This highlights an important direction for future improvements in both LLM training and memory frameworks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑