评估LLM智能体的超长期对话记忆
Evaluating Very Long-Term Conversational Memory of LLM Agents
AI总结：
本文针对超长期对话记忆评估的空白，提出机器-人工流水线生成高质量超长期对话并构建LoCoMo数据集，通过问答、事件摘要和多模态生成基准揭示现有LLMs与人类在长程记忆上的显著差距。
摘要：
现有的长期开放域对话研究主要关注在不超过五个聊天会话的上下文中评估模型响应。尽管长上下文大语言模型（LLMs）和检索增强生成（RAG）技术取得了进展，但它们在超长期对话中的有效性仍未得到探索。为填补这一研究空白，我们提出了一种机器-人工流水线，通过利用基于LLM的智能体架构，并将其对话建立在人物设定和时间事件图之上，来生成高质量的超长期对话。此外，我们为每个智能体配备了分享图像并对图像做出反应的能力。生成的对话经过人工标注者验证和编辑，以确保长程一致性以及对事件图的依据性。利用该流水线，我们收集了LoCoMo，一个超长期对话数据集，每个对话平均包含300轮次和9K个token，跨越最多35个会话。基于LoCoMo，我们提出了一个全面的评估基准，用于衡量模型的长期记忆能力，涵盖问答、事件摘要和多模态对话生成任务。我们的实验结果表明，LLMs在理解冗长对话以及把握对话中的长程时间和因果动态方面面临挑战。采用长上下文LLMs或RAG等策略可以带来改进，但这些模型仍然显著落后于人类表现。
英文摘要：
Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
