arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.09461cs.CL
收藏

H2HMem: 面向人际交互中智能体的多模态记忆基准

H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactions

Jilin University(吉林大学);Shanghai Jiao Tong University(上海交通大学);University of California at Merced(加州大学默塞德分校)

Shiping Zhu, Yibo Yang, Zhengyang Wang, Tiancheng Shen, Dandan Guo, Ming-Hsuan Yang

AI总结:

提出H2HMem基准,通过双人和多人多模态对话评估智能体在记忆召回、推理和应用方面的能力,揭示现有模型在多模态、多参与者场景下的显著局限。

摘要:

大型语言模型智能体越来越多地部署在人际交互场景中,例如会议助手和临床文档系统,在这些场景中,它们必须观察对话并保留信息以供后续查询。与传统的人助交互不同,这些环境本质上是多模态的,涉及复杂的语篇现象,如回指和指示,并且包含来自多个参与者的异步或冲突信息。然而,现有的记忆基准主要关注单用户、纯文本交互,未能捕捉这些挑战。为填补这一空白,我们引入了H2HMem,一个面向复杂人际交互中记忆能力评估的人-人多模态记忆基准。H2HMem包括双人和多人对话,包含多模态信息流,并从三个维度评估智能体:记忆召回、推理和应用。使用先进智能体的实验揭示了在跨模态、参与者和会话中构建、保留和利用记忆方面的显著局限性,凸显了下一代LLM智能体需要大幅改进的空间。

英文摘要:

Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑