Large language model agents are increasingly deployed in human-human interaction settings, such as meeting assistants and clinical documentation systems, where they must observe conversations and retain information for downstream queries. Unlike traditional human-assistant settings, these environments are inherently multimodal, involve complex discourse phenomena such as anaphora and deixis, and contain asynchronous or conflicting information from multiple participants. However, existing memory benchmarks largely focus on single-user, text-only interactions, failing to capture these challenges. To address this gap, we introduce H2HMem, a Human-to-Human Multimodal Memory Benchmark for evaluating memory capabilities in complex human-human interactions. H2HMem includes both dyadic and multi-party conversations with multimodal information streams, and evaluates agents along three dimensions: memory recall, reasoning, and application. Experiments with advanced agents reveal substantial limitations in constructing, retaining, and utilizing memories across modalities, participants, and sessions, highlighting substantial room for improvement in next-generation LLM agents.
Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractions
School of Computer Science(计算机学院);Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering(智能交互与认知工程重点实验室);Shanghai Jiao Tong University(上海交通大学);Shanghai Key Laboratory of Trusted Data Circulation and Governance in Web3(Web3可信数据流通与治理上海市重点实验室);GenAI, Meta(Meta GenAI)
WorldMemArena: Evaluating Multimodal Agent Memory Through Action-World Interaction
University of California, Santa Barbara(加州大学圣芭芭拉分校);J.P. Morgan Chase(摩根大通);ETH Zurich(苏黎世联邦理工学院);Stanford University(斯坦福大学);Johns Hopkins University(约翰霍普金斯大学);Carnegie Mellon University(卡内基梅隆大学)
Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agents
The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences(认知与决策智能复杂系统重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
M$^3$Exam: Benchmarking Multimodal Memory for Realistic User-Agent Interactions
The Hong Kong University of Science and Technology(香港科学与技术大学);Beijing University of Chemical Technology(北京化工大学);The Hong Kong University of Science and Technology (Guangzhou)(香港科学与技术大学(广州));Harbin Institute of Technology (Shenzhen)(哈尔滨工业大学(深圳));Beijing Institute of Technology (Zhuhai)(北京理工大学(珠海));Tencent Hy(腾讯(深圳));Peng Cheng Laboratory(鹏城实验室)
SentiMM: A Multimodal Multi-Agent Framework for Sentiment Analysis in Social Media
College of Information and Electrical Engineering, China Agricultural University(信息与电气工程学院,中国农业大学);College of Software, Jilin University(软件学院,吉林大学);College of Communication Engineering, Jilin University(通信工程学院,吉林大学)
MedPrune: Topology-Efficient Multimodal Multi-Agent Communication Evolution for Medical VQA Tasks
Hefei University of Technology(合肥工业大学);Nanjing University(南京大学);Guangdong University of Finance and Economics(广东财经大学);East China Normal University(华东师范大学);Lenovo(联想集团)
DELTA: Deliberative Multi-Agent Reasoning with Reinforcement Learning for Multimodal Psychological Counseling
Anhui University(安徽大学);Hefei University of Technology(合肥工业大学);Institute of Artificial Intelligence, Hefei Comprehensive National Science Center(合肥综合国家科学中心人工智能研究所)
UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp
Key Laboratory of Computational Linguistics, MOE, Peking University(教育部计算语言学重点实验室,北京大学);School of Software and Microelectronics, Peking University(北京大学软件与微电子学院);School of Computer Science, Peking University(北京大学计算机科学学院)
TAMA: Tool-Augmented Multimodal Agent for Procedural Activity Understanding
Language Technologies Institute, Carnegie Mellon University(卡内基梅隆大学语言技术研究所);National Institute of Advanced Industrial Science and Technology (AIST)(国家先进工业科学与技术研究院)
Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agents
University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);MIT-IBM Watson AI Lab, IBM Research(MIT-IBM沃森人工智能实验室,IBM研究);Stony Brook University(石溪大学);Brookhaven National Laboratory(布鲁赫斯国家实验室)
Cooperative Sentiment Agents for Multimodal Sentiment Analysis
College of Computer Science and Technology in Nanjing University of Aeronautics and Astronautics(南京航空航天大学计算机科学与技术学院);School of Computer Science in Nanjing University of Posts and Telecommunications(南京邮电大学计算机学院);Nanjing University of Aeronautics and Astronautics(南京航空航天大学);Functional Brain Imaging Institute(功能脑成像研究所);Nanjing Medical University(南京医科大学)