Verses Research Lab(Verses研究实验室);DeepMind(深度思维);Google Research(谷歌研究院);University College London(伦敦大学学院);University of Toronto(多伦多大学);University of Berkeley, California(加州大学伯克利分校)
Jurgis Pasukonis, Timothy Lillicrap, Danijar Hafner
Intelligent agents need to remember salient information to reason in partially-observed environments. For example, agents with a first-person view should remember the positions of relevant objects even if they go out of view. Similarly, to effectively navigate through rooms agents need to remember the floor plan of how rooms are connected. However, most benchmark tasks in reinforcement learning do not test long-term memory in agents, slowing down progress in this important research direction. In this paper, we introduce the Memory Maze, a 3D domain of randomized mazes specifically designed for evaluating long-term memory in agents. Unlike existing benchmarks, Memory Maze measures long-term memory separate from confounding agent abilities and requires the agent to localize itself by integrating information over time. With Memory Maze, we propose an online reinforcement learning benchmark, a diverse offline dataset, and an offline probing evaluation. Recording a human player establishes a strong baseline and verifies the need to build up and retain memories, which is reflected in their gradually increasing rewards within each episode. We find that current algorithms benefit from training with truncated backpropagation through time and succeed on small mazes, but fall short of human performance on the large mazes, leaving room for future algorithmic designs to be evaluated on the Memory Maze.
Beyond the Shadows of Plato's Cave: Evaluating False Memory in Autonomous Agents via Counterfactual Reasoning
Sydney AI Centre, The University of Sydney(悉尼大学悉尼人工智能中心);Australian Institute for Machine Learning, Adelaide University(阿德莱德大学澳大利亚机器学习研究所);Australian Artificial Intelligence Institute, University of Technology Sydney(悉尼科技大学澳大利亚人工智能研究所);Wuhan University(武汉大学);School of Mathematics and Statistics, The University of Melbourne(墨尔本大学数学与统计学院)
AriadneMem: Threading the Maze of Lifelong Memory for LLM Agents
Arizona State University(亚利桑那州立大学);Morgan Stanley(摩根大通);Rice University(里奇大学);Clemson University(克莱姆森大学);Northwestern University(西北大学);UC Davis(加州大学戴维斯分校);Iowa State University(爱荷华州立大学);Washington University in St. Louis(圣路易斯华盛顿大学);Columbia University(哥伦比亚大学);Halmstad University(哈马格大学)
MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Models
University of Chicago(芝加哥大学);Toyota Technological Institute at Chicago(芝加哥丰田技术研究所);Syracuse University(雪城大学);WeChat AI(微信人工智能);New Jersey Institute of Technology(新泽西理工学院)
Beyond Memory Leaderboards: Evaluating Scientific Memory as Budgeted Context Restoration
Quantellence Research(Quantellence研究公司);St. Petersburg Department of the Steklov Institute of Mathematics(斯捷克洛夫数学研究所圣彼得堡分部);St. Petersburg State University(圣彼得堡国立大学)
StratMem-Bench: Evaluating Strategic Memory Use in Virtual Character Conversation Beyond Factual Recall
School of Computer Science and Engineering, Southeast University, China(东南大学计算机科学与工程学院,中国);Key Laboratory of New Generation Artificial Intelligence Technology and its Interdisciplinary Applications (Southeast University), Ministry of Education, China(新一代人工智能技术及其交叉应用国家重点实验室(东南大学),中华人民共和国教育部,中国);College of Design and Innovation, Tongji University, China(同济大学设计与创新学院,中国)
StoryBench: A Dynamic Benchmark for Evaluating Long-Term Memory with Multi Turns
Institute for AI Industry Research (AIR), Tsinghua University, Beijing, China(人工智能产业研究院(AIR),清华大学,北京,中国);University of Electronic Science and Technology of China, Chengdu, China(电子科技大学,成都,中国)
MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
Gaoling School of Artificial Intelligence, Renmin University of China(中国人民大学高瓴人工智能学院);Huawei Technologies Ltd.(华为技术有限公司);Huawei Noah’s Ark Lab(华为诺亚方舟实验室)
Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks
Max Planck Institute for Software Systems(马克斯·普朗克软件系统研究所);Intel Labs(英特尔实验室);University of Texas at Austin(德克萨斯大学奥斯汀分校);Princeton University(普林斯顿大学);Princeton Neuroscience Institute(普林斯顿神经科学研究所);New York University(纽约大学)