ImplicitMemBench: 测量大语言模型中的无意识行为适应
ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models
AI总结:
本文提出ImplicitMemBench,首个评估大语言模型隐性记忆的系统基准,通过三个认知基础构念测试隐性记忆,揭示模型在自动执行任务中的局限性。
摘要:
本文提出了ImplicitMemBench,首个评估大语言模型隐性记忆的系统基准,通过三个认知基础构念测试隐性记忆,揭示模型在自动执行任务中的局限性。现有针对LLM代理的记忆基准主要评估显性事实回忆,却忽略了隐性记忆,即经验转化为自动行为而无需意识检索。这一缺口至关重要:有效的助手必须能够自动应用学习的程序或避免失败的行为而无需显式的提醒。我们引入ImplicitMemBench,首个系统性评估隐性记忆的基准,通过三个源自标准认知科学非声明性记忆账户的基础构念:程序记忆(干扰后的单次技能获取)、诱动(通过配对实验/控制实例产生的主题驱动偏见)以及经典条件反射(条件刺激-无条件刺激(CS-US)关联塑造首次决策)。我们的300项测试套件采用统一的学-诱动-测试协议,并采用首次尝试评分。对17个模型的评估揭示了严重的局限性:没有模型能超过66%的总体成绩,表现最佳的模型DeepSeek-R1(65.3%)、Qwen3-32B(64.1%)和GPT-5(63.0%)远低于人类基准。分析揭示了显著的不对称性(抑制17.6% vs. 偏好75.0%)和普遍的瓶颈,需要超越参数扩展的架构创新。ImplicitMemBench将评估从“代理回忆什么”转向“代理自动执行什么”。
英文摘要:
Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval. This gap is critical: effective assistants must automatically apply learned procedures or avoid failed actions without explicit reminders. We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three cognitively grounded constructs drawn from standard cognitive-science accounts of non-declarative memory: Procedural Memory (one-shot skill acquisition after interference), Priming (theme-driven bias via paired experimental/control instances), and Classical Conditioning (Conditioned Stimulus--Unconditioned Stimulus (CS--US) associations shaping first decisions). Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring. Evaluation of 17 models reveals severe limitations: no model exceeds 66% overall, with top performers DeepSeek-R1 (65.3%), Qwen3-32B (64.1%), and GPT-5 (63.0%) far below human baselines. Analysis uncovers dramatic asymmetries (inhibition 17.6% vs. preference 75.0%) and universal bottlenecks requiring architectural innovations beyond parameter scaling. ImplicitMemBench reframes evaluation from "what agents recall" to "what they automatically enact".
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。