Long-term memory has become a core component of large language model (LLM) agents. As agents accumulate personal information over long-term use, users increasingly ask them to delete sensitive or outdated facts. A memory system should honor such requests. It should remove exactly what was requested and keep the rest of its memory intact. However, existing memory benchmarks evaluate only memory writing, retrieval, and organization, so whether agents can forget at the user's explicit request remains under-evaluated. To address this gap, we propose Bemula, the first benchmark dedicated to memory unlearning. It comprises 50 multi-session dialogue instances, 135 forget requests of six types, and 1,742 test questions. Each dialogue has a control version without the requests, so that only systems that have stored the target can earn credit for forgetting. Across the paired versions, five test suites measure storage, removal of the requested information, preservation of unrelated memories, relearning of similar facts, and resistance to extraction attacks. On Bemula, we evaluate six mainstream memory systems under three backbone LLMs, together with the Full Context LLM on two of them. The results are negative. Across all evaluated systems and backbones, no system passes more than 1.5% of forget requests end to end, and most pass none at all. These systems either leave the requested information extractable under simple attacks or delete far more than requested. Memory unlearning therefore remains an open challenge for agent memory design. We release our data and code at https://github.com/Nboxff/bemula-benchmark.
Can Agents Truly Forget? Explore Machine Unlearning for LLM Agents
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；The Chinese University of Hong Kong（香港中文大学）；Nanyang Technological University（南洋理工大学）；University of Jinan（济南大学）；Pengcheng Laboratory（鹏城实验室）
Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
Hefei University of Technology（合肥工业大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；University of Amsterdam（阿姆斯特丹大学）
Quantization-Robust Unlearning through the Lens of Retain-Forget Loss Landscapes Interaction
University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）；University of Pittsburgh（匹兹堡大学）；University of Arizona（亚利桑那大学）；University of Central Florida（中佛罗里达大学）
Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models
Harbin Institute of Technology（哈尔滨工业大学）；Huawei Noah’s Ark Lab（华为诺亚方舟实验室）；Unlimidata Ltd（Unlimidata有限公司）；The Chinese University of Hong Kong（香港中文大学）；University of Calabria（卡拉布里亚大学）
Attention Smoothing Is All You Need For Unlearning
Department of Computer Science, Wayne State University（韦恩州立大学计算机科学系）；Department of Computer Science and Engineering, Michigan State University（密歇根州立大学计算机科学与工程系）；Institute for AI and Data Science, Wayne State University（韦恩州立大学人工智能与数据科学研究所）
Forgetting-MarI: LLM Unlearning via Marginal Information Regularization
Department of Mathematics, University of California Davis, USA（加州大学戴维斯分校数学系）；SLAC National Accelerator Laboratory, Stanford University, USA（斯坦福大学SLAC国家加速器实验室）；Department of Computer Science, University of California Davis, USA（加州大学戴维斯分校计算机科学系）
Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs
The Hong Kong Polytechnic University（香港理工大学）；Carnegie Mellon University（卡内基梅隆大学）；University of California, Santa Cruz（加州大学圣克ruz分校）；Huawei Technologies（华为技术有限公司）；Research Centre for Privacy and Security Technologies in Future Smart Systems, PolyU（未来智能系统中的隐私与安全技术研究中心，PolyU）
Unlearning's Blind Spots: Over-Unlearning and Prototypical Relearning Attack
Graduate School of Artificial Intelligence, Ulsan National Institute of Science and Technology (UNIST), Ulsan, South Korea（人工智能研究生院，乌山国立科学与技术研究所（UNIST），乌山，韩国）；Department of Industrial Engineering, UNIST, Ulsan, South Korea（工业工程系，UNIST，乌山，韩国）；Department of Electrical Engineering, UNIST, Ulsan, South Korea（电气工程系，UNIST，乌山，韩国）
Forget and Explain: Transparent Verification of GNN Unlearning
Chung-Ang University Department of Smart Cities Seoul Republic of Korea（Chung-Ang大学智慧城市系 首尔 大韩民国）；Chung-Ang University Department of Computer Science（Chung-Ang大学计算机科学系）；Chung-Ang University（Chung-Ang大学）
University of Trento（特伦托大学）；University of Bergamo（贝加莫大学）；Inria Grenoble（格勒诺布尔研究所）；Univ. Grenoble Alpes（格勒诺布尔阿尔卑斯大学）；Fondazione Bruno Kessler（布鲁诺·凯斯勒基金会）
