arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.28889cs.LGcs.AI
收藏

上下文蒸馏作为潜在记忆管理

Context Distillation as Latent Memory Management

The Chinese University of Hong Kong(香港中文大学);Huawei Noah’s Ark Lab(华为诺亚实验室)

Ziyang Zheng, Zeju Li, Xiangyu Wen, Jianyuan Zhong, Junhua Huang, Lei Chen, Mingxuan Yuan, Qiang Xu

AI总结:

将上下文蒸馏视为潜在记忆管理问题,通过独立LoRA适配器形成模块化记忆库,并利用自门控机制决定是否激活潜在记忆,以提升检索鲁棒性和效率。

摘要:

上下文蒸馏将上下文信息压缩到模型参数中,然而现有方法常常忽略多个蒸馏后的潜在记忆应如何在非预言机设置下存储、检索和安全激活。我们将上下文蒸馏表述为一个潜在记忆管理问题。我们将每个上下文蒸馏成一个独立的LoRA适配器,形成一个模块化记忆库,从而实现显式的记忆选择。给定一个查询,我们的框架检索候选记忆,将查询路由到最合适的适配器,并使用自门控机制决定是否应激活潜在记忆。为了提高效率,我们进一步引入缓存共享以减少推理过程中的管理开销。实验表明,我们的方法在检索方面显著优于基线,而自门控通过停用不必要的潜在记忆提高了鲁棒性。

英文摘要:

Context distillation compresses contextual information into model parameters, yet existing methods often ignore how multiple distilled latent memories should be stored, retrieved, and safely activated in non-oracle settings. We formulate context distillation as a latent memory management problem. We distill each context into an independent LoRA adapter, forming a modular memory bank that enables explicit memory selection. Given a query, our framework retrieves candidate memories, routes the query to the most suitable adapter, and uses a Self-Gating mechanism to decide whether latent memory should be activated. To improve efficiency, we further introduce cache sharing to reduce management overhead during inference. Experiments show that our method substantially outperforms baselines with retrieval, while Self-Gating improves robustness by deactivate unnecessary latent memories.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑