MemLoRA: 通过专家适配器实现设备端记忆系统
MemLoRA: Distilling Expert Adapters for On-Device Memory Systems
AI总结：
MemLoRA通过为小型语言模型添加专用记忆适配器，实现设备端记忆系统的本地部署，并扩展至视觉理解任务，提升多模态场景下的性能。
摘要：
记忆增强型大语言模型（LLMs）通过存储相关记忆并在对话中整合这些记忆来展示出在长时间对话中的一致性。这种基于记忆的个性化在允许用户保留对话和数据隐私的设备端设置中也至关重要。然而，记忆增强型系统通常依赖于成本过高的LLMs，这使得本地设备部署变得困难。尽管小型语言模型（SLMs）比LLMs更适合设备端推理，但它们无法达到足够的性能。此外，基于LLM的系统缺乏原生的视觉能力，限制了它们在多模态环境中的应用。在本文中，我们引入了（i）MemLoRA，一种新的记忆系统，通过为SLMs配备专用的记忆适配器实现本地部署，以及（ii）其视觉扩展MemLoRA-V，该扩展将小型视觉-语言模型（SVLMs）整合到记忆系统中，从而实现原生的视觉理解。遵循知识蒸馏原则，每个适配器分别针对特定的记忆操作——知识提取、记忆更新和记忆增强生成进行训练。配备了记忆适配器的小型模型能够实现准确的设备端记忆操作，而无需依赖云服务。在纯文本操作上，MemLoRA在10倍于基线模型（如Gemma2-27B）的性能上表现更优，并在LoCoMo基准测试中与60倍于基线模型（如GPT-OSS-120B）的性能相当。为了评估视觉理解操作，我们扩展了LoCoMo，加入了具有挑战性的视觉问答任务，这些任务需要直接的视觉推理。在这些任务上，我们的集成VLM的MemLoRA-V在基于描述符的方法上实现了巨大的改进（81.3 vs. 23.7准确率），同时在基于文本的任务中保持了强劲的性能，证明了我们方法在多模态环境中的有效性。
英文摘要：
Memory-augmented Large Language Models (LLMs) have demonstrated remarkable consistency during prolonged dialogues by storing relevant memories and incorporating them as context. Such memory-based personalization is also key in on-device settings that allow users to keep their conversations and data private. However, memory-augmented systems typically rely on LLMs that are too costly for local on-device deployment. Even though Small Language Models (SLMs) are more suitable for on-device inference than LLMs, they cannot achieve sufficient performance. Additionally, these LLM-based systems lack native visual capabilities, limiting their applicability in multimodal contexts. In this paper, we introduce (i) MemLoRA, a novel memory system that enables local deployment by equipping SLMs with specialized memory adapters, and (ii) its vision extension MemLoRA-V, which integrates small Vision-Language Models (SVLMs) to memory systems, enabling native visual understanding. Following knowledge distillation principles, each adapter is trained separately for specific memory operations$\unicode{x2013}$knowledge extraction, memory update, and memory-augmented generation. Equipped with memory adapters, small models enable accurate on-device memory operations without cloud dependency. On text-only operations, MemLoRA outperforms 10$\times$ larger baseline models (e.g., Gemma2-27B) and achieves performance comparable to 60$\times$ larger models (e.g., GPT-OSS-120B) on the LoCoMo benchmark. To evaluate visual understanding operations instead, we extend LoCoMo with challenging Visual Question Answering tasks that require direct visual reasoning. On this, our VLM-integrated MemLoRA-V shows massive improvements over caption-based approaches (81.3 vs. 23.7 accuracy) while keeping strong performance in text-based tasks, demonstrating the efficacy of our method in multimodal contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
