NeuPAT：面向语言保留多模态大语言模型的神经元感知可塑性分配调优

中文摘要 AI 辅助

大语言模型（LLM）的多模态扩展使其具备了新的感知能力，但往往会损害其预训练阶段获得的语言智能。本研究从内部适应动力学角度探究该现象，发现预训练LLM的神经元在多模态学习中表现出异质性可塑性：部分神经元对语言能力的保留至关重要，而其他神经元更适配多模态知识。基于此，我们提出NeuPAT（Neuron-aware Plasticity Allocation Tuning，神经元感知可塑性分配调优），这是一种轻量且架构无关的框架，可在多模态指令调优期间分配神经元级别的更新约束。NeuPAT先通过小规模探测阶段估计神经元适应模式，选择性保护语言敏感神经元，同时利用可塑性更强的神经元促进多模态适应。在不同LLM家族上开展的实验表明，NeuPAT在11个语言基准测试中恢复了普通调优导致的94.5%语言能力下降，同时保持了相当的多模态性能，为能力保留型多模态扩展提供了有效方法。

英文摘要

Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.

发表机构

• College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院）
• Zhongguancun Academy（中关村学院）
• Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）
• School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）
• Tianjin University（天津大学）
• Nankai University（南开大学）

机构由 AI 辅助整理，请以论文原文为准。