保持通用，注入特定：用于无灾难性遗忘的知识注入结构化对话微调
Keep the General, Inject the Specific: Structured Dialogue Fine-Tuning for Knowledge Injection without Catastrophic Forgetting
AI总结：
针对大型视觉语言模型注入专业知识时易引发灾难性遗忘的困境，提出结构化对话微调（SDFT）方法，通过基础保持、对比消歧和知识专业化三阶段对话结构，在注入领域知识的同时有效保持通用能力。
摘要：
大型视觉语言模型通过广泛的多模态预训练展现出了令人印象深刻的多功能能力，但在纳入其训练分布之外的专业知识领域时面临重大局限。这些模型陷入了一个基本困境：注入领域特定知识的直接适应方法往往会引发对基础视觉语言能力的灾难性遗忘。我们引入了结构化对话微调（SDFT），这是一种在最小化灾难性遗忘的同时有效注入领域特定知识的方法。受大型语言模型中的监督微调以及文本到图像扩散模型中主体驱动个性化的启发，我们的方法采用三阶段对话结构：基础保持通过描述任务强化预训练的视觉语言对齐；对比消歧引入精心设计的反事实示例以维持语义边界；知识专业化通过思维链推理嵌入专业信息。跨多个领域的实验结果证实了SDFT在平衡专业知识获取与通用能力保持方面的有效性。我们的主要贡献包括一个以数据为中心的对话模板，该模板平衡了基础对齐与目标知识整合，一个加权多轮监督框架，以及跨不同知识类型的全面评估。
英文摘要：
Large Vision Language Models have demonstrated impressive versatile capabilities through extensive multimodal pre-training, but face significant limitations when incorporating specialized knowledge domains beyond their training distribution. These models struggle with a fundamental dilemma: direct adaptation approaches that inject domain-specific knowledge often trigger catastrophic forgetting of foundational visual-linguistic abilities. We introduce Structured Dialogue Fine-Tuning (SDFT), an effective approach that effectively injects domain-specific knowledge while minimizing catastrophic forgetting. Drawing inspiration from supervised fine-tuning in LLMs and subject-driven personalization in text-to-image diffusion models, our method employs a three-phase dialogue structure: Foundation Preservation reinforces pre-trained visual-linguistic alignment through caption tasks; Contrastive Disambiguation introduces carefully designed counterfactual examples to maintain semantic boundaries; and Knowledge Specialization embeds specialized information through chain-of-thought reasoning. Experimental results across multiple domains confirm SDFT's effectiveness in balancing specialized knowledge acquisition with general capability retention. Our key contributions include a data-centric dialogue template that balances foundational alignment with targeted knowledge integration, a weighted multi-turn supervision framework, and comprehensive evaluation across diverse knowledge types.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
