LLaVA-c：持续改进的视觉指令微调
LLaVA-c: Continual Improved Visual Instruction Tuning
AI总结：
针对多模态视觉指令微调中多任务学习的任务平衡难、易遗忘及基础模型退化问题，提出含谱感知巩固和无监督查询正则化的LLaVA-c方法，首次实现逐任务持续学习效果匹配或超越多任务联合学习。
摘要：
诸如LLaVA-1.5之类的多模态模型通过在多任务数据集上进行视觉指令微调，实现了当前最优的视觉理解效果，具备出色的指令遵循能力与多模态性能。然而，多任务学习面临任务平衡（需要精细调整数据比例）以及扩展成本方面的挑战：新增任务存在灾难性遗忘风险，且需要成本高昂的重新训练。持续学习为在保留现有能力的前提下增量获取新知识提供了一种有前景的替代方案。但现有方法优先考虑任务特定性能，忽视了基础模型因过拟合特定指令而出现的性能退化，这会损害通用能力。在本研究中，我们在LLaVA-1.5的基础上提出了一种简单却有效的方法，包含两处改进：用于改善任务平衡的谱感知巩固（spectral-aware consolidation），以及防止基础模型退化的无监督查询正则化（unsupervised inquiry regularization）。我们在持续预训练和微调两个阶段，同时评估了通用性能和任务特定性能。实验表明，LLaVA-c持续提升了标准基准测试的性能，同时保留了通用能力。我们首次证明，逐任务的持续学习能够取得与多任务联合学习相当甚至更优的结果。相关代码将公开发布。
英文摘要：
Multimodal models like LLaVA-1.5 achieve state-of-the-art visual understanding through visual instruction tuning on multitask datasets, enabling strong instruction-following and multimodal performance. However, multitask learning faces challenges such as task balancing, requiring careful adjustment of data proportions, and expansion costs, where new tasks risk catastrophic forgetting and need costly retraining. Continual learning provides a promising alternative to acquiring new knowledge incrementally while preserving existing capabilities. However, current methods prioritize task-specific performance, neglecting base model degradation from overfitting to specific instructions, which undermines general capabilities. In this work, we propose a simple but effective method with two modifications on LLaVA-1.5: spectral-aware consolidation for improved task balance and unsupervised inquiry regularization to prevent base model degradation. We evaluate both general and task-specific performance across continual pretraining and fine-tuning. Experiments demonstrate that LLaVA-c consistently enhances standard benchmark performance and preserves general capabilities. For the first time, we show that task-by-task continual learning can achieve results that match or surpass multitask joint learning. The code will be publicly released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
