学习融合：面向鲁棒多模态基础模型的模态感知自适应调度
Learning to Fuse: Modality-Aware Adaptive Scheduling for Robust Multimodal Foundation Models
AI总结：
针对现有多模态基础模型融合策略固定、忽略模态可靠性差异的问题，提出MA-AFS自适应融合调度框架，通过轻量调度器动态调整模态权重，在多任务上超越强基线且提升了鲁棒性与泛化性。
摘要：
多模态基础模型已在各类视觉-语言任务上取得了显著进展。然而现有方法通常采用固定或面向特定任务的融合策略，忽略了模态可靠性与样本复杂度的内在差异性。本文提出模态感知自适应融合调度框架（Modality-Aware Adaptive Fusion Scheduling，MA-AFS），这是一种通用框架，能够学习针对每个样本动态调整各模态的贡献占比。MA-AFS引入了一个轻量级神经调度器，通过整合视觉与文本熵信号以及跨模态一致性线索来预测模态融合权重，使模型能够自适应地侧重更可靠的模态，尤其在输入存在噪声、缺失或错位的场景下效果突出。我们将融合过程建模为可微调度机制，分析了其理论一致性与正则化效果，证明该方法在不显著增加模型容量的前提下提升了鲁棒性。在图文检索、图像描述生成和视觉问答任务上的大量实验表明，MA-AFS相比CLIP、ALBEF、BLIP等强基线模型取得了稳定的性能提升。此外，MA-AFS在模态损坏场景下鲁棒性更强，在域迁移场景下泛化能力更优。本研究凸显了自适应融合的重要性，为实现可靠且具备不确定性感知的多模态学习开辟了有前景的方向。
英文摘要：
Multimodal foundation models have achieved impressive progress across a wide range of vision-language tasks. However, existing approaches often adopt fixed or task-specific fusion strategies, neglecting the intrinsic variability of modality reliability and sample complexity. In this paper, we propose Modality-Aware Adaptive Fusion Scheduling (MA-AFS), a general framework that learns to dynamically modulate the contribution of each modality on a per-instance basis. MA-AFS introduces a lightweight neural scheduler that predicts modality fusion weights by integrating visual and textual entropy signals along with cross-modal agreement cues. This enables the model to adaptively emphasize more reliable modalities, especially under noisy, missing, or misaligned inputs. We formulate the fusion process as a differentiable scheduling mechanism, analyze its theoretical consistency and regularization effect, and demonstrate that it improves robustness without increasing model capacity significantly. Extensive experiments on image-text retrieval, captioning, and visual question answering show that MA-AFS achieves consistent performance gains over strong baselines such as CLIP, ALBEF, and BLIP. Moreover, MA-AFS exhibits improved robustness under modality corruption and enhanced generalization under domain shifts. Our work highlights the importance of adaptive fusion and opens a promising direction toward reliable and uncertainty-aware multimodal learning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
