二阶多级方差校正用于多模态模型中的模态竞争
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
AI总结：
本文提出ML-FOP-SOAP框架，通过多级方差校正提升多模态对齐稳定性，实验显示在Janus和Emu3数据集上，该方法提高了样本效率和训练速度，适用于大规模多模态基础模型。
摘要：
自回归的下一个标记训练为图像生成和文本理解提供统一框架，但同时也导致强模态竞争，破坏了优化稳定性并限制了大批次扩展。我们发现一阶优化器如AdamW易受跨模态梯度异质性影响，而二阶预条件，特别是SOAP，为多模态对齐提供了更稳定的基。基于此，我们提出ML-FOP-SOAP，一个带有多级方差校正的二阶优化框架。我们的Fisher-正交投影抑制由方差引起的模态冲突，减少视觉生成与文本理解之间的权衡。为在大梯度累积下实用，我们引入了分层折叠策略，以低微步开销捕获细粒度方差。在Janus和Emu3上的实验显示，在两个模态上均获得一致收益，并在8192批次大小下实现稳定训练。与AdamW相比，我们的方法提高了样本效率高达1.4倍，并加速了实时时钟训练高达1.5倍，为扩展多模态基础模型提供了一个稳健的优化器。
英文摘要：
Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality competition that destabilizes optimization and limits large-batch scaling. We show that first-order optimizers such as AdamW are vulnerable to cross-modality gradient heterogeneity, while second-order preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment. Building on this insight, we propose \emph{ML-FOP-SOAP}, a second-order optimization framework with Multi-Level Variance Correction. Our Fisher-Orthogonal Projection suppresses variance-induced modality conflicts, reducing the trade-off between visual generation and textual understanding. To make this practical under large gradient accumulation, we introduce a hierarchical folding strategy that captures fine-grained variance with low micro-step overhead. Experiments on Janus and Emu3 show consistent gains across both modalities and stable training at batch size 8192. Compared with AdamW, our method improves sample efficiency by up to $1.4\times$ and accelerates wall-clock training by up to $1.5\times$, offering a robust optimizer for scaling multimodal foundation models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
