arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17554cs.LGstat.ML
收藏

模块化鲁棒生成模型的理论框架

A Theoretical Framework for Modular Learning of Robust Generative Models

Google Research(谷歌研究院)

Corinna Cortes, Mehryar Mohri, Yutao Zhong

AI总结:

本文提出模块化鲁棒生成模型的理论框架,通过门控机制结合预训练专家,理论证明其在数据混合下的稳健性,并通过实验验证其优于整体模型的性能。

摘要:

训练大规模生成模型耗费资源且依赖于启发式数据集加权。我们解决两个根本问题:能否模块化地训练大型语言模型(LLMs),通过结合小规模、领域特定的专家来达到整体性能,并且能否在任何数据混合情况下稳健地进行,消除启发式调参?我们提出了一个模块化生成建模的理论框架,其中一组预训练的专家通过门控机制进行组合。我们定义了归一化门控函数的空间$G_{1}$,并将其问题表述为一个极小极大游戏,以找到一个单个稳健的门,最小化与最坏情况数据混合的发散度。我们利用Kakutani的不动点定理证明了此类稳健门的存在,并表明模块化作用作为强大的正则化器,泛化界限随轻量级门的复杂性而变化。此外,我们证明这种模块化方法在理论上可以超越重新训练聚合数据的模型,差距由Jensen-Shannon散度表征。最后,我们引入了可扩展的随机对偶算法和结构蒸馏方法,以实现高效的推断。在合成和真实世界数据集上的实验证明,我们的模块化架构有效缓解了梯度冲突,并能稳健地超越整体基线。

英文摘要:

Training large-scale generative models is resource-intensive and relies heavily on heuristic dataset weighting. We address two fundamental questions: Can we train Large Language Models (LLMs) modularly, combining small, domain-specific experts to match monolithic performance, and can we do so robustly for any data mixture, eliminating heuristic tuning? We present a theoretical framework for modular generative modeling where a set of pre-trained experts are combined via a gating mechanism. We define the space of normalized gating functions $\mathcal{G}_{1}$ and formulate the problem as a minimax game to find a single robust gate that minimizes divergence to the worst-case data mixture. We prove the existence of such a robust gate using Kakutani's fixed-point theorem and show that modularity acts as a strong regularizer, with generalization bounds scaling with the lightweight gate's complexity. Furthermore, we prove that this modular approach can theoretically outperform models retrained on aggregate data, with the gap characterized by the Jensen-Shannon Divergence. Finally, we introduce a scalable Stochastic Primal-Dual algorithm and a Structural Distillation method for efficient inference. Empirical results on synthetic and real-world datasets confirm that our modular architecture effectively mitigates gradient conflict and can robustly outperform monolithic baselines.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑