arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2212.10445cs.LGcs.AIcs.CV
收藏

模型杂烩:回收多样化模型以实现分布外泛化

Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalization

Meta AI;Sorbonne Université(索邦大学);NYU(纽约大学)

Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, David Lopez-Paz

AI总结:

本文提出模型杂烩策略,通过回收同一基础模型在多样化辅助任务上的微调权重作为目标任务并行微调的初始化并取平均,从而提升分布外泛化性能,并在DomainBed基准上达到最优。

摘要:

基础模型正在重新定义人工智能系统的构建方式。从业者现在遵循一套标准流程来构建他们的机器学习解决方案:从预训练的基础模型出发,在感兴趣的目标任务上微调权重。因此,互联网上充斥着少数基础模型在众多多样化任务上微调得到的版本:这些单独的微调彼此孤立存在,无法相互受益。我们认为,这是一个被错失的机会,因为这些专用模型包含丰富且多样的特征。因此,在本文中,我们提出了模型杂烩(Model Ratatouille),这是一种新策略,用于回收同一基础模型在多样化辅助任务上的多个微调结果。具体来说,我们将这些辅助权重重新用作目标任务上多个并行微调的初始化;然后,我们对所有微调后的权重进行平均,以获得最终模型。这种回收策略旨在通过利用辅助任务的多样性来最大化权重的多样性。实验表明,该方法在分布外泛化的参考基准 DomainBed 上改进了当前最优水平。展望未来,这项工作为可更新机器学习这一新兴范式做出了贡献,在该范式中,类似于开源软件开发,社区协作以可靠地更新机器学习模型。我们的代码已发布:https://github.com/facebookresearch/ModelRatatouille。

英文摘要:

Foundation models are redefining how AI systems are built. Practitioners now follow a standard procedure to build their machine learning solutions: from a pre-trained foundation model, they fine-tune the weights on the target task of interest. So, the Internet is swarmed by a handful of foundation models fine-tuned on many diverse tasks: these individual fine-tunings exist in isolation without benefiting from each other. In our opinion, this is a missed opportunity, as these specialized models contain rich and diverse features. In this paper, we thus propose model ratatouille, a new strategy to recycle the multiple fine-tunings of the same foundation model on diverse auxiliary tasks. Specifically, we repurpose these auxiliary weights as initializations for multiple parallel fine-tunings on the target task; then, we average all fine-tuned weights to obtain the final model. This recycling strategy aims at maximizing the diversity in weights by leveraging the diversity in auxiliary tasks. Empirically, it improves the state of the art on the reference DomainBed benchmark for out-of-distribution generalization. Looking forward, this work contributes to the emerging paradigm of updatable machine learning where, akin to open-source software development, the community collaborates to reliably update machine learning models. Our code is released: https://github.com/facebookresearch/ModelRatatouille.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑