arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2302.05738cs.LG
收藏

跨模态微调:先对齐再优化

Cross-Modal Fine-Tuning: Align then Refine

Carnegie Mellon University(卡内基梅隆大学);Hewlett Packard Enterprise(慧与公司)

Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, Ameet Talwalkar

摘要:

对大规模预训练模型的微调已经在视觉和NLP等研究充分的模式方面取得了巨大进展。然而,由于缺乏相关的预训练模型,在许多其他模式中没有观察到类似的收益。在这项工作中,我们提出了ORCA,这是一个通用的跨模态微调框架,可以将单个大规模预训练模型的适用性扩展到多种模态。ORCA通过一个先搜索再细化的工作流程来适应目标任务:给定目标输入,ORCA首先学习一个嵌入网络,将嵌入的特征分布与预训练模态对齐。然后,预训练的模型在嵌入的数据上进行微调,以利用跨模态共享的知识。通过大量的实验,我们证明了ORCA在3个基准测试中获得了最先进的结果,其中包含来自12种模式的60多个数据集,优于各种手工设计,AutoML,通用和特定任务的方法。我们通过一系列消融研究强调了数据对齐的重要性,并证明了ORCA在数据有限的情况下的实用性。

英文摘要:

Fine-tuning large-scale pretrained models has led to tremendous progress in well-studied modalities such as vision and NLP. However, similar gains have not been observed in many other modalities due to a lack of relevant pretrained models. In this work, we propose ORCA, a general cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. ORCA adapts to a target task via an align-then-refine workflow: given the target input, ORCA first learns an embedding network that aligns the embedded feature distribution with the pretraining modality. The pretrained model is then fine-tuned on the embedded data to exploit the knowledge shared across modalities. Through extensive experiments, we show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities, outperforming a wide range of hand-designed, AutoML, general-purpose, and task-specific methods. We highlight the importance of data alignment via a series of ablation studies and demonstrate ORCA's utility in data-limited regimes.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑