MELTR：用于学习微调视频基础模型的Meta Loss Transformer
MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models
摘要：
基础模型已经显示出跨领域的出色性能和泛化能力。由于大多数关于基础模型的研究主要集中在预训练阶段，因此采用了一种朴素的策略来最小化单个任务特定的损失。然而，这样的微调方法不能充分利用对目标任务潜在有益的其他损失。因此，我们提出了Meta损失Transformer（MELTR），一个插件模块，自动和非线性地结合各种损失函数，以帮助学习的目标任务，通过辅助学习。我们制定的辅助学习作为一个双层优化问题，并提出了一个有效的优化算法的基础上近似隐式微分（AID）。为了进行评估，我们将我们的框架应用于各种视频基础模型（UniVL，Violet和All-in-one），并在所有四个下游任务上显示出显着的性能增益：文本到视频检索，视频问答，视频描述和多模态情感分析。我们的定性分析表明，MELTR充分“转换”个人损失函数和“融化"成一个有效的统一损失。代码可在https://github.com/mlvlab/MELTR上获得。
英文摘要：
Foundation models have shown outstanding performance and generalization capabilities across domains. Since most studies on foundation models mainly focus on the pretraining phase, a naive strategy to minimize a single task-specific loss is adopted for fine-tuning. However, such fine-tuning methods do not fully leverage other losses that are potentially beneficial for the target task. Therefore, we propose MEta Loss TRansformer (MELTR), a plug-in module that automatically and non-linearly combines various loss functions to aid learning the target task via auxiliary learning. We formulate the auxiliary learning as a bi-level optimization problem and present an efficient optimization algorithm based on Approximate Implicit Differentiation (AID). For evaluation, we apply our framework to various video foundation models (UniVL, Violet and All-in-one), and show significant performance gain on all four downstream tasks: text-to-video retrieval, video question answering, video captioning, and multi-modal sentiment analysis. Our qualitative analyses demonstrate that MELTR adequately `transforms' individual loss functions and `melts' them into an effective unified loss. Code is available at https://github.com/mlvlab/MELTR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
