DialCLIP:赋能 CLIP 作为多模态对话检索器
DialCLIP: Empowering CLIP as Multi-Modal Dialog Retriever
AI总结:
本文提出参数高效的提示调优方法 DialCLIP,通过多模态上下文提示生成器、领域提示及多专家设计,在仅调优 0.04% 参数下于两个基准数据集上取得最先进的多模态对话检索性能。
摘要:
近期,预训练视觉-语言模型的实质性进展极大地提升了多模态对话系统的能力。这些模型通过在下游任务上进行微调,展现出了显著的性能提升。然而,现有的预训练模型主要侧重于有效捕捉视觉与语言模态之间的对齐,往往忽略了对话上下文的复杂特性。在本文中,我们提出了一种名为 DialCLIP 的参数高效的提示调优方法,用于多模态对话检索。具体而言,我们的方法引入了一个多模态上下文提示生成器来学习上下文特征,随后将其蒸馏到预训练视觉-语言模型 CLIP 内部的提示中。此外,我们引入了领域提示以缓解下游对话数据带来的差异。为了促进多种类型的检索,我们还设计了多个专家来学习从 CLIP 输出到多模态表示空间的映射,每个专家负责一种特定的检索类型。大量实验表明,DialCLIP 仅通过调优 0.04% 的总参数,便在两个广泛认可的基准数据集(即 PhotoChat 和 MMDialog)上取得了最先进的性能。这些结果突显了我们提出方法的有效性和高效性,强调了其推动多模态对话检索领域发展的潜力。
英文摘要:
Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。