arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.03253cs.CVcs.CLcs.LG
收藏

VLLaVO:通过大语言模型缓解视觉差距

VLLaVO: Mitigating Visual Gap through LLMs

Southern University of Science and Technology(南方科技大学);Zhejiang University(浙江大学);Hong Kong University of Science and Technology(香港科技大学)

Shuhao Chen, Yulong Zhang, Weisen Jiang, Jiangang Lu, Yu Zhang

AI总结:

提出VLLaVO,结合视觉语言模型与大语言模型,将图像转为文本描述并微调大语言模型,以缓解视觉跨域学习中的领域偏移,实验验证其有效性。

摘要:

深度学习模型近期取得的进展依赖于独立同分布假设,这阻碍了它们在存在领域偏移的真实场景中的应用。为解决这一问题,跨域学习旨在提取领域不变知识以减少训练与测试数据之间的领域偏移。然而,在视觉跨域学习中,传统方法仅关注图像模态,忽视了融合文本模态的潜在益处。在本工作中,我们提出VLLaVO,将视觉语言模型和大语言模型结合作为视觉跨域学习者。VLLaVO使用视觉语言模型将图像转换为详细的文本描述。随后,在由设计的指令模板生成的源/目标域的文本描述上对大语言模型进行微调。在领域泛化和无监督领域适应设置下的大量实验结果证明了所提出方法的有效性。

英文摘要:

Recent advances achieved by deep learning models rely on the independent and identically distributed assumption, hindering their applications in real-world scenarios with domain shifts. To tackle this issue, cross-domain learning aims at extracting domain-invariant knowledge to reduce the domain shift between training and testing data. However, in visual cross-domain learning, traditional methods concentrate solely on the image modality, disregarding the potential benefits of incorporating the text modality. In this work, we propose VLLaVO, combining Vision language models and Large Language models as Visual cross-dOmain learners. VLLaVO uses vision-language models to convert images into detailed textual descriptions. A large language model is then finetuned on textual descriptions of the source/target domain generated by a designed instruction template. Extensive experimental results under domain generalization and unsupervised domain adaptation settings demonstrate the effectiveness of the proposed method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑