arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2503.08048cs.CVcs.LG
收藏

LongProLIP:一种具有长上下文文本的概率视觉-语言模型

LongProLIP: A Probabilistic Vision-Language Model with Long Context Text

NAVER AI Lab(NAVER AI实验室)

Sanghyuk Chun, Sangdoo Yun

AI总结:

本文针对 ProLIP 无法处理超过 64 长度文本的问题,提出支持 256 token 的 LongProLIP 微调方案,在提升长上下文理解的同时尽量保持零样本能力。

摘要:

最近,概率语言-图像预训练(Probabilistic Language-Image Pre-Training,ProLIP)被提出,用于解决视觉-语言(VL)任务的多重性问题。尽管 ProLIP 模型在大规模概率表征学习方面取得了成功,但其无法处理上下文长度超过 64 的长上下文文本,这限制了它们从更长文本序列中捕捉丰富上下文信息的能力。为解决这一问题,本文提出一种面向 ProLIP 的微调策略,使其能够接受更长的文本,例如 256 个文本 token。在 Urban-1k 和 DataComp 评估套件上的实验结果表明,所提出的 LongProLIP 方案能够提升对长上下文的理解,同时最大限度降低微调带来的负面影响。我们还观察到,长上下文理解(由 Urban-1k 衡量)与通用零样本能力(由 DataComp 的评估数据集衡量)之间存在权衡。代码可在 https://github.com/naver-ai/prolip 获取。

英文摘要:

Recently, Probabilistic Language-Image Pre-Training (ProLIP) has been proposed to tackle the multiplicity issue of vision-language (VL) tasks. Despite their success in probabilistic representation learning at a scale, the ProLIP models cannot handle long context texts longer than 64 context length, which limits their ability to capture rich contextual information from longer text sequences. To address this issue, this paper proposes a fine-tuning strategy for ProLIP to accept longer texts, e.g., 256 text tokens. Experimental results on Urban-1k and the DataComp evaluation suite show that the proposed LongProLIP recipe can improve understanding of long contexts while minimizing the negative effect of fine-tuning.We also observe a trade-off between the long context understanding (measured by Urban-1k) and general zero-shot capability (measured by evaluation datasets by DataComp). Code is available at https://github.com/naver-ai/prolip

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑