arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2607.00465cs.CVcs.CLcs.LG
收藏

StochasT:基于随机轮次深度的视觉指令微调学习

StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuning

Boston University(波士顿大学);Rutgers University(罗格斯大学)

Yuan Qing, Chengzhi Mao, Boqing Gong

AI总结:

针对视觉指令微调中多轮训练与单轮测试不匹配的问题,提出StochasT方法,通过随机分组语言任务为不同大小的轮次深度,增强模型在单轮和多轮场景下的鲁棒性。

摘要:

大型视觉语言模型(LVLMs)严重依赖视觉指令微调(VIT)来激发其多模态推理能力。然而,我们发现一个差异:VIT通常将同一图像的多项语言任务打包用于对话式多轮训练,而现有基准测试在孤立的单轮场景中评估LVLMs。模型在多轮训练中可能遭受视觉注意力衰减和上下文过拟合,使其难以在错配的测试阶段发挥全部潜力。为缩小这一差距,我们提出基于随机轮次深度(StochasT)的学习方法,该方法在保留语言任务自然顺序的同时,将同一图像的语言任务随机分组为不同大小的簇(轮次深度)。因此,虽然StochasT借鉴了ResNet的Dropout和随机深度思想,但它实际上并未丢弃任何内容,以最大化训练数据的利用率。此外,我们引入了一种基于平衡拉丁方的、与基准无关的挑战性评估机制,以衡量LVLMs在不同上下文依赖下的鲁棒性。大量实验表明,StochasT有效地赋予LVLMs在单轮和多轮使用场景中强大且协调的能力。

英文摘要:

Large Vision-Language Models (LVLMs) rely extensively on Visual Instruction Tuning (VIT) to elicit their multimodal reasoning capabilities. However, we find a discrepancy: VIT often packs multiple language tasks about the same image for conversational, multi-turn training, whereas existing benchmarks evaluate LVLMs in isolated, single-turn scenarios. The models can suffer from visual attention decay and contextual overfitting during multi-turn training, making it hard for them to realize their full potential in the mismatched test phase. To close the gap, we propose learning with Stochastic Turn Depth (StochasT), which stochastically groups language tasks for the same image into clusters of varying sizes (turn depth) while preserving their organic order. Hence, while StochasT draws on Dropout and stochastic depth for ResNets, it does not actually drop anything to maximize the utility of the training data. Furthermore, we introduce a challenging, benchmark-agnostic evaluation mechanism based on the Balanced Latin Square to measure LVLMs' robustness under varying contextual dependencies. Extensive experiments demonstrate that StochasT effectively grants LVLMs strong, harmonized capabilities for both single-turn and multi-turn use cases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑