arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.06166cs.CVcs.CL
收藏

从文本到视频的时间推理迁移

Temporal Reasoning Transfer from Text to Video

The University of Hong Kong(香港大学);Peking University(北京大学);University of California, San Diego(加州大学圣迭戈分校)

Lei Li, Yuanxin Liu, Linli Yao, Peiyuan Zhang, Chenxin An, Lean Wang, Xu Sun, Lingpeng Kong, Qi Liu

AI总结:

本研究发现Video LLMs时间推理瓶颈源于底层LLM的时间概念处理缺陷,提出纯文本时间推理迁移方法T3,无需视频数据即可提升模型时间推理性能,验证了跨域迁移的有效性。

摘要:

视频大语言模型(Video LLMs)在视频理解方面展现出可观的能力,但在追踪时间变化和推理时间关系方面仍存在困难。尽管此前研究将这一局限归因于视觉输入的时间编码效果不佳,我们的诊断研究显示,视频表征中包含的信息足以让小型探测分类器达到完美准确率。出人意料的是,我们发现Video LLMs时间推理能力的关键瓶颈在于底层LLM本身对时间概念的处理困难,这一点可由其在文本时间问答任务上的糟糕表现得到佐证。基于这一发现,我们提出了Textual Temporal reasoning Transfer(T3,文本时间推理迁移)方法。T3从现有图文数据集中合成纯文本格式的多样化时间推理任务,解决了具备复杂时间场景的视频样本稀缺的问题。值得注意的是,在未使用任何视频数据的情况下,T3增强了LongVA-7B的时间理解能力,在具有挑战性的TempCompass基准上实现了5.3的绝对准确率提升,这使得我们的模型性能超过了在28000个视频样本上训练的ShareGPT4Video-8B。此外,增强后的LongVA-7B模型在综合视频基准上取得了具有竞争力的表现。例如,它在Video-MME的时间推理任务上达到了49.7的准确率,超过了InternVL-Chat-V1.5-20B和VILA1.5-40B等强大的大规模模型。进一步分析显示,文本和视频时间任务性能之间存在强相关性,验证了将时间推理能力从文本领域迁移到视频领域的有效性。

英文摘要:

Video Large Language Models (Video LLMs) have shown promising capabilities in video comprehension, yet they struggle with tracking temporal changes and reasoning about temporal relationships. While previous research attributed this limitation to the ineffective temporal encoding of visual inputs, our diagnostic study reveals that video representations contain sufficient information for even small probing classifiers to achieve perfect accuracy. Surprisingly, we find that the key bottleneck in Video LLMs' temporal reasoning capability stems from the underlying LLM's inherent difficulty with temporal concepts, as evidenced by poor performance on textual temporal question-answering tasks. Building on this discovery, we introduce the Textual Temporal reasoning Transfer (T3). T3 synthesizes diverse temporal reasoning tasks in pure text format from existing image-text datasets, addressing the scarcity of video samples with complex temporal scenarios. Remarkably, without using any video data, T3 enhances LongVA-7B's temporal understanding, yielding a 5.3 absolute accuracy improvement on the challenging TempCompass benchmark, which enables our model to outperform ShareGPT4Video-8B trained on 28,000 video samples. Additionally, the enhanced LongVA-7B model achieves competitive performance on comprehensive video benchmarks. For example, it achieves a 49.7 accuracy on the Temporal Reasoning task of Video-MME, surpassing powerful large-scale models such as InternVL-Chat-V1.5-20B and VILA1.5-40B. Further analysis reveals a strong correlation between textual and video temporal task performance, validating the efficacy of transferring temporal reasoning abilities from text to video domains.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑