DisTime: 面向视频大语言模型的基于分布的时间表示
DisTime: Distribution-based Time Representation for Video Large Language Models
AI总结：
针对视频大语言模型时间定位不准、时间感知数据不足的问题，提出基于分布的轻量时间表示框架DisTime，还构建了大规模时间标注数据集InternVid-TG，在多类时间相关任务上取得SOTA性能。
摘要：
尽管通用视频理解已取得进展，但视频大语言模型（Video-LLMs）因离散时间表示和时间感知数据集有限，在精准时间定位方面仍面临挑战。现有的时间表达方法要么将时间与基于文本的数值混为一谈，要么添加一系列专用时间令牌，要么使用专门的时间定位头回归时间。为解决这些问题，我们提出了DisTime，这是一个旨在增强Video-LLMs时间理解能力的轻量级框架。DisTime采用可学习令牌构建连续时间嵌入空间，并引入基于分布的时间解码器生成时间概率分布，有效缓解边界模糊问题并保持时间连续性。此外，基于分布的时间编码器对时间戳进行重新编码，为Video-LLMs提供时间标记。为克服现有数据集的时间粒度限制，我们提出了一种自动化标注范式，结合Video-LLMs的字幕生成能力与专用时间模型的定位专长。由此构建了InternVid-TG数据集，这是一个包含17.9万段视频中125万个带时间标注事件的大规模数据集，规模是ActivityNet-Caption的55倍。大量实验表明，DisTime在三项时间敏感任务的基准测试中取得了最优性能，同时在视频问答任务中保持了具有竞争力的表现。代码和数据已发布于https://github.com/josephzpng/DisTime。
英文摘要：
Despite advances in general video understanding, Video Large Language Models (Video-LLMs) face challenges in precise temporal localization due to discrete time representations and limited temporally aware datasets. Existing methods for temporal expression either conflate time with text-based numerical values, add a series of dedicated temporal tokens, or regress time using specialized temporal grounding heads. To address these issues, we introduce DisTime, a lightweight framework designed to enhance temporal comprehension in Video-LLMs. DisTime employs a learnable token to create a continuous temporal embedding space and incorporates a Distribution-based Time Decoder that generates temporal probability distributions, effectively mitigating boundary ambiguities and maintaining temporal continuity. Additionally, the Distribution-based Time Encoder re-encodes timestamps to provide time markers for Video-LLMs. To overcome temporal granularity limitations in existing datasets, we propose an automated annotation paradigm that combines the captioning capabilities of Video-LLMs with the localization expertise of dedicated temporal models. This leads to the creation of InternVid-TG, a substantial dataset with 1.25M temporally grounded events across 179k videos, surpassing ActivityNet-Caption by 55 times. Extensive experiments demonstrate that DisTime achieves state-of-the-art performance across benchmarks in three time-sensitive tasks while maintaining competitive performance in Video QA tasks. Code and data are released at https://github.com/josephzpng/DisTime.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
