时间的本质：揭示视觉-语言模型中的时间结构
A Matter of Time: Revealing the Structure of Time in Vision-Language Models
AI总结：
该研究聚焦视觉-语言模型的时间感知问题，构建含超万张图像的TIME10k基准，评估37个VLM后发现其嵌入空间中时间呈低维非线性流形结构，据此提出的时间线表征方法在时间推理任务上效率与准确率均优于提示基线。
摘要：
CLIP等大规模视觉-语言模型（VLM）凭借其具备泛化性与表达力的多模态表征得到广泛应用。通过利用带有多样化文本元数据的大规模训练数据，VLM获得了开放词汇能力，能够解决训练范围之外的任务。本文研究VLM的时间感知能力，评估其将视觉内容在时间维度上定位的能力。我们提出了TIME10k，这是一个包含超过10000张带有时间真值标注图像的基准数据集，并通过一种新方法评估了37个VLM的时间感知能力。研究发现，时间信息在VLM嵌入空间中沿低维非线性流形结构化分布。基于这一洞见，我们提出了从嵌入空间中推导显式“时间线”表征的方法。这些表征对时间及其时间先后顺序进行建模，从而为时间推理任务提供支持。我们的时间线方法在计算效率更高的同时，准确率达到了与基于提示的基线相当甚至更优的水平。所有代码和数据可在https://tekayanidham.github.io/timeline-page/获取。
英文摘要：
Large-scale vision-language models (VLMs) such as CLIP have gained popularity for their generalizable and expressive multimodal representations. By leveraging large-scale training data with diverse textual metadata, VLMs acquire open-vocabulary capabilities, solving tasks beyond their training scope. This paper investigates the temporal awareness of VLMs, assessing their ability to position visual content in time. We introduce TIME10k, a benchmark dataset of over 10,000 images with temporal ground truth, and evaluate the time-awareness of 37 VLMs by a novel methodology. Our investigation reveals that temporal information is structured along a low-dimensional, non-linear manifold in the VLM embedding space. Based on this insight, we propose methods to derive an explicit ``timeline'' representation from the embedding space. These representations model time and its chronological progression and thereby facilitate temporal reasoning tasks. Our timeline approaches achieve competitive to superior accuracy compared to a prompt-based baseline while being computationally efficient. All code and data are available at https://tekayanidham.github.io/timeline-page/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
