TRANSPORTER：从VLM流形转移视觉语义
TRANSPORTER: Transferring Visual Semantics from VLM Manifolds
AI总结：
本文提出TRANSPORTER方法，通过logits-to-video任务生成视频，揭示VLM预测背后的规则，为模型可解释性提供新方向。
摘要：
视频理解模型如何获得答案？尽管当前视觉语言模型（VLMs）能处理复杂场景，但理解并控制其内部过程仍是挑战。受文本到视频（T2V）生成模型的启发，本文引入logits-to-video（L2V）任务及模型无关方法TRANSPORTER，生成捕捉VLM预测底层规则的视频。借助T2V模型的高视觉保真度，TRANSPORTER学习最优传输耦合到VLM的高语义嵌入空间。logit分数定义嵌入方向用于条件视频生成。TRANSPORTER生成反映标题变化的视频，涵盖多样物体属性、动作副词和场景上下文。跨VLM的定量和定性评估表明，L2V能提供高保真、新颖的模型可解释性方向。
英文摘要：
How do video understanding models acquire their answers? Although current Vision Language Models (VLMs) reason over complex scenes with diverse objects, action performances, and scene dynamics, understanding and controlling their internal processes remains an open challenge. Motivated by recent advancements in text-to-video (T2V) generative models, this paper introduces a logits-to-video (L2V) task alongside a model-independent approach, TRANSPORTER, to generate videos that capture the underlying rules behind VLMs' predictions. Given the high-visual-fidelity produced by T2V models, TRANSPORTER learns an optimal transport coupling to VLM's high-semantic embedding spaces. In turn, logit scores define embedding directions for conditional video generation. TRANSPORTER generates videos that reflect caption changes over diverse object attributes, action adverbs, and scene context. Quantitative and qualitative evaluations across VLMs demonstrate that L2V can provide a fidelity-rich, novel direction for model interpretability that has not been previously explored.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
