揭示不可见之物：用隐喻为视频生成字幕
Unveiling the Invisible: Captioning Videos with Metaphors
AI总结：
本文提出视频隐喻字幕生成新任务，构建含705个视频、2115条字幕的数据集与ACD指标，并提出低资源GIT-LLaVA系统，性能媲美SoTA模型。
摘要：
隐喻是我们日常生活中常用的交流工具。文本形式隐喻的检测与生成已得到广泛研究，但其他形式的隐喻仍探索不足。近期研究表明，视觉-语言（Vision-Language，VL）模型无法理解表情包和广告中的视觉隐喻。截至目前，还没有针对视频中隐喻等复杂语言现象开展的探测研究。因此，我们在工作中提出了一项新的VL任务，即描述视频中存在的隐喻。为推动这一新任务，我们构建并发布了一个人工创建的数据集，包含705个视频和2115条人工撰写的字幕，同时提出名为平均概念距离（Average Concept Distance，ACD）的新指标，用于自动评估所生成隐喻的创造性。我们还提出了一个新颖的低资源视频隐喻字幕生成系统GIT-LLaVA，其在所提出任务上取得了与SoTA视频语言模型相当的性能。我们对现有视频语言模型在该任务上的表现进行了综合分析，并公开数据集、模型和基准测试结果，以支持进一步研究。
英文摘要：
Metaphors are a common communication tool used in our day-to-day life. The detection and generation of metaphors in textual form have been studied extensively but metaphors in other forms have been under-explored. Recent studies have shown that Vision-Language (VL) models cannot understand visual metaphors in memes and adverts. As of now, no probing studies have been done that involve complex language phenomena like metaphors with videos. Hence, we introduce a new VL task of describing the metaphors present in the videos in our work. To facilitate this novel task, we construct and release a manually created dataset with 705 videos and 2115 human-written captions, along with a new metric called Average Concept Distance (ACD), to automatically evaluate the creativity of the metaphors generated. We also propose a novel low-resource video metaphor captioning system: GIT-LLaVA, which obtains comparable performance to SoTA video language models on the proposed task. We perform a comprehensive analysis of existing video language models on this task and publish our dataset, models, and benchmark results to enable further research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
