有效利用 CLIP 生成图像和视频的情境摘要
Effectively Leveraging CLIP for Generating Situational Summaries of Images and Videos
AI总结:
本文提出 ClipSitu 及 ClipSitu XTF,利用 CLIP 嵌入和交叉注意力实现图像情境识别、定位与摘要,并扩展至视频任务。
摘要:
情境识别是指智能体基于可用信息和感官输入识别并理解各种情境或上下文的能力。它涉及解释来自环境的数据,以确定正在发生什么、涉及哪些因素以及哪些动作导致了这些情境的认知过程。在基于计算机视觉的情境识别中,这种对情境的解释被表述为一个语义角色标注问题。图像和视频中描绘的情境包含关键信息,对图像和视频字幕、多媒体检索、自主系统以及事件监测等多种应用至关重要。然而,现有方法在生成有意义且准确的预测时,往往受到歧义性和上下文缺乏的困扰。利用 CLIP 等多模态模型,我们提出了 ClipSitu,它避免了全量微调的需要,并在情境识别和定位任务中取得了最先进结果。ClipSitu 利用基于 CLIP 的图像、动词和角色嵌入来预测满足与某个动词相关的所有角色的名词,从而全面理解所描绘的场景。通过交叉注意力 Transformer,ClipSitu XTF 增强了语义角色查询与视觉 token 表示之间的联系,从而在情境识别中取得更优性能。我们还提出了一个具有近乎完美准确率的按动词角色预测模型,以创建一个端到端框架,用于为域外图像生成情境摘要。我们表明,与通用字幕相比,情境摘要使我们的 ClipSitu 模型能够生成歧义更少的结构化描述。最后,我们将 ClipSitu 扩展到视频情境识别,以展示其多功能性,并取得了与最先进方法相当的性能。
英文摘要:
Situation recognition refers to the ability of an agent to identify and understand various situations or contexts based on available information and sensory inputs. It involves the cognitive process of interpreting data from the environment to determine what is happening, what factors are involved, and what actions caused those situations. This interpretation of situations is formulated as a semantic role labeling problem in computer vision-based situation recognition. Situations depicted in images and videos hold pivotal information, essential for various applications like image and video captioning, multimedia retrieval, autonomous systems and event monitoring. However, existing methods often struggle with ambiguity and lack of context in generating meaningful and accurate predictions. Leveraging multimodal models such as CLIP, we propose ClipSitu, which sidesteps the need for full fine-tuning and achieves state-of-the-art results in situation recognition and localization tasks. ClipSitu harnesses CLIP-based image, verb, and role embeddings to predict nouns fulfilling all the roles associated with a verb, providing a comprehensive understanding of depicted scenarios. Through a cross-attention Transformer, ClipSitu XTF enhances the connection between semantic role queries and visual token representations, leading to superior performance in situation recognition. We also propose a verb-wise role prediction model with near-perfect accuracy to create an end-to-end framework for producing situational summaries for out-of-domain images. We show that situational summaries empower our ClipSitu models to produce structured descriptions with reduced ambiguity compared to generic captions. Finally, we extend ClipSitu to video situation recognition to showcase its versatility and produce comparable performance to state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。