arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2005.08646cs.CVeess.IV
收藏

角色至关重要:视频故事理解与客户感知关系

Character Matters: Video Story Understanding with Character-Aware Relations

Rutgers University(罗格斯大学);The Chinese University of Hong Kong(香港中文大学);Amazon AI(亚马逊人工智能)

Shijie Geng, Ji Zhang, Zuohui Fu, Peng Gao, Hang Zhang, Gerard de Melo

摘要:

与短视频和GIF不同,视频故事包含清晰的情节和主要人物列表。如果不确定出现的人物和人物名字之间的联系,模型就无法真正理解情节。视频故事问答(VSQA)提供了一种有效的方法来衡量模型的更高层次的理解能力。然而,目前的VSQA方法只是从场景中提取通用的视觉特征。通过这种方法,他们仍然倾向于学习肤浅的相关性。为了真正理解谁对谁做了什么,我们提出了一个新的模型,不断完善的字符意识的关系。该模型特别考虑了视频故事中的角色,以及连接不同角色和对象的关系。基于这些信号,我们的框架,使弱监督的人脸命名,通过多实例共现匹配,并支持高层次的推理,利用Transformer结构。我们在TVQA数据集中的六个不同的电视节目上对我们的模型进行了训练和测试,TVQA数据集是迄今为止VSQA最大的也是唯一公开可用的数据集。我们通过广泛的消融研究在TVQA数据集上验证了我们提出的方法。

英文摘要:

Different from short videos and GIFs, video stories contain clear plots and lists of principal characters. Without identifying the connection between appearing people and character names, a model is not able to obtain a genuine understanding of the plots. Video Story Question Answering (VSQA) offers an effective way to benchmark higher-level comprehension abilities of a model. However, current VSQA methods merely extract generic visual features from a scene. With such an approach, they remain prone to learning just superficial correlations. In order to attain a genuine understanding of who did what to whom, we propose a novel model that continuously refines character-aware relations. This model specifically considers the characters in a video story, as well as the relations connecting different characters and objects. Based on these signals, our framework enables weakly-supervised face naming through multi-instance co-occurrence matching and supports high-level reasoning utilizing Transformer structures. We train and test our model on the six diverse TV shows in the TVQA dataset, which is by far the largest and only publicly available dataset for VSQA. We validate our proposed approach over TVQA dataset through extensive ablation study.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑