arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2007.10040cs.CLcs.AI
收藏

视频中知识图谱的提取

Knowledge Graph Extraction from Videos

Louis Mahon, Eleonora Giunchiglia, Bowen Li, Thomas Lukasiewicz

摘要:

几乎所有现有的用于自动视频注释(或字幕)的技术都使用自然语言句子来描述视频。然而,这有几个缺点:(i)很难在自动数据处理中进一步使用所生成的自然语言注释,(ii)生成自然语言注释需要解决生成语义上精确且语法上正确的自然语言句子的困难子任务,这实际上与视频注释的任务无关,(iii)难以定量地测量性能,作为标准度量(例如,准确性和F1分数)是不适用的,和(iv)注释是语言特定的。本文提出了从视频中提取知识图谱的新任务,以给定视频的内容的知识图谱的形式产生描述。由于不存在用于此任务的数据集,因此我们还包括一种自动生成它们的方法,从使用自然语言注释视频的数据集开始。然后,我们描述了从视频中提取知识图谱的初始深度学习模型,并报告了MSVD* 和MSR-VTT* 的结果,这两个数据集是使用我们的方法从MSVD和MSR-VTT获得的。

英文摘要:

Nearly all existing techniques for automated video annotation (or captioning) describe videos using natural language sentences. However, this has several shortcomings: (i) it is very hard to then further use the generated natural language annotations in automated data processing, (ii) generating natural language annotations requires to solve the hard subtask of generating semantically precise and syntactically correct natural language sentences, which is actually unrelated to the task of video annotation, (iii) it is difficult to quantitatively measure performance, as standard metrics (e.g., accuracy and F1-score) are inapplicable, and (iv) annotations are language-specific. In this paper, we propose the new task of knowledge graph extraction from videos, i.e., producing a description in the form of a knowledge graph of the contents of a given video. Since no datasets exist for this task, we also include a method to automatically generate them, starting from datasets where videos are annotated with natural language. We then describe an initial deep-learning model for knowledge graph extraction from videos, and report results on MSVD* and MSR-VTT*, two datasets obtained from MSVD and MSR-VTT using our method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑