Exo2EgoDVC：利用网络教学视频对第一视角程序性活动进行密集视频字幕生成
Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities Using Web Instructional Videos
另 1 家 · 查看机构详情
- The University of Tokyo（东京大学）
- OMRON SINIC X Corp.（欧姆龙SINIC X公司）
- National Institute of Advanced Industrial Science and Technology (AIST)（日本产业技术综合研究所）
- LY Corporation（LY公司）
机构由 AI 辅助整理，请以论文原文为准。
浏览论文内容
中文总结 AI 辅助
针对第一视角密集视频字幕数据稀缺问题，提出Exo2EgoDVC跨视角迁移基准，构建EgoYC2数据集，提出基于对抗训练的视角不变学习方法，实现外中心网络教学视频到第一视角的知识迁移。
中文摘要 AI 辅助
我们提出了一个用于密集视频字幕生成跨视角知识迁移的新型基准，将具有外中心视角的网络教学视频模型适配到第一视角。密集视频字幕生成（预测时间段及其字幕）主要以外中心视角视频（如YouCook2）为研究对象，而第一视角视频的基准因数据稀缺受到限制。为解决视频数据有限的问题，从海量外中心视角网络视频中迁移知识是一种实用的方法。然而，由于外中心视角与第一视角存在动态视角变化，学习两者之间的对应关系十分困难。网络视频包含展示全身或手部区域的镜头，而第一视角则在不断变化，因此有必要深入研究复杂视角变化下的跨视角迁移。为此，我们首先构建了一个真实场景第一视角数据集EgoYC2，其字幕遵循YouCook2的字幕定义，可在获取两类数据集真值的前提下实现迁移学习。为缩小视角差距，我们提出了一种采用对抗训练的视角不变学习方法，该方法包含预训练和微调两个阶段。我们的实验证实了该方法在解决视角变化问题以及向第一视角迁移知识方面的有效性。我们的基准将跨视角迁移研究推进到密集视频字幕生成这一新任务领域，并为用自然语言描述第一视角视频的方法提供了展望。
英文摘要
We propose a novel benchmark for cross-view knowledge transfer of dense video captioning, adapting models from web instructional videos with exocentric views to an egocentric view. While dense video captioning (predicting time segments and their captions) is primarily studied with exocentric videos (e.g., YouCook2), benchmarks with egocentric videos are restricted due to data scarcity. To overcome the limited video availability, transferring knowledge from abundant exocentric web videos is demanded as a practical approach. However, learning the correspondence between exocentric and egocentric views is difficult due to their dynamic view changes. The web videos contain shots showing either full-body or hand regions, while the egocentric view is constantly shifting. This necessitates the in-depth study of cross-view transfer under complex view changes. To this end, we first create a real-life egocentric dataset (EgoYC2) whose captions follow the definition of YouCook2 captions, enabling transfer learning between these datasets with access to their ground-truth. To bridge the view gaps, we propose a view-invariant learning method using adversarial training, which consists of pre-training and fine-tuning stages. Our experiments confirm the effectiveness of overcoming the view change problem and knowledge transfer to egocentric views. Our benchmark pushes the study of cross-view transfer into a new task domain of dense video captioning and envisions methodologies that describe egocentric videos in natural language.
