COM Kitchens：一个作为视觉-语言基准的未经编辑俯视视频数据集
COM Kitchens: An Unedited Overhead-view Video Dataset as a Vision-Language Benchmark
AI总结：
本文构建 COM Kitchens 俯视烹饪视频数据集，并提出 OnRR 检索与 DVC-OV 字幕任务，用于评估现有网络视频 SOTA 方法。
摘要：
程序性视频理解正受到视觉与语言领域的关注。基于深度学习的视频分析需要大量数据。因此，现有工作通常使用网络视频作为训练资源，这使得从原始视频观测中查询教学内容具有挑战性。为解决这一问题，我们提出了一个新数据集 COM Kitchens。该数据集由智能手机拍摄的未经编辑的俯视视频组成，参与者在视频中根据给定食谱进行食物准备。由于高昂的相机设置成本，固定视角视频数据集往往缺乏环境多样性。我们使用现代广角智能手机镜头以俯视视角覆盖从水槽到灶台的烹饪台面，无需现场协助即可捕获活动。通过这种设置，我们向参与者分发智能手机，收集了一个多样化的数据集。基于该数据集，我们提出了新颖的视频到文本检索任务 Online Recipe Retrieval（OnRR，在线食谱检索），以及新的视频字幕领域 Dense Video Captioning on unedited Overhead-View videos（DVC-OV，未经编辑俯视视频的密集视频字幕生成）。我们的实验验证了当前基于网络视频的 SOTA 方法在处理这些任务时的能力与局限。
英文摘要：
Procedural video understanding is gaining attention in the vision and language community. Deep learning-based video analysis requires extensive data. Consequently, existing works often use web videos as training resources, making it challenging to query instructional contents from raw video observations. To address this issue, we propose a new dataset, COM Kitchens. The dataset consists of unedited overhead-view videos captured by smartphones, in which participants performed food preparation based on given recipes. Fixed-viewpoint video datasets often lack environmental diversity due to high camera setup costs. We used modern wide-angle smartphone lenses to cover cooking counters from sink to cooktop in an overhead view, capturing activity without in-person assistance. With this setup, we collected a diverse dataset by distributing smartphones to participants. With this dataset, we propose the novel video-to-text retrieval task Online Recipe Retrieval (OnRR) and new video captioning domain Dense Video Captioning on unedited Overhead-View videos (DVC-OV). Our experiments verified the capabilities and limitations of current web-video-based SOTA methods in handling these tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
