DenseStep2M: 一种可扩展且无需训练的密集指令视频标注流水线
DenseStep2M: A Scalable, Training-Free Pipeline for Dense Instructional Video Annotation
AI总结：
本文提出无需训练的DenseStep2M流水线，通过分割视频、过滤对齐不佳内容并利用先进多模态模型生成高质量步骤注释，构建了包含10万视频和200万步骤的大型数据集，支持长期视频理解。
摘要：
长期视频理解需要解释复杂的时序事件并推理过程性活动。尽管如HowTo100M这样的教学视频语料库为模型训练提供了丰富的资源，但存在噪声的ASR转录和叙述与视觉内容之间的时间对齐不一致等显著挑战。本文引入了自动化的、无需训练的流水线，从真实世界中的教学视频中提取高质量的过程注释。我们的方法将视频分割为连贯的镜头，过滤对齐不佳的内容，并利用最先进的多模态和大语言模型（Qwen2.5-VL和DeepSeek-R1）生成结构化且时间相关的步骤。该流水线产生了DenseStep2M，一个包含约10万视频和200万详细教学步骤的大型数据集，旨在支持全面的长视频理解。为了严格评估我们的流水线，我们精心编纂了DenseCaption100，一个高质量的人工编写注释基准。评估显示我们的自动生成步骤与人工注释有很强的一致性。此外，我们验证了DenseStep2M在三个核心下游任务中的效用：密集视频注释、过程步骤定位和跨模态检索。在DenseStep2M上微调的模型在注释质量和时间定位方面取得了显著提升，同时在眼动、外视和混合视角领域表现出稳健的零样本泛化能力。这些结果凸显了DenseStep2M在促进高级多模态对齐和长期活动推理方面的有效性。我们的数据集可在https://huggingface.co/datasets/mingjige/DenseStep2M获取。
英文摘要：
Long-term video understanding requires interpreting complex temporal events and reasoning over procedural activities. While instructional video corpora, like HowTo100M, offer rich resources for model training, they present significant challenges, including noisy ASR transcripts and inconsistent temporal alignments between narration and visual content. In this work, we introduce an automated, training-free pipeline to extract high-quality procedural annotations from in-the-wild instructional videos. Our approach segments videos into coherent shots, filters poorly aligned content, and leverages state-of-the-art multimodal and large language models (Qwen2.5-VL and DeepSeek-R1) to generate structured, temporally grounded procedural steps. This pipeline yields DenseStep2M, a large-scale dataset comprising approximately 100K videos and 2M detailed instructional steps, designed to support comprehensive long-form video understanding. To rigorously evaluate our pipeline, we curate DenseCaption100, a benchmark of high-quality, human-written captions. Evaluations demonstrate strong alignment between our auto-generated steps and human annotations. Furthermore, we validate the utility of DenseStep2M across three core downstream tasks: dense video captioning, procedural step grounding, and cross-modal retrieval. Models fine-tuned on DenseStep2M achieve substantial gains in captioning quality and temporal localization, while exhibiting robust zero-shot generalization across egocentric, exocentric, and mixed-perspective domains. These results underscore the effectiveness of DenseStep2M in facilitating advanced multimodal alignment and long-term activity reasoning. Our dataset is available at https://huggingface.co/datasets/mingjige/DenseStep2M.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
