长时教学视频的多句定位
Multi-Sentence Grounding for Long-term Instructional Video
AI总结：
本文提出自动化去噪流水线构建高质量教学视频数据集HowToStep，并设计基于Transformer的文本查询迭代注意力架构，实现多句时间定位，在三个基准上显著超越现有方法。
摘要：
在本文中，我们旨在建立一个自动化的、可扩展的流水线，用于对大规模教学数据集进行去噪，并构建一个具有多描述步骤监督的高质量视频-文本数据集，命名为HowToStep。我们做出以下贡献：（i）通过升级ASR系统以减少语音识别错误，并提示大型语言模型将嘈杂的ASR转录文本转换为描述性步骤，从而提高数据集中句子的质量；（ii）提出一种基于Transformer的架构，以所有文本作为查询，迭代地关注视觉特征，从而将生成的步骤与相应的视频片段进行时间对齐。为了衡量我们整理的数据集的质量，我们在其上训练模型用于多句定位任务，即给定一个长视频和关联的多条句子，同时确定它们在视频中对应的时间戳，结果表明，该模型在一系列多句定位任务上表现出优越的性能，在三个公开基准上大幅超越现有最先进的方法，即在HT-Step上提升9.0%，在HTM-Align上提升5.1%，在CrossTask上提升1.9%。所有代码、模型及生成的数据集均已公开发布。
英文摘要：
In this paper, we aim to establish an automatic, scalable pipeline for denoising the large-scale instructional dataset and construct a high-quality video-text dataset with multiple descriptive steps supervision, named HowToStep. We make the following contributions: (i) improving the quality of sentences in dataset by upgrading ASR systems to reduce errors from speech recognition and prompting a large language model to transform noisy ASR transcripts into descriptive steps; (ii) proposing a Transformer-based architecture with all texts as queries, iteratively attending to the visual features, to temporally align the generated steps to corresponding video segments. To measure the quality of our curated datasets, we train models for the task of multi-sentence grounding on it, i.e., given a long-form video, and associated multiple sentences, to determine their corresponding timestamps in the video simultaneously, as a result, the model shows superior performance on a series of multi-sentence grounding tasks, surpassing existing state-of-the-art methods by a significant margin on three public benchmarks, namely, 9.0% on HT-Step, 5.1% on HTM-Align and 1.9% on CrossTask. All codes, models, and the resulting dataset have been publicly released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
