DiffusionVMR：用于联合视频片段检索与高光检测的扩散模型
DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection
AI总结：
本文提出 DiffusionVMR 框架，将视频片段检索与高光检测统一为条件扩散去噪生成过程，通过由粗到细迭代优化边界，并在五个基准上验证其有效性与灵活性。
摘要：
在视频内容激增的当下，视频片段检索与高光检测受到关注，其目标是根据用户特定查询定位片段并估计视频剪辑的相关性。鉴于视频内容在时间上具有连续性，视频中的时间事件之间往往缺乏清晰边界。这种边界模糊性使模型难以学习文本与视频剪辑之间的对应关系，导致现有方法在预测目标片段时性能欠佳。为缓解该问题，本文提出从去噪生成的视角联合解决这两项任务，并通过由粗到细的迭代细化清晰定位目标边界。具体而言，本文提出一种名为 DiffusionVMR 的新型框架，通过结合扩散模型，将这两项任务重新定义为统一的条件去噪生成过程。训练期间，向真实标注加入高斯噪声进行破坏，并生成带噪候选作为输入；模型被训练以逆转这一加噪过程。在推理阶段，DiffusionVMR 直接从高斯噪声开始，逐步将建议从噪声细化为有意义的输出。值得注意的是，所提出的 DiffusionVMR 继承了扩散模型的优势，可在推理过程中迭代细化结果，增强由粗到细的边界过渡。此外，DiffusionVMR 的训练与推理解耦，推理时可采用任意设置，无需与训练阶段保持一致。在两个任务（片段检索和/或高光检测）上，针对五个广泛使用的基准（即 QVHighlight、Charades-STA、TACoS、YouTubeHighlights 和 TVSum）开展的大量实验，证明了所提出 DiffusionVMR 的有效性与灵活性。
英文摘要：
Video moment retrieval and highlight detection have received attention in the current era of video content proliferation, aiming to localize moments and estimate clip relevances based on user-specific queries. Given that the video content is continuous in time, there is often a lack of clear boundaries between temporal events in a video. This boundary ambiguity makes it challenging for the model to learn text-video clip correspondences, resulting in the subpar performance of existing methods in predicting target segments. To alleviate this problem, we propose to solve the two tasks jointly from the perspective of denoising generation. Moreover, the target boundary can be localized clearly by iterative refinement from coarse to fine. Specifically, a novel framework, DiffusionVMR, is proposed to redefine the two tasks as a unified conditional denoising generation process by combining the diffusion model. During training, Gaussian noise is added to corrupt the ground truth, with noisy candidates produced as input. The model is trained to reverse this noise addition process. In the inference phase, DiffusionVMR initiates directly from Gaussian noise and progressively refines the proposals from the noise to the meaningful output. Notably, the proposed DiffusionVMR inherits the advantages of diffusion models that allow for iteratively refined results during inference, enhancing the boundary transition from coarse to fine. Furthermore, the training and inference of DiffusionVMR are decoupled. An arbitrary setting can be used in DiffusionVMR during inference without consistency with the training phase. Extensive experiments conducted on five widely-used benchmarks (i.e., QVHighlight, Charades-STA, TACoS, YouTubeHighlights and TVSum) across two tasks (moment retrieval and/or highlight detection) demonstrate the effectiveness and flexibility of the proposed DiffusionVMR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
