arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.09263cs.CVcs.AI
收藏

任务驱动的探索:面向联合时刻检索与高光检测的解耦及任务间反馈

Task-Driven Exploration: Decoupling and Inter-Task Feedback for Joint Moment Retrieval and Highlight Detection

National Key Laboratory of Human-Machine Hybrid Augmented Intelligence(人机混合增强智能国家重点实验室)

Jin Yang, Ping Wei, Huan Li, Ziyang Ren

AI总结:

针对联合视频时刻检索与高光检测,本文提出任务驱动的 TaskWeave 框架,通过任务解耦、任务间反馈和任务相关联合损失提升性能,并在三个数据集上验证其有效性。

摘要:

视频时刻检索和高光检测是视频理解中两项极具价值的任务,但直到最近它们才被联合研究。尽管现有研究近期已取得令人瞩目的进展,但它们主要遵循数据驱动的自底向上范式。这种范式忽视了任务特定效应和任务间效应,导致模型性能较差。本文提出一种新颖的任务驱动自顶向下框架 TaskWeave,用于联合时刻检索和高光检测。该框架引入任务解耦单元,以捕获任务特定表示和共同表示。为研究两项任务之间的相互作用,我们提出任务间反馈机制,将一个任务的结果转换为引导掩码,以辅助另一任务。与现有方法不同,我们提出任务相关的联合损失函数来优化模型。在 QVHighlights、TVSum 和 Charades-STA 数据集上进行的综合实验和深入消融研究,证实了所提框架的有效性和灵活性。代码可在 https://github.com/EdenGabriel/TaskWeave 获取。

英文摘要:

Video moment retrieval and highlight detection are two highly valuable tasks in video understanding, but until recently they have been jointly studied. Although existing studies have made impressive advancement recently, they predominantly follow the data-driven bottom-up paradigm. Such paradigm overlooks task-specific and inter-task effects, resulting in poor model performance. In this paper, we propose a novel task-driven top-down framework TaskWeave for joint moment retrieval and highlight detection. The framework introduces a task-decoupled unit to capture task-specific and common representations. To investigate the interplay between the two tasks, we propose an inter-task feedback mechanism, which transforms the results of one task as guiding masks to assist the other task. Different from existing methods, we present a task-dependent joint loss function to optimize the model. Comprehensive experiments and in-depth ablation studies on QVHighlights, TVSum, and Charades-STA datasets corroborate the effectiveness and flexibility of the proposed framework. Codes are available at https://github.com/EdenGabriel/TaskWeave.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑