arXiv 2605.307422026-05-29cs.CV

注释并非全部所需：面向无监督时间语句定位的跨模态知识迁移网络

Annotations Are Not All You Need: A Cross-modal Knowledge Transfer Network for Unsupervised Temporal Sentence Grounding

Hubei Key Laboratory of Distributed System Security（湖北分布式系统安全重点实验室）；Hubei Engineering Research Center on Big Data Security（湖北大数据安全工程研究中心）；School of Cyber Science and Engineering（网络安全学院）；Huazhong University of Science and Technology（华中科技大学）；Peking University（北京大学）；Henan University（河南大学）；The Chinese University of Hong Kong（香港中文大学）；Guangzhou University（广州大学）；Protagolabs Inc.（Protagolabs公司）

Xiang Fang, Daizong Liu, Wanlong Fang, Pan Zhou, Yu Cheng, Keke Tang, Kai Zou

备注：: Published in Findings of EMNLP 2023

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出跨模态知识迁移网络，通过从图像-名词和视频-动词任务中迁移实体感知和事件感知知识，实现无监督时间语句定位，无需配对视频-查询标注。

摘要：

本文研究时间语句定位（TSG）任务。尽管许多优秀工作在该重要课题上取得了显著成就，但它们严重依赖于大量昂贵的视频-查询配对标注，这在现实应用中需要大量人力收集。为此，本文针对更实际但更具挑战性的TSG设置：无监督时间语句定位，其中网络训练期间既没有配对视频-查询标注，也没有片段边界标注。考虑到其他跨模态任务提供了许多易于获取且廉价的标签，我们倾向于收集并将其简单的跨模态对齐知识迁移到我们的复杂场景中：1）首先从配对的图像-名词任务中探索实体感知的对象引导外观知识，并将其适应到每个独立视频帧；2）然后从配对的视频-动词任务中提取事件感知的动作表示，并通过新提出的复制-粘贴方法进一步将动作表示精炼为更实际但复杂的现实案例；3）通过将外观和动作知识调制并迁移到我们具有挑战性的无监督任务中，我们的模型可以直接利用这些通用知识来关联视频和查询，并在无需训练的情况下准确检索相关片段。在两个具有挑战性的数据集（ActivityNet Captions和Charades-STA）上的大量实验证明了我们的有效性，优于现有无监督方法，甚至与有监督方法竞争。

英文摘要：

This paper addresses the task of temporal sentence grounding (TSG). Although many respectable works have made decent achievements in this important topic, they severely rely on massive expensive video-query paired annotations, which require a tremendous amount of human effort to collect in real-world applications. To this end, in this paper, we target a more practical but challenging TSG setting: unsupervised temporal sentence grounding, where both paired video-query and segment boundary annotations are unavailable during the network training. Considering that some other cross-modal tasks provide many easily available yet cheap labels, we tend to collect and transfer their simple cross-modal alignment knowledge into our complex scenarios: 1) We first explore the entity-aware object-guided appearance knowledge from the paired Image-Noun task, and adapt them into each independent video frame; 2) Then, we extract the event-aware action representation from the paired Video-Verb task, and further refine the action representation into more practical but complicated real-world cases by a newly proposed copy-paste approach; 3) By modulating and transferring both appearance and action knowledge into our challenging unsupervised task, our model can directly utilize this general knowledge to correlate videos and queries, and accurately retrieve the relevant segment without training. Extensive experiments on two challenging datasets (ActivityNet Captions and Charades-STA) show our effectiveness, outperforming existing unsupervised methods and even competitively beating supervised works.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。