Detecting customized moments and highlights from videos given natural language (NL) user queries is an important but under-studied topic. One of the challenges in pursuing this direction is the lack of annotated data. To address this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset. It consists of over 10,000 YouTube videos, covering a wide range of topics, from everyday activities and travel in lifestyle vlog videos to social and political activities in news videos. Each video in the dataset is annotated with: (1) a human-written free-form NL query, (2) relevant moments in the video w.r.t. the query, and (3) five-point scale saliency scores for all query-relevant clips. This comprehensive annotation enables us to develop and evaluate systems that detect relevant moments as well as salient highlights for diverse, flexible user queries. We also present a strong baseline for this task, Moment-DETR, a transformer encoder-decoder model that views moment retrieval as a direct set prediction problem, taking extracted video and query representations as inputs and predicting moment coordinates and saliency scores end-to-end. While our model does not utilize any human prior, we show that it performs competitively when compared to well-engineered architectures. With weakly supervised pretraining using ASR captions, MomentDETR substantially outperforms previous methods. Lastly, we present several ablations and visualizations of Moment-DETR. Data and code is publicly available at https://github.com/jayleicn/moment_detr
Context-Enhanced Video Moment Retrieval with Large Language Models
School of Cyber Science and Engineering, Southeast University（东南大学网络空间科学与工程学院）；School of Computer Science and Software Engineering, University of Western Australia（西澳大学计算机科学与软件工程学院）；School of Computer Science and Engineering, Southeast University（东南大学计算机科学与工程学院）
GIRL-DETR: Gradient-Isolated Reinforcement Learning for Video Moment Retrieval
College of Electronics and Information Engineering, Sichuan University（四川大学电子信息工程学院）；School of Intelligence Science and Technology, Nanjing University（南京大学智能科学与技术学院）
Saying What You're Looking For: Linguistics Meets Video Search
Purdue University（普渡大学）
我们提出了一种方法来搜索大型视频语料库中的视频剪辑描述的自然语言查询的形式的句子。这种方法使用组合语义来编码在其他系统中丢失的微妙含义，例如具有相同单词但含义完全不同的两个句子之间的差异：“The person rode the horse}与\emph {The horse rode the person”。给定一个视频句子对和一个自然语言解析器，以及一个描述语义查询空间的语法，我们会产生一个分数，表明视频对句子的描述有多好。我们为语料库中的每个视频片段生成这样的分数，并返回片段的排名列表。此外，该方法同时解决了两个基本问题：检测和跟踪对象，并识别这些轨迹是否描述了查询。因为跟踪和目标检测都是不可靠的，所以这使用关于预期的初步查询的知识来将跟踪器集中在相关参与者上，并确保所得到的轨迹由初步查询描述。虽然早期的工作仅限于对应于动词或名词的单字查询，我们展示了如何搜索包含多个短语（如介词短语）和修饰语（如副词）的复杂查询。我们证明了这种方法，通过搜索141个查询，涉及人和马相互作用，在10个全长好莱坞电影。
DiffusionVMR: Diffusion Model for Joint Video Moment Retrieval and Highlight Detection
School of Computer Science and Engineering, Nanjing University of Science and Technology（南京理工大学计算机科学与工程学院）；National University of Singapore（新加坡国立大学）；Department of Computer Science and Technology, Nanjing University（南京大学计算机科学与技术系）
