面向医学教学视频的视觉提示时间答案定位
Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video
AI总结：
针对医学教学视频中时间答案定位任务，提出视觉提示文本跨度定位方法，利用带时间戳字幕和视觉高亮特征，在MedVidQA上以mIOU提升28.36%超越现有方法。
摘要：
视频中的时间答案定位（TAGV）是从视频中的时间句子定位（TSGV）自然衍生出的新任务。给定一个未裁剪的视频和一个文本问题，该任务旨在从视频中定位能够从语义上回答该问题的匹配片段。现有方法倾向于通过匹配文本问题所查询的视觉帧片段，采用基于视觉跨度的问题回答（QA）方法将TAGV任务形式化。然而，由于文本问题与视觉答案之间的语义特征相关性弱且存在巨大差距，采用视觉跨度预测器的现有方法在TAGV任务中表现不佳。为弥合这些差距，我们提出了一种视觉提示文本跨度定位（VPTSL）方法，该方法引入带时间戳的字幕作为段落，对输入文本问题进行文本跨度定位，并将视觉高亮特征提示到预训练语言模型（PLM）中，以增强联合语义表示。具体而言，利用上下文查询注意力在提取的文本和视觉特征之间进行跨模态交互。然后，通过视频-文本高亮获得用于视觉提示的高亮特征。为缓解文本与视觉特征之间的语义差异，我们通过使用PLM编码问题、字幕和提示的视觉高亮特征来设计文本跨度预测器。因此，TAGV任务被形式化为预测与视觉答案匹配的字幕跨度。在医学教学数据集MedVidQA上进行的大量实验表明，所提出的VPTSL在mIOU指标上以较大优势超越最先进（SOTA）方法28.36%，这证明了所提出的视觉提示和文本跨度预测器的有效性。
英文摘要：
The temporal answering grounding in the video (TAGV) is a new task naturally derived from temporal sentence grounding in the video (TSGV). Given an untrimmed video and a text question, this task aims at locating the matching span from the video that can semantically answer the question. Existing methods tend to formulate the TAGV task with a visual span-based question answering (QA) approach by matching the visual frame span queried by the text question. However, due to the weak correlations and huge gaps of the semantic features between the textual question and visual answer, existing methods adopting visual span predictor perform poorly in the TAGV task. To bridge these gaps, we propose a visual-prompt text span localizing (VPTSL) method, which introduces the timestamped subtitles as a passage to perform the text span localization for the input text question, and prompts the visual highlight features into the pre-trained language model (PLM) for enhancing the joint semantic representations. Specifically, the context query attention is utilized to perform cross-modal interaction between the extracted textual and visual features. Then, the highlight features are obtained through the video-text highlighting for the visual prompt. To alleviate semantic differences between textual and visual features, we design the text span predictor by encoding the question, the subtitles, and the prompted visual highlight features with the PLM. As a result, the TAGV task is formulated to predict the span of subtitles matching the visual answer. Extensive experiments on the medical instructional dataset, namely MedVidQA, show that the proposed VPTSL outperforms the state-of-the-art (SOTA) method by 28.36% in terms of mIOU with a large margin, which demonstrates the effectiveness of the proposed visual prompt and the text span predictor.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
