TOGA：弱监督下的时序定位开放式视频问答
TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision
AI总结：
本文提出视觉-语言模型 TOGA，在无时序标注的弱监督设置下，利用伪标签与一致性约束联合生成开放式答案及时序定位，并在 NExT-GQA、MSVD-QA 和 ActivityNet-QA 上达到最优性能。
摘要：
我们研究在没有任何时序标注的弱监督设置下，带时序定位的视频问答（video QA）问题。给定一个视频和一个问题，我们生成一个由开始和结束时间支撑定位的开放式答案。针对该任务，我们提出了 TOGA：一种用于弱监督时序定位开放式视频问答的视觉-语言模型。我们对 TOGA 进行指令微调，使其联合生成答案和时序定位。我们在无法获得时序定位标注的弱监督设置中开展工作。我们生成时序定位的伪标签，并通过在定位响应所对应的问题与引用同一时间段的问题所生成的响应之间施加一致性约束，来确保这些标签的有效性。我们注意到，将答案与定位联合生成可以同时提升问答和定位的性能。我们在定位问答和开放式问答任务上评估 TOGA。对于定位问答，我们采用 NExT-GQA 基准，该基准旨在评估弱监督定位问答。对于开放式问答，我们采用 MSVD-QA 和 ActivityNet-QA 基准。我们在这些基准上的两类任务中均取得了最先进的性能。
英文摘要：
We address the problem of video question answering (video QA) with temporal grounding in a weakly supervised setup, without any temporal annotations. Given a video and a question, we generate an open-ended answer grounded with the start and end time. For this task, we propose TOGA: a vision-language model for Temporally Grounded Open-Ended Video QA with Weak Supervision. We instruct-tune TOGA to jointly generate the answer and the temporal grounding. We operate in a weakly supervised setup where the temporal grounding annotations are not available. We generate pseudo labels for temporal grounding and ensure the validity of these labels by imposing a consistency constraint between the question of a grounding response and the response generated by a question referring to the same temporal segment. We notice that jointly generating the answers with the grounding improves performance on question answering as well as grounding. We evaluate TOGA on grounded QA and open-ended QA tasks. For grounded QA, we consider the NExT-GQA benchmark which is designed to evaluate weakly supervised grounded question answering. For open-ended QA, we consider the MSVD-QA and ActivityNet-QA benchmarks. We achieve state-of-the-art performance for both tasks on these benchmarks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
