面向基于文本的视频问答的场景文本定位
Scene-Text Grounding for Text-Based Video Question Answering
AI总结：
针对现有TextVideoQA决策不透明、依赖场景文本识别的问题，本文提出带定位的TextVideoQA任务与T2S-QA模型，构建ViTXT-GQA数据集，实验表明现有技术存在局限，场景文本识别是主要挑战。
摘要：
现有基于文本的视频问答（TextVideoQA）研究因决策过程不透明、过度依赖场景文本识别而饱受诟病。本文提出研究带定位的TextVideoQA（Grounded TextVideoQA），强制模型在回答问题的同时，对相关场景文本区域进行时空定位，从而将问答与场景文本识别解耦，推动可解释问答方向的研究。该任务具有三重意义：其一，它鼓励模型利用场景文本证据而非其他捷径进行答案预测；其二，它直接将场景文本区域作为可视化答案，规避了严格字符串匹配导致的答案评估失效问题；其三，它分离了VideoQA和场景文本识别各自固有的挑战，能够诊断预测失败的根本原因，例如是问答错误还是场景文本识别错误。为实现带定位的TextVideoQA，我们提出T2S-QA模型，该模型采用解耦的时域到空域对比学习策略，用于弱监督场景文本定位和带定位的TextVideoQA。为便于评估，我们构建了新数据集ViTXT-GQA，该数据集包含729个视频、2000个问题对应的2200个时间片段内的52000个场景文本边界框。基于ViTXT-GQA，我们开展了大量实验，证明现有技术在带定位的TextVideoQA任务中存在严重局限性。尽管T2S-QA取得了更优的结果，但其与人类表现之间仍存在较大性能差距，有充足的提升空间。我们对理想场景文本输入的进一步分析表明，主要挑战来自场景文本识别。为推动带定位的TextVideoQA研究，我们的数据集和代码可在https://github.com/zhousheng97/ViTXT-GQA.git获取。
英文摘要：
Existing efforts in text-based video question answering (TextVideoQA) are criticized for their opaque decisionmaking and heavy reliance on scene-text recognition. In this paper, we propose to study Grounded TextVideoQA by forcing models to answer questions and spatio-temporally localize the relevant scene-text regions, thus decoupling QA from scenetext recognition and promoting research towards interpretable QA. The task has three-fold significance. First, it encourages scene-text evidence versus other short-cuts for answer predictions. Second, it directly accepts scene-text regions as visual answers, thus circumventing the problem of ineffective answer evaluation by stringent string matching. Third, it isolates the challenges inherited in VideoQA and scene-text recognition. This enables the diagnosis of the root causes for failure predictions, e.g., wrong QA or wrong scene-text recognition? To achieve Grounded TextVideoQA, we propose the T2S-QA model that highlights a disentangled temporal-to-spatial contrastive learning strategy for weakly-supervised scene-text grounding and grounded TextVideoQA. To facilitate evaluation, we construct a new dataset ViTXT-GQA which features 52K scene-text bounding boxes within 2.2K temporal segments related to 2K questions and 729 videos. With ViTXT-GQA, we perform extensive experiments and demonstrate the severe limitations of existing techniques in Grounded TextVideoQA. While T2S-QA achieves superior results, the large performance gap with human leaves ample space for improvement. Our further analysis of oracle scene-text inputs posits that the major challenge is scene-text recognition. To advance the research of Grounded TextVideoQA, our dataset and code are at https://github.com/zhousheng97/ViTXT-GQA.git
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
