追踪答案：利用时空线索将TextVQA从图像扩展到视频
Track the Answer: Extending TextVQA from Image to Video with Spatio-Temporal Clues
AI总结：
针对视频文本视觉问答中时空关系易被破坏的问题，提出TEA方法，通过恢复时空关系并整合OCR线索，大幅超越现有方法。
摘要：
基于视频文本的视觉问答（Video TextVQA）是一项实际任务，旨在通过联合推理给定视频中的文本和视觉信息来回答问题。受图像领域TextVQA发展的启发，现有的Video TextVQA方法利用语言模型（如T5）处理富含文本的多个帧，并以自回归方式生成答案。然而，视觉实体（包括场景文本和对象）之间的时空关系会被破坏，模型容易受到无关信息的干扰，导致推理不合理和答案不准确。为了应对这些挑战，我们提出了TEA（代表“追踪答案”）方法，该方法更好地将生成式TextVQA框架从图像扩展到视频。TEA以互补的方式恢复时空关系，并整合OCR感知线索以增强推理问题的质量。在多个公开的Video TextVQA数据集上进行的大量实验验证了我们框架的有效性和泛化能力。TEA在性能上大幅超越了现有的TextVQA方法、视频语言预训练方法和视频大语言模型。
英文摘要：
Video text-based visual question answering (Video TextVQA) is a practical task that aims to answer questions by jointly reasoning textual and visual information in a given video. Inspired by the development of TextVQA in image domain, existing Video TextVQA approaches leverage a language model (e.g. T5) to process text-rich multiple frames and generate answers auto-regressively. Nevertheless, the spatio-temporal relationships among visual entities (including scene text and objects) will be disrupted and models are susceptible to interference from unrelated information, resulting in irrational reasoning and inaccurate answering. To tackle these challenges, we propose the TEA (stands for ``\textbf{T}rack th\textbf{E} \textbf{A}nswer'') method that better extends the generative TextVQA framework from image to video. TEA recovers the spatio-temporal relationships in a complementary way and incorporates OCR-aware clues to enhance the quality of reasoning questions. Extensive experiments on several public Video TextVQA datasets validate the effectiveness and generalization of our framework. TEA outperforms existing TextVQA methods, video-language pretraining methods and video large language models by great margins.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
