面向视频-语言对齐的不确定性引导自问自答方法
Uncertainty-Guided Self-Questioning and Answering for Video-Language Alignment
AI总结：
本文提出BoViLA自训练框架，通过LLM自问自答扩充VideoQA训练样本，并利用证据深度学习估计不确定性以过滤低质量问题，从而改善视频-语言模态对齐，在多个基准上超越现有方法。
摘要：
多模态模型的发展正在快速推进，其中一些模型展现出卓越的能力。然而，视频-文本对的标注仍然昂贵且不足。以视频问答（VideoQA）任务为例，人工标注的问题和答案往往只覆盖视频的一部分，因为相应的文本通常较短且单调，导致视频信息利用不充分。为了解决这一问题，我们提出了一种自举式视频-语言对齐框架（BoViLA），这是一种自训练方法，在训练过程中通过基于大语言模型（LLM）的自问自答来扩充问题样本，从而帮助模型更充分地利用视频信息和LLM的内部知识，以改善模态对齐。然而，低质量的自生成问题反而可能污染性能，尤其是在训练早期阶段，正如我们在实验中所观察到的那样。为了过滤质量较差的自生成问题，我们引入证据深度学习（EDL）来估计不确定性，并通过评估上下文中的模态对齐程度来判断自生成问题的质量。据我们所知，这项工作是首次探索基于LLM的自训练框架用于模态对齐。我们在五个具有代表性的VideoQA基准上评估了BoViLA，其性能优于多个当前最先进的方法，证明了其有效性和通用性。此外，我们对自训练框架和基于EDL的不确定性过滤机制进行了广泛分析。代码将予以公开。
英文摘要：
The development of multi-modal models has been rapidly advancing, with some demonstrating remarkable capabilities. However, annotating video-text pairs remains expensive and insufficient. Take video question answering (VideoQA) tasks as an example, human annotated questions and answers often cover only part of the video, since the corresponding text is often short and monotonous, leading to underutilization of video. To address this, we propose a Bootstrapping Video-Language Alignment framework (BoViLA), a self-training method that augments question samples during training process through LLM-based self-questioning and answering, which help model exploit video information and the internal knowledge of LLMs more thoroughly to improve modality alignment. However, low-quality self-generated questions may instead contaminate the performance, especially in the early stages of training, as we have observed in our experiments. To filter bad self-generated questions, we introduce Evidential Deep Learning (EDL) to estimate uncertainty and assess the quality of self-generated questions by evaluating the modality alignment within the context. To the best of our knowledge, this work is the first to explore LLM-based self-training frameworks for modality alignment. We evaluate BoViLA on five strong VideoQA benchmarks, where it outperforms several state-of-the-art methods and demonstrate its effectiveness and generality. Additionally, we provide extensive analyses of the self-training framework and the EDL-based uncertainty filtering mechanism. The code will be made available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
