幻觉缓解提示长期视频理解
Hallucination Mitigation Prompts Long-term Video Understanding
AI总结：
针对长视频理解中的幻觉问题，本文提出基于问题引导帧采样、Q-former查询注入及思维链生成的缓解流程，在MovieChat上大幅超越基线并获CVPR LOVEU 2024挑战赛第三名。
摘要：
近年来，多模态大语言模型在视频理解任务上取得了显著进展。然而，它们理解未经处理的长视频的能力仍然非常有限，主要原因在于难以支撑巨大的内存开销。尽管现有方法通过聚合帧在内存与信息之间取得平衡，但不可避免地引入严重的幻觉问题。为解决该问题，本文基于现有MLLM构建了一套全面的幻觉缓解流程。具体而言，我们使用CLIP Score结合问题引导帧采样过程，选择与问题相关的关键帧。随后，我们将问题信息注入图像Q-former的查询中，以获取更重要的视觉特征。最后，在答案生成阶段，我们利用思维链和上下文学习技术显式控制答案生成。值得一提的是，在断点模式下，我们发现图像理解模型取得了优于视频理解模型的结果。因此，我们通过比较机制聚合两类模型的答案。最终，我们在MovieChat数据集上的全局模式和断点模式分别取得84.2%和62.9%的成绩，较官方基线模型分别高出29.1%和24.1%。此外，所提方法在CVPR LOVEU 2024长期视频问答挑战赛中获得第三名。代码可在https://github.com/lntzm/CVPR24Track-LongVideo获取。
英文摘要：
Recently, multimodal large language models have made significant advancements in video understanding tasks. However, their ability to understand unprocessed long videos is very limited, primarily due to the difficulty in supporting the enormous memory overhead. Although existing methods achieve a balance between memory and information by aggregating frames, they inevitably introduce the severe hallucination issue. To address this issue, this paper constructs a comprehensive hallucination mitigation pipeline based on existing MLLMs. Specifically, we use the CLIP Score to guide the frame sampling process with questions, selecting key frames relevant to the question. Then, We inject question information into the queries of the image Q-former to obtain more important visual features. Finally, during the answer generation stage, we utilize chain-of-thought and in-context learning techniques to explicitly control the generation of answers. It is worth mentioning that for the breakpoint mode, we found that image understanding models achieved better results than video understanding models. Therefore, we aggregated the answers from both types of models using a comparison mechanism. Ultimately, We achieved 84.2\% and 62.9\% for the global and breakpoint modes respectively on the MovieChat dataset, surpassing the official baseline model by 29.1\% and 24.1\%. Moreover the proposed method won the third place in the CVPR LOVEU 2024 Long-Term Video Question Answering Challenge. The code is avaiable at https://github.com/lntzm/CVPR24Track-LongVideo
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
