ResNetVLLM-2：解决ResNetVLLM的多模态幻觉问题
ResNetVLLM-2: Addressing ResNetVLLM's Multi-Modal Hallucinations
AI总结：
本文针对视频-语言模型ResNetVLLM的多模态幻觉问题，提出基于Lynx模型检测和检索增强生成缓解的两步策略，构建ResNetVLLM-2，在ActivityNet-QA上准确率从54.8%提升至65.3%。
摘要：
大型语言模型（LLMs）已经变革了自然语言处理（NLP）任务，但它们存在幻觉问题，即生成看似合理但事实不准确的内容。这一问题也延伸到视频-语言模型（VideoLLMs），其中文本描述可能不准确地表示视觉内容，导致多模态幻觉。在本文中，我们解决了ResNetVLLM中的幻觉问题，这是一种结合ResNet视觉编码器与LLMs的视频-语言模型。我们引入了一个两步协议：（1）一种忠实性检测策略，使用修改后的Lynx模型来评估生成字幕与地面真实视频参考之间的语义对齐；（2）一种幻觉缓解策略，使用检索增强生成（RAG），并利用在推理过程中动态构建的临时知识库。我们增强的模型ResNetVLLM-2通过交叉验证生成内容与外部知识，减少了多模态幻觉，提高了事实一致性。在ActivityNet-QA基准上的评估显示，准确率从54.8%大幅提升至65.3%，突显了我们的幻觉检测和缓解策略在增强视频-语言模型可靠性方面的有效性。
英文摘要：
Large Language Models (LLMs) have transformed natural language processing (NLP) tasks, but they suffer from hallucination, generating plausible yet factually incorrect content. This issue extends to Video-Language Models (VideoLLMs), where textual descriptions may inaccurately represent visual content, resulting in multi-modal hallucinations. In this paper, we address hallucination in ResNetVLLM, a video-language model combining ResNet visual encoders with LLMs. We introduce a two-step protocol: (1) a faithfulness detection strategy that uses a modified Lynx model to assess semantic alignment between generated captions and ground-truth video references, and (2) a hallucination mitigation strategy using Retrieval-Augmented Generation (RAG) with an ad-hoc knowledge base dynamically constructed during inference. Our enhanced model, ResNetVLLM-2, reduces multi-modal hallucinations by cross-verifying generated content against external knowledge, improving factual consistency. Evaluation on the ActivityNet-QA benchmark demonstrates a substantial accuracy increase from 54.8% to 65.3%, highlighting the effectiveness of our hallucination detection and mitigation strategies in enhancing video-language model reliability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
