视觉验证增强的VLMs融合以实现高效的视觉推理
Vision Verification Enhanced Fusion of VLMs for Efficient Visual Reasoning
AI总结：
本文提出V3Fusion方法，通过融合多种视觉语言模型，利用焦点误差多样性与CKA-focal度量来提升视觉推理性能，实验显示在多个基准上优于现有模型。
摘要：
随着视觉语言模型（VLMs）数量和多样性的增加，许多研究探索了跨多个VLMs的语言基于的集成、协作和路由技术以提升多模型推理。相反，我们通过视觉和语言模态解决多样模型选择问题。我们引入了焦点误差多样性来捕捉VLMs之间的互补推理，并引入基于CKA的焦点多样性度量（CKA-focal）来衡量其视觉嵌入中的分歧。在由候选VLMs池构建的集成表面上，我们应用遗传算法有效剔除那些对融合性能无贡献的组件VLMs。我们识别出每个任务的最佳组合，并融合每个VLMs的输出，证明异构模型可以动态捕捉epistemic不确定性并缓解幻觉。我们的V3Fusion方法能够产生双焦点多样性融合预测，即使没有多数共识或多数VLMs做出错误预测时也能保持高性能。在四个流行的VLM基准（A-OKVQA、MMMU、MMMU-Pro和OCR-VQA）上进行了大量实验。结果表明，V3Fusion在MMMU上比最佳表现的VLM高出8.09%，在MMMU-Pro上高出4.87%。在生成任务中，V3Fusion在A-OKVQA和OCR-VQA上优于Intern-VL2-8b和Qwen2.5-VL-7b，这两个是当前表现最好的VLM。我们的代码和数据集可在https://github.com/sftekin/v3fusion获取。
英文摘要：
With the growing number and diversity of Vision-Language Models (VLMs), many works explore language-based ensemble, collaboration, and routing techniques across multiple VLMs to improve multi-model reasoning. In contrast, we address the diverse model selection using both vision and language modalities. We introduce focal error diversity to capture complementary reasoning across VLMs and a CKA-based focal diversity metric (CKA-focal) to measure disagreement in their visual embeddings. On the constructed ensemble surface from a pool of candidate VLMs, we applied a Genetic Algorithm to effectively prune out those component VLMs that do not add value to the fusion performance. We identify the best combination for each task as well as fuse the outputs of each VLMs in the model pool, and show that heterogeneous models can capture epistemic uncertainty dynamically and mitigate hallucinations. Our V3Fusion approach is capable of producing dual focal-diversity fused predictions with high performance for vision-language reasoning, even when there is no majority consensus or the majority of VLMs make incorrect predictions. Extensive experiments validate V3Fusion on four popular VLM benchmarks (A-OKVQA, MMMU, MMMU-Pro, and OCR-VQA). The results show that V3Fusion outperforms the best-performing VLM on MMMU by 8.09% and MMMU-Pro by 4.87% gain in accuracy. For generative tasks, V3Fusion outperforms Intern-VL2-8b and Qwen2.5-VL-7b, the top-2 VLM performers on both A-OKVQA and OCR-VQA. Our code and datasets are available at https://github.com/sftekin/v3fusion.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
