增强多模态基础模型的认知与可解释性：基于自合成数据的方法
Enhancing Cognition and Explainability of Multimodal Foundation Models with Self-Synthesized Data
AI总结：
针对大型多模态模型细粒度视觉推理与解释能力不足的问题，本文提出基于自合成数据的视觉拒绝采样框架，通过迭代生成与筛选可解释答案，提升模型在专门视觉分类任务上的准确性和可解释性。
摘要：
大型多模态模型（LMMs），或称视觉-语言模型（VLMs），在广泛的视觉任务中展现了令人印象深刻的能力。然而，它们往往在细粒度视觉推理方面存在困难，无法识别特定领域的目标，也无法为其预测提供合理的解释。为应对上述挑战，我们提出了一种新颖的视觉拒绝采样框架，利用自合成数据来提升LMMs的认知能力和可解释性。具体而言，视觉微调需要图像、查询和目标答案。我们的方法首先合成包含人类可验证视觉特征的可解释答案。这些特征基于专家定义的概念，并根据其与图像内容的一致性进行仔细筛选。在每轮微调之后，我们应用一种无需奖励模型的过滤机制，为下一轮微调选择最高质量的可解释答案。这种合成数据生成与微调的迭代过程逐步提升模型生成准确且合理解释的能力。实验结果表明，我们的方法在提升专门视觉分类任务的准确性和可解释性方面均具有有效性。
英文摘要：
Large Multimodal Models (LMMs), or Vision-Language Models (VLMs), have shown impressive capabilities in a wide range of visual tasks. However, they often struggle with fine-grained visual reasoning, failing to identify domain-specific objectives and provide justifiable explanations for their predictions. To address the above challenge, we propose a novel visual rejection sampling framework to improve the cognition and explainability of LMMs using self-synthesized data. Specifically, visual fine-tuning requires images, queries, and target answers. Our approach begins by synthesizing interpretable answers that include human-verifiable visual features. These features are based on expert-defined concepts, and carefully selected based on their alignment with the image content. After each round of fine-tuning, we apply a reward model-free filtering mechanism to select the highest-quality interpretable answers for the next round of tuning. This iterative process of synthetic data generation and fine-tuning progressively improves the model's ability to generate accurate and reasonable explanations. Experimental results demonstrate the effectiveness of our method in improving both the accuracy and explainability of specialized visual classification tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
