SDRT：通过多样化推理轨迹的自蒸馏增强视觉语言模型
SDRT: Enhance Vision-Language Models by Self-Distillation with Diverse Reasoning Traces
AI总结：
针对视觉语言模型推理能力难发挥的问题，提出含定制提示库、两步推理及架构创新的自蒸馏框架SDRT，在5个VQA数据集上显著提升基线性能。
摘要：
推理在各类任务中愈发重要。尽管思维链提示能让大语言模型有效利用推理能力，但如何发挥视觉语言模型（VLM）的推理能力仍是难题。为解决该问题，我们提出一种新型自蒸馏框架以增强模型的推理能力。该框架包含多项关键创新：首先采用为视觉推理任务定制的提示库生成多样化的上下文相关问题，并通过两步推理流程生成推理引导的响应，再将这些响应用于自蒸馏，使模型内化推理过程。此外，我们还通过多个创新组件改进模型架构，包括用于高效参数更新的干预适配器、促进模态间信息交换的跨模态跳跃连接，以及整合多个上下文问题的多样化推理结果的集成学习算法。大量实验表明，该方法在5个VQA数据集上显著提升了基线性能。
英文摘要：
Reasoning is increasingly crucial for various tasks. While chain-of-thought prompting enables large language models to leverage reasoning effectively, harnessing the reasoning capabilities of Vision-Language Models (VLMs) remains challenging. To solve this problem, we propose a novel self-distillation framework that enhances the reasoning capabilities of the model. The proposed framework introduces several key innovations. We start by employing a prompt library tailored to visual reasoning tasks to generate diverse in-context questions and utilize a two-step reasoning procedure to derive reasoning-guided responses. These responses are then used for self-distillation, enabling the model to internalize the reasoning process. Additionally, we improve the model architecture with several innovative components, including an intervention adapter for efficient parameter updates, a cross-modal skip connection to facilitate information exchange between modalities, and an ensemble learning algorithm to integrate diverse reasoning from multiple in-context questions. Extensive experiments show that our method significantly improves the baseline performance across five VQA datasets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
