在多模态大语言模型中引导式落地思维链以实现数据高效的模型适配
Bootstrapping Grounded Chain-of-Thought in Multimodal LLMs for Data-Efficient Model Adaptation
AI总结：
针对多模态大语言模型难以在数据受限条件下适配图表等专门视觉任务的问题，本文提出GCoT方法，通过自举方式将边界框落地信息注入思维链数据以减少推理错误，并在五个专门视觉任务上验证其优于微调和蒸馏。
摘要：
多模态大语言模型（MLLMs）在使用自然语言解释图像方面展现了卓越的能力。然而，若不使用大规模数据集进行重新训练，这些模型难以适配专门的视觉任务，例如图表理解。该问题源于预训练数据集与下游数据集之间的不匹配：预训练数据集主要集中于场景和物体，但包含的关于专门的、非物体类图像（如图表和表格）的信息有限。在本文中，我们分享了一个有趣的发现：使用思维链（CoT）推理数据训练MLLM可以促进其在专门视觉任务中的模型适配，尤其是在数据受限的情况下。然而，我们发现从预训练MLLM中蒸馏得到的CoT数据存在一个关键问题，即这些数据的推理步骤中通常包含多个事实性错误。为解决该问题，我们提出了落地思维链（Grounded Chain-of-Thought，GCoT），这是一种简单的基于自举的方法，旨在将落地信息（即边界框）注入CoT数据，从根本上使推理步骤更加忠实于输入图像。我们在五个专门的视觉任务上评估了我们的方法，这些任务涵盖了包括图表、表格、收据和报告在内的多种视觉格式。结果表明，在数据受限的情况下，我们的方法相比微调和蒸馏有显著提升。
英文摘要：
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in interpreting images using natural language. However, without using large-scale datasets for retraining, these models are difficult to adapt to specialized vision tasks, e.g., chart understanding. This problem is caused by a mismatch between pre-training and downstream datasets: pre-training datasets primarily concentrate on scenes and objects but contain limited information about specialized, non-object images, such as charts and tables. In this paper, we share an interesting finding that training an MLLM with Chain-of-Thought (CoT) reasoning data can facilitate model adaptation in specialized vision tasks, especially under data-limited regimes. However, we identify a critical issue within CoT data distilled from pre-trained MLLMs, i.e., the data often contains multiple factual errors in the reasoning steps. To address the problem, we propose Grounded Chain-of-Thought (GCoT), a simple bootstrapping-based approach that aims to inject grounding information (i.e., bounding boxes) into CoT data, essentially making the reasoning steps more faithful to input images. We evaluate our approach on five specialized vision tasks, which cover a variety of visual formats including charts, tables, receipts, and reports. The results demonstrate that under data-limited regimes our approach significantly improves upon fine-tuning and distillation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
