SchröMind: 通过求解薛定谔桥问题减轻多模态大语言模型中的幻觉
SchröMind: Mitigating Hallucinations in Multimodal Large Language Models via Solving the Schrödinger Bridge Problem
AI总结：
SchröMind通过求解薛定谔桥问题，有效减轻多模态大语言模型中的幻觉问题，提升模型在医疗等高风险领域的应用能力。
摘要：
近期多模态大语言模型（MLLMs）在各种领域取得了显著成功。然而，其在高风险领域如医疗中的应用仍受到持续幻觉的限制，即生成的文本与或忽视视觉输入相矛盾。我们认为MLLMs可以理解图像，但难以生成准确的token序列。微小的扰动可以将注意力从真实状态转移到不真实状态，而文本生成的自回归性质往往阻止错误纠正。为了解决这个问题，我们提出了SchröMind，一个通过求解薛定谔桥问题来减少幻觉的新型框架。它通过轻量级训练在幻觉和真实激活之间建立token级映射，同时保持模型的原始能力。在POPE和MME基准上的广泛实验表明，Schrödinger在引入极小计算开销的同时实现了最先进的性能。
英文摘要：
Recent advancements in Multimodal Large Language Models (MLLMs) have achieved significant success across various domains. However, their use in high-stakes fields like healthcare remains limited due to persistent hallucinations, where generated text contradicts or ignores visual input. We contend that MLLMs can comprehend images but struggle to produce accurate token sequences. Minor perturbations can shift attention from truthful to untruthful states, and the autoregressive nature of text generation often prevents error correction. To address this, we propose SchröMind-a novel framework reducing hallucinations via solving the Schrödinger bridge problem. It establishes a token-level mapping between hallucinatory and truthful activations with minimal transport cost through lightweight training, while preserving the model's original capabilities. Extensive experiments on the POPE and MME benchmarks demonstrate the superiority of Schrödinger, which achieves state-of-the-art performance while introducing only minimal computational overhead.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
