AnE: 通过锚点进化推动多模态大语言模型的推理前沿
AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution
AI总结:
提出锚点进化(AnE)范式,通过真值锚点数据策展和脚手架剥离机制,解决多模态大模型推理中的认知漂移和幻觉路径问题,显著提升推理性能。
摘要:
通过监督微调(SFT)和强化学习(RL)进行的后训练对于增强多模态大语言模型(MLLMs)的推理能力至关重要,然而现有范式由于静态数据的限制常常达到性能瓶颈。虽然当前方法利用自我反思或自我进化来突破这些界限,但它们仍然受到低质量合成数据导致的认知漂移和幻觉推理路径的影响。为了解决这些挑战,我们提出了锚点进化(AnE),一种整合了真值锚点数据策展和模型进化的新范式,在推理前沿实现了忠实且稳定的性能提升。具体来说,我们提出了真值锚点扩展,通过轨迹展开定位模型失败前沿,并利用真实数据库检索高保真锚点以进行忠实的数据策展。随后,我们引入了脚手架剥离机制来内化推理能力。该机制首先通过脚手架增强监督来锚定推理路径,以减轻直接在原始数据上进行SFT的学习复杂性和分布漂移,然后利用强化学习剥离脚手架模板,从而有效地将推理路径转化为内在模型能力。在多模态推理基准上的实验结果表明,我们的方法显著推进了模型性能前沿,在八个多模态基准上将基础模型提升了10.3%,并达到了最先进的结果。代码将公开提供。
英文摘要:
Post-training via Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) is crucial for enhancing reasoning in Multimodal Large Language Models (MLLMs), yet existing paradigms often reach a performance bottleneck due to the limitations of static data. While current methods leverage self-reflection or self-evolution to push these boundaries, they still suffer from cognitive drift and hallucinated reasoning paths caused by low-quality synthetic data. To address these challenges, we propose Anchor Evolution (AnE), a new paradigm that integrates truth-anchored data curation and model evolution, achieving faithful and steady performance gains at the reasoning frontier. Specifically, we propose Truth Anchor Expansion, which pinpoints the model failing frontier via trajectory rollouts and leverages ground-truth databases to retrieve high-fidelity anchors for faithful data curation. Subsequently, we introduce the Scaffold-Stripping Mechanism to internalize reasoning capabilities. This mechanism first anchors reasoning paths via scaffold-augmented supervision to mitigate the learning complexity and distribution drift of direct SFT on raw data, then leverages RL to strip the scaffold template, thereby effectively transitioning the reasoning paths into intrinsic model capabilities. Experimental results on multimodal reasoning benchmarks show that our method substantially advances the model performance frontier, improving the base model by 10.3\% across eight multimodal benchmarks and achieving state-of-the-art results. The code will be made publicly available.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。