arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2603.04415cs.CLcs.CV
收藏

双调优:面向多模态大语言模型训练的推理效能驱动的数据筛选

Dual Tuning for Reasoning Efficacy-Driven Data Curation in Multimodal LLM Training

Ant Group(蚂蚁集团)

Ruobing Zheng, Tianqi Li, Jianing Li, Qingpei Guo, Yi Yuan, Jingdong Chen

AI总结:

本文提出双调优框架,通过评估训练数据对推理训练的收益及推理训练的效能,指导多模态大语言模型的数据筛选与训练策略匹配。

摘要:

推理训练能提升大语言模型(LLMs)在数学和编程等复杂任务上的表现,但其在多样化多模态任务中的效果尚不确定。领先的团队释放平行的“Instruct”和“Thinking”模型趋势既资源密集又用户不友好。先前研究发现,推理训练的收益受多种因素影响,如基础模型能力、任务特征和链式思维(CoT)数据质量。然而,确定推理训练后何时有益以及哪些数据应支持其的系统性标准仍然缺乏。本文提出双调优,一种面向多模态LLM训练的推理效能驱动的数据筛选框架。给定一个目标任务和一个基础模型,双调优联合评估训练数据是否有益,以及当前CoT内容的推理训练是否在非推理替代方案上产生积极收益。我们将在空间、数学和多学科任务上应用双调优,并进一步分析强化学习和思考模式如何影响推理效能。双调优结果通过识别有益于推理训练的数据、更适合直接回答训练的数据以及在两种训练模式下有害的数据来指导数据筛选。我们的工作提供了选择合适训练数据和匹配后训练策略的定量标准。

英文摘要:

Reasoning post-training improves Large Language Models (LLMs) on complex tasks such as mathematics and coding, but its benefits across diverse multimodal tasks remains uncertain. The trend of releasing parallel "Instruct" and "Thinking" models by leading teams is both resource-intensive and user-unfriendly. Prior work finds that the gains from reasoning training are influenced by multiple factors, such as base model capabilities, task characteristics, and Chain-of-Thought (CoT) data quality. However, principled criteria for determining when reasoning post-training is beneficial and which data should support it are still lacking. In this paper, we propose Dual Tuning, a reasoning efficacy-driven data curation framework for multimodal LLMs training. Given a target task and a base model, Dual Tuning jointly evaluates whether the training data is beneficial and whether reasoning training with current CoT content yields positive gains over non-reasoning alternatives. We apply Dual Tuning across spatial, mathematical, and multi-disciplinary tasks, and further analyze how reinforcement learning and thinking patterns affect reasoning efficacy. The Dual Tuning results guide data curation by identifying data that benefit reasoning training, data better suited to direct-answer training, and data that are detrimental under both training modes. Our work provides quantitative criteria for selecting appropriate training data and matching post-training strategies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑