arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2412.14780cs.CL
收藏

为语言模型微调解开推理令牌与样板令牌

Disentangling Reasoning Tokens and Boilerplate Tokens For Language Model Fine-tuning

University of Science and Technology of China(中国科学技术大学);Alibaba Group(阿里巴巴集团);National University of Singapore(新加坡国立大学)

Ziang Ye, Zhenru Zhang, Yang Zhang, Jianxin Ma, Junyang Lin, Fuli Feng

AI总结:

针对LLMs智能体微调中令牌同等对待的问题,提出SHAD判别器区分推理与样板令牌,并设计RFT方法自适应强调推理令牌,显著超越常规SFT。

摘要:

在使用智能体任务数据集增强大型语言模型(LLMs)的智能体能力时,当前方法通常将样本中的所有令牌同等对待。然而,我们认为扮演不同角色的令牌——特别是推理令牌与样板令牌(例如,控制输出格式的令牌)——在重要性和学习复杂性上存在显著差异,需要将它们解开并区别对待。为此,我们提出了一种新颖的Shuffle-Aware Discriminator (SHAD)用于自适应令牌判别。SHAD利用在跨样本打乱输入-输出组合后观察到的可预测性差异来对令牌进行分类:样板令牌由于在样本间的重复性而保持可预测性,而推理令牌则不然。基于SHAD,我们提出了Reasoning-highlighted Fine-Tuning (RFT)方法,该方法在微调期间自适应地强调推理令牌,与常见的监督微调(SFT)相比,取得了显著的性能提升。

英文摘要:

When using agent-task datasets to enhance agent capabilities for Large Language Models (LLMs), current methodologies often treat all tokens within a sample equally. However, we argue that tokens serving different roles - specifically, reasoning tokens versus boilerplate tokens (e.g., those governing output format) - differ significantly in importance and learning complexity, necessitating their disentanglement and distinct treatment. To address this, we propose a novel Shuffle-Aware Discriminator (SHAD) for adaptive token discrimination. SHAD classifies tokens by exploiting predictability differences observed after shuffling input-output combinations across samples: boilerplate tokens, due to their repetitive nature among samples, maintain predictability, whereas reasoning tokens do not. Using SHAD, we propose the Reasoning-highlighted Fine-Tuning (RFT) method, which adaptively emphasizes reasoning tokens during fine-tuning, yielding notable performance gains over common Supervised Fine-Tuning (SFT).

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑