测试时强化学习对齐揭示LLM基准中的任务熟悉性 artifacts
Test-time RL alignment exposes task familiarity artifacts in LLM benchmarks
AI总结:
本文提出一种两阶段测试时强化学习对齐方法,用于训练前测试,通过单样本强化学习和多数投票奖励对齐模型,无需特定任务训练数据,提升基准评估的准确性。
摘要:
直接在基准上评估LLM可能会产生误导,因为较强的性能可能反映任务熟悉度而非能力。训练前测试方法通过在评估前给模型相关任务训练来控制任务熟悉度,最初通过监督微调实现。然而,合适的训练数据往往难以获得,评估结果会因所选数据而异。本文提出一种两阶段测试时强化学习(RL)对齐方法用于训练前测试。首先,单样本强化学习对模型进行初步对齐,其次,多数投票奖励的测试时强化学习对模型对齐到基准分布。我们的测试时强化学习对齐方法效果与基于SFT的训练前测试相似,但无需特定任务训练集。在无训练数据的领域特定基准上,我们显示直接评估低估了基础模型,一旦对齐后,其表现显著提升,从而更真实地评估其能力。此外,对于推理任务,微调模型与基础模型之间的性能差距在对齐后大幅消失,表明文献中报告的许多来自RLVR/SFT的增益并非推理能力差异,而是任务熟悉度的 artifacts。
英文摘要:
Direct evaluation of LLMs on benchmarks can be misleading because comparatively strong performance may reflect task familiarity rather than capability. The train-before-test approach controls for task familiarity by giving each model task-relevant training before evaluation, originally through supervised finetuning. However, suitable training data is often hard to come by, and evaluation results vary with the data chosen. In this paper, we propose a two-stage test-time reinforcement learning (RL) alignment method for train-before-test. First, RL with a single sample provides a first alignment of the model to the task format, and second, test-time RL with majority-voting reward aligns the model to the benchmark distribution. Our test-time RL alignment method aligns similarly well as SFT-based train-before test, but without requiring a task-specific training set. On a domain-specific benchmark without training data, we show that direct evaluation underestimates base models which perform substantially better once aligned, yielding a more faithful evaluation of their capabilities. Moreover, for reasoning tasks, the performance gap between fine-tuned models and their base models largely disappears after alignment, suggesting that many gains from RLVR/SFT reported in the literature are not a difference in reasoning capability, but rather artifacts of task familiarity.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。