arXiv 2512.071752025-12-08cs.LG

SPACE：噪声对比估计稳定了大语言模型的自我对战微调

SPACE: Noise Contrastive Estimation Stabilizes Self-Play Fine-Tuning for Large Language Models

National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）；School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）；Alibaba International Digital Commerce（阿里巴巴国际数字商业）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））

Yibo Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, Lijun Zhang

备注：: NeurIPS 2025

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

SPACE通过噪声对比估计稳定大语言模型的自我对战微调，提升下游任务性能

摘要：

自我对战微调在将大语言模型（LLMs）适应下游任务时，通过迭代地使用真实样本和自身生成的合成样本来优化模型。然而，现有方法主要关注两种数据类型奖励之间的相对差距，忽略了它们的绝对值。通过理论分析，我们发现基于差距的方法由于潜在退化的目标而存在不稳定的进化。为解决这一限制，我们引入了一种新的自我对战微调方法，即通过噪声对比估计（SPACE），利用噪声对比估计来捕捉真实数据分布。具体而言，SPACE将合成样本作为辅助组件，并通过二元分类方式区分它们与真实样本。结果，SPACE独立优化每种数据类型的绝对奖励值，确保了持续有意义的目标，从而避免了不稳定性问题。理论上，我们证明了SPACE的目标最优解与真实数据分布一致，且SPACE保证了能够证明稳定收敛到最优分布。实验上，我们证明SPACE在各种任务上显著提高了LLMs的性能，并优于使用大量真实样本的监督微调。与基于差距的自我对战微调方法相比，SPACE表现出显著的优越性和稳定的进化。

英文摘要：

Self-play fine-tuning has demonstrated promising abilities in adapting large language models (LLMs) to downstream tasks with limited real-world data. The basic principle is to iteratively refine the model with real samples and synthetic ones generated from itself. However, the existing methods primarily focus on the relative gaps between the rewards for two types of data, neglecting their absolute values. Through theoretical analysis, we identify that the gap-based methods suffer from unstable evolution, due to the potentially degenerated objectives. To address this limitation, we introduce a novel self-play fine-tuning method, namely Self-PlAy via Noise Contrastive Estimation (SPACE), which leverages noise contrastive estimation to capture the real-world data distribution. Specifically, SPACE treats synthetic samples as auxiliary components, and discriminates them from the real ones in a binary classification manner. As a result, SPACE independently optimizes the absolute reward values for each type of data, ensuring a consistently meaningful objective and thereby avoiding the instability issue. Theoretically, we show that the optimal solution of the objective in SPACE aligns with the underlying distribution of real-world data, and SPACE guarantees a provably stable convergence to the optimal distribution. Empirically, we show that SPACE significantly improves the performance of LLMs over various tasks, and outperforms supervised fine-tuning that employs much more real-world samples. Compared to gap-based self-play fine-tuning methods, SPACE exhibits remarkable superiority and stable evolution.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。