基于双自适应加权的团队自博弈：用于微调大语言模型
Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs
AI总结：
本文提出TPAW算法，通过团队协作与竞争机制提升自监督下LLM对齐效果，采用双自适应加权机制提高优化效率，实验证明其在多种模型和基准测试中表现优异。
摘要：
尽管最近的自训练方法减少了对人类标注数据的依赖，但仍面临关键限制：(i) 对合成数据质量敏感，导致迭代训练不稳定和偏见放大；(ii) 由于正负响应间差距逐渐缩小，优化效果不佳。本文提出团队自博弈与双自适应加权（TPAW）算法，采用团队框架，使当前策略模型与历史检查点协作与竞争，促进更稳定高效的优化。设计两种自适应加权机制：响应重加权方案调整目标响应重要性，玩家加权策略动态调节训练中每个团队成员的贡献。从SFT模型初始化，TPAW迭代优化对齐无需额外人工监督。实验结果表明，TPAW在各种基础模型和LLM基准测试中均优于现有基线。代码已公开于https://github.com/lab-klc/TPAW。
英文摘要：
While recent self-training approaches have reduced reliance on human-labeled data for aligning LLMs, they still face critical limitations: (i) sensitivity to synthetic data quality, leading to instability and bias amplification in iterative training; (ii) ineffective optimization due to a diminishing gap between positive and negative responses over successive training iterations. In this paper, we propose Team-based self-Play with dual Adaptive Weighting (TPAW), a novel self-play algorithm designed to improve alignment in a fully self-supervised setting. TPAW adopts a team-based framework in which the current policy model both collaborates with and competes against historical checkpoints, promoting more stable and efficient optimization. To further enhance learning, we design two adaptive weighting mechanisms: (i) a response reweighting scheme that adjusts the importance of target responses, and (ii) a player weighting strategy that dynamically modulates each team member's contribution during training. Initialized from a SFT model, TPAW iteratively refines alignment without requiring additional human supervision. Experimental results demonstrate that TPAW consistently outperforms existing baselines across various base models and LLM benchmarks. Our code is publicly available at https://github.com/lab-klc/TPAW.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
