arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.13575cs.CLcs.AI
收藏

Elo-Evolve:一种用于语言模型对齐的共进化框架

Elo-Evolve: A Co-evolutionary Framework for Language Model Alignment

Zuoyebang, Beijing, China(北京中关村)

Jing Zhao, Ting Zhen, Junwei Bao, Hongfei Jiang, Yang Song

AI总结:

Elo-Evolve通过动态多代理竞争和自适应对手选择,提升大型语言模型对齐的稳定性与效果。

摘要:

当前大型语言模型(LLM)的对齐方法依赖于将大量人类偏好数据压缩成静态的绝对奖励函数,导致数据稀缺、噪声敏感和训练不稳定。我们引入了Elo-Evolve,一种将对齐重新定义为动态多代理竞争的共进化框架,其中包含一个适应性对手池。我们的方法有两个关键创新:(1)通过直接从成对竞争中的二元胜负结果学习,消除了Bradley-Terry模型的依赖;(2)实现由Elo主导的对手选择,通过温度控制的采样实现自动课程学习。我们的方法基于PAC学习理论,证明成对比较实现了更优的样本复杂性,并实验证明与绝对评分方法相比噪声减少了4.5倍。实验中,我们使用我们的框架训练了一个Qwen2.5-7B模型,对手包括Qwen2.5-14B、Qwen2.5-32B和Qwen3-8B模型。结果显示出明显的性能层级:基于点的方法<静态成对训练<Elo-Evolve在Alpaca Eval 2.0和MT-Bench上,验证了成对比较和动态对手选择对LLM对齐的逐步优势。

英文摘要:

Current alignment methods for Large Language Models (LLMs) rely on compressing vast amounts of human preference data into static, absolute reward functions, leading to data scarcity, noise sensitivity, and training instability. We introduce Elo-Evolve, a co-evolutionary framework that redefines alignment as dynamic multi-agent competition within an adaptive opponent pool. Our approach makes two key innovations: (1) eliminating Bradley-Terry model dependencies by learning directly from binary win/loss outcomes in pairwise competitions, and (2) implementing Elo-orchestrated opponent selection that provides automatic curriculum learning through temperature-controlled sampling. We ground our approach in PAC learning theory, demonstrating that pairwise comparison achieves superior sample complexity and empirically validate a 4.5x noise reduction compared to absolute scoring approaches. Experimentally, we train a Qwen2.5-7B model using our framework with opponents including Qwen2.5-14B, Qwen2.5-32B, and Qwen3-8B models. Results demonstrate a clear performance hierarchy: point-based methods < static pairwise training < Elo-Evolve across Alpaca Eval 2.0 and MT-Bench, validating the progressive benefits of pairwise comparison and dynamic opponent selection for LLM alignment.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑