直接密度比优化：一种对齐大语言模型的统计一致性方法
Direct Density Ratio Optimization: A Statistically Consistent Approach to Aligning Large Language Models
AI总结：
针对现有大语言模型对齐方法依赖特定偏好模型导致统计不一致的问题，提出直接密度比优化方法DDRO，通过直接估计偏好与非偏好输出的密度比实现对齐，理论证明其具有统计一致性，实验显示其在多个基准上性能优于现有方法。
摘要：
使大语言模型（LLM）与人类偏好对齐对其安全部署至关重要，但现有方法均假设存在特定偏好模型，例如Bradley-Terry模型。该假设会导致统计不一致性，即更多数据也无法保证模型收敛至真实人类偏好。为填补这一关键空白，我们提出一种新型对齐方法Direct Density Ratio Optimization（DDRO，直接密度比优化）。DDRO直接估计偏好输出分布与非偏好输出分布之间的密度比，无需显式构建人类偏好模型。我们从理论上证明DDRO具有统计一致性，无论底层偏好结构如何，随着数据量增长，其都能确保收敛至真实偏好分布。实验表明，在多个主流基准测试上，DDRO的性能均优于现有方法。DDRO释放了真正数据驱动对齐的潜力，为打造更可靠、更符合人类偏好的LLM铺平了道路。
英文摘要：
Aligning large language models (LLMs) with human preferences is crucial for safe deployment, yet existing methods assume specific preference models like Bradley-Terry model. This assumption leads to statistical inconsistency, where more data doesn't guarantee convergence to true human preferences. To address this critical gap, we introduce a novel alignment method Direct Density Ratio Optimization (DDRO). DDRO directly estimates the density ratio between preferred and unpreferred output distributions, circumventing the need for explicit human preference modeling. We theoretically prove that DDRO is statistically consistent, ensuring convergence to the true preferred distribution as the data size grows, regardless of the underlying preference structure. Experiments demonstrate that DDRO achieves superior performance compared to existing methods on many major benchmarks. DDRO unlocks the potential for truly data-driven alignment, paving the way for more reliable and human-aligned LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
