TODO：利用三元偏好增强LLM对齐
TODO: Enhancing LLM Alignment with Ternary Preferences
AI总结：
针对二元偏好模型难以处理噪声和平局的问题，提出三元偏好对齐算法TODO，通过引入支持平局的TOBT模型改进偏好表示，在多项基准上显著提升了LLM的对齐性能。
摘要：
将大型语言模型（LLM）与人类意图对齐对于提升其在各种任务上的性能至关重要。标准的对齐技术，如Direct Preference Optimization (DPO)，通常依赖二元的Bradley-Terry (BT)模型，该模型在捕捉人类偏好的复杂性方面存在困难——尤其是在存在噪声或不一致标签以及频繁平局的情况下。为了解决这些局限性，我们引入了Tie-rank Oriented Bradley-Terry model (TOBT)，它是BT模型的扩展，明确纳入了平局，从而实现了更细致的偏好表示。在此基础上，我们提出了Tie-rank Oriented Direct Preference Optimization (TODO)，这是一种新颖的对齐算法，利用TOBT的三元排序系统来改善偏好对齐。在Mistral-7B和Llama 3-8B模型上的评估中，TODO在分布内和分布外数据集上的偏好建模方面始终优于DPO。使用MT Bench以及Piqa、ARC-c和MMLU等基准进行的额外评估进一步证明了TODO卓越的对齐性能。值得注意的是，TODO在二元偏好对齐方面也显示出强劲的结果，突显了其多功能性及更广泛集成到LLM对齐中的潜力。实现细节可在 https://github.com/XXares/TODO 找到。
英文摘要：
Aligning large language models (LLMs) with human intent is critical for enhancing their performance across a variety of tasks. Standard alignment techniques, such as Direct Preference Optimization (DPO), often rely on the binary Bradley-Terry (BT) model, which can struggle to capture the complexities of human preferences -- particularly in the presence of noisy or inconsistent labels and frequent ties. To address these limitations, we introduce the Tie-rank Oriented Bradley-Terry model (TOBT), an extension of the BT model that explicitly incorporates ties, enabling more nuanced preference representation. Building on this, we propose Tie-rank Oriented Direct Preference Optimization (TODO), a novel alignment algorithm that leverages TOBT's ternary ranking system to improve preference alignment. In evaluations on Mistral-7B and Llama 3-8B models, TODO consistently outperforms DPO in modeling preferences across both in-distribution and out-of-distribution datasets. Additional assessments using MT Bench and benchmarks such as Piqa, ARC-c, and MMLU further demonstrate TODO's superior alignment performance. Notably, TODO also shows strong results in binary preference alignment, highlighting its versatility and potential for broader integration into LLM alignment. The implementation details can be found in https://github.com/XXares/TODO.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
