将LLM对齐视为检索器优化：信息检索视角
LLM Alignment as Retriever Optimization: An Information Retrieval Perspective
AI总结：
本文从信息检索视角建立LLM对齐与检索器-重排序器范式的映射，提出直接偏好优化方法LarPO，并在AlpacaEval2和MixEval-Hard上验证其显著提升。
摘要：
大语言模型（LLM）凭借推理、编程和交流能力革新了人工智能，推动了各行业的创新。它们真正的潜力取决于有效的对齐，以确保正确、可信且合乎道德的行为，并应对错误信息、幻觉、偏见和滥用等挑战。尽管现有的基于强化学习（RL）的对齐方法以复杂著称，但直接优化方法提供了一种更简单的替代方案。在本研究中，我们借鉴成熟的信息检索（IR）原则，提出了一种新颖的LLM对齐直接优化方法。我们给出了一个连接LLM对齐与IR方法论的系统性框架，将LLM生成模型和奖励模型映射到IR的检索器-重排序器范式。在此基础上，我们提出了LLM Alignment as Retriever Preference Optimization（LarPO），这是一种新的对齐方法，可提升整体对齐质量。大量实验验证了LarPO的有效性：其在AlpacaEval2和MixEval-Hard上分别取得38.9%和13.7%的平均提升。我们的工作通过融合IR基础，为推进LLM对齐开辟了新途径，并为未来研究提供了有前景的方向。
英文摘要：
Large Language Models (LLMs) have revolutionized artificial intelligence with capabilities in reasoning, coding, and communication, driving innovation across industries. Their true potential depends on effective alignment to ensure correct, trustworthy and ethical behavior, addressing challenges like misinformation, hallucinations, bias and misuse. While existing Reinforcement Learning (RL)-based alignment methods are notoriously complex, direct optimization approaches offer a simpler alternative. In this work, we introduce a novel direct optimization approach for LLM alignment by drawing on established Information Retrieval (IR) principles. We present a systematic framework that bridges LLM alignment and IR methodologies, mapping LLM generation and reward models to IR's retriever-reranker paradigm. Building on this foundation, we propose LLM Alignment as Retriever Preference Optimization (LarPO), a new alignment method that enhances overall alignment quality. Extensive experiments validate LarPO's effectiveness with 38.9 % and 13.7 % averaged improvement on AlpacaEval2 and MixEval-Hard respectively. Our work opens new avenues for advancing LLM alignment by integrating IR foundations, offering a promising direction for future research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
