arXiv 2505.049932025-05-08cs.CL

潜在偏好编码：通过离散潜在码对齐大语言模型

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）；Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系，威特沃特斯兰大学）

Zhuocheng Gong, Jian Guan, Wei Wu, Huishuai Zhang, Dongyan Zhao

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出潜在偏好编码（LPC），用离散潜在码建模人类偏好中的隐式因素及其组合，无需预定义奖励函数即可与多种离线对齐算法集成，并在多个基准上一致提升 DPO、SimPO 和 IPO 的对齐性能与鲁棒性。

摘要：

大语言模型（LLMs）已取得显著成功，但使其生成内容与人类偏好对齐仍是一项关键挑战。现有的偏好建模方法通常依赖显式或隐式的奖励函数，忽视了人类偏好复杂且多面的本质，这些偏好可能在不同任务和人群中包含相互冲突的因素。为解决这一局限，我们提出潜在偏好编码（LPC），这是一种新颖的框架，利用离散潜在码对整体偏好背后的隐式因素及其组合进行建模。LPC 可与多种离线对齐算法无缝集成，无需依赖预定义的奖励函数和手工设计的组合权重，即可从数据中自动推断潜在因素及其重要性。在多个基准上的大量实验表明，LPC 在三种对齐算法（DPO、SimPO 和 IPO）和三种基础模型（Mistral-7B、Llama3-8B 和 Llama3-8B-Instruct）上均取得一致改进。进一步的分析揭示，学习到的潜在码能够有效捕捉人类偏好分布中的差异，并显著增强对齐对数据噪声的鲁棒性。通过为多样化的偏好因素提供统一表示，LPC 为开发更鲁棒、更通用的对齐技术铺平了道路，从而支持强大 LLMs 的负责任部署。

英文摘要：

Large language models (LLMs) have achieved remarkable success, yet aligning their generations with human preferences remains a critical challenge. Existing approaches to preference modeling often rely on an explicit or implicit reward function, overlooking the intricate and multifaceted nature of human preferences that may encompass conflicting factors across diverse tasks and populations. To address this limitation, we introduce Latent Preference Coding (LPC), a novel framework that models the implicit factors as well as their combinations behind holistic preferences using discrete latent codes. LPC seamlessly integrates with various offline alignment algorithms, automatically inferring the underlying factors and their importance from data without relying on pre-defined reward functions and hand-crafted combination weights. Extensive experiments on multiple benchmarks demonstrate that LPC consistently improves upon three alignment algorithms (DPO, SimPO, and IPO) using three base models (Mistral-7B, Llama3-8B, and Llama3-8B-Instruct). Furthermore, deeper analysis reveals that the learned latent codes effectively capture the differences in the distribution of human preferences and significantly enhance the robustness of alignment against noise in data. By providing a unified representation for the multifarious preference factors, LPC paves the way towards developing more robust and versatile alignment techniques for the responsible deployment of powerful LLMs.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。