DPO Kernels：一种语义感知、核增强且散度丰富的直接偏好优化范式
DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization
AI总结：
本文提出DPO-Kernels，通过核化表示、多种散度替代、数据驱动选择和分层核混合，解决直接偏好优化的固定散度与特征变换限制，在12个数据集上实现最先进性能。
摘要：
大型语言模型（LLM）的迅速崛起解锁了许多应用，但也凸显了使其与多样化的价值观和偏好对齐的挑战。直接偏好优化（DPO）是对齐的核心，但受限于固定的散度和有限的特征变换。我们提出了DPO-Kernels，它集成核方法以通过四个关键贡献解决这些问题：（i）核化表示，使用多项式、RBF、马氏距离和谱核进行更丰富的变换，以及结合基于嵌入和基于概率目标的混合损失；（ii）散度替代方案（Jensen-Shannon、Hellinger、Renyi、Bhattacharyya、Wasserstein和f-散度）以增强稳定性；（iii）数据驱动的选择指标，自动选择最佳的核-散度对；（iv）核的分层混合，用于局部精度和全局建模。在12个数据集上的评估展示了在事实性、安全性、推理和指令遵循方面的最先进性能。基于重尾自正则化，DPO-Kernels保持了LLM的稳健泛化，为进一步的对齐研究提供了综合资源。
英文摘要：
The rapid rise of large language models (LLMs) has unlocked many applications but also underscores the challenge of aligning them with diverse values and preferences. Direct Preference Optimization (DPO) is central to alignment but constrained by fixed divergences and limited feature transformations. We propose DPO-Kernels, which integrates kernel methods to address these issues through four key contributions: (i) Kernelized Representations with polynomial, RBF, Mahalanobis, and spectral kernels for richer transformations, plus a hybrid loss combining embedding-based and probability-based objectives; (ii) Divergence Alternatives (Jensen-Shannon, Hellinger, Renyi, Bhattacharyya, Wasserstein, and f-divergences) for greater stability; (iii) Data-Driven Selection metrics that automatically choose the best kernel-divergence pair; and (iv) a Hierarchical Mixture of Kernels for both local precision and global modeling. Evaluations on 12 datasets demonstrate state-of-the-art performance in factuality, safety, reasoning, and instruction following. Grounded in Heavy-Tailed Self-Regularization, DPO-Kernels maintains robust generalization for LLMs, offering a comprehensive resource for further alignment research.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
