arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.02745cs.AIcs.CL
收藏

CURATRON:用于大语言模型严格对齐的完整且鲁棒的偏好数据

CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models

University of Illinois Chicago(伊利诺伊大学芝加哥分校);Independent Researcher(独立研究者)

Son The Nguyen, Niranjan Uma Naresh, Theja Tulabandhula

AI总结:

针对大语言模型偏好学习对齐中偏好数据不完整、被污染的问题,提出CURATRON方法及多项式时间排序算法,可鲁棒恢复排序,实验验证其抗噪与处理缺失数据能力,助力可靠伦理对齐AI的开发。

摘要:

本文针对通过偏好学习(PL)将大语言模型(LLM)与人类价值观对齐过程中面临的挑战展开研究,聚焦偏好数据集中存在的数据不完整与被污染的问题。我们提出了一种新颖方法,可对这些数据集中的价值观进行鲁棒且完整的重新校准,以增强LLM对上述问题的抵御能力。具体而言,我们设计了一种具备性能保证的多项式时间排序算法,该算法可提升多种现有模型的鲁棒性,例如经典的Bradley-Terry-Luce(BTL)(Bradley和Terry,1952)模型及其部分推广模型。据我们所知,本研究首次提出了一种能以高概率可证明地恢复ε-最优排序的算法,同时该算法可承受每个模型响应下多达O(n)条被扰动的成对比较结果。此外,我们还在部分观测场景下得出了鲁棒恢复结果。我们的实验证实,在通用场景与LLM偏好数据集场景中,我们的算法均能很好地应对对抗性噪声与未观测到的比较情况。本研究通过为数据集整理流程赋予处理缺失输入与恶意篡改输入的能力,为更可靠、更符合伦理对齐的AI模型的开发与规模化应用作出了贡献。

英文摘要:

This paper addresses the challenges of aligning large language models (LLMs) with human values via preference learning (PL), focusing on incomplete and corrupted data in preference datasets. We propose a novel method for robustly and completely recalibrating values within these datasets to enhance LLMs' resilience against the issues. In particular, we devise a guaranteed polynomial time ranking algorithm that robustifies several existing models, such as the classic Bradley-Terry-Luce (BTL) (Bradley and Terry, 1952) model and certain generalizations of it. To the best of our knowledge, our present work is the first to propose an algorithm that provably recovers an $ε$-optimal ranking with high probability while allowing as large as $O(n)$ perturbed pairwise comparison results per model response. Furthermore, we show robust recovery results in the partially observed setting. Our experiments confirm that our algorithms handle adversarial noise and unobserved comparisons well in both general and LLM preference dataset settings. This work contributes to the development and scaling of more reliable and ethically aligned AI models by equipping the dataset curation pipeline with the ability to handle missing and maliciously manipulated inputs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑