偏好一致性至关重要：通过训练语料的自动自我筛选增强语言模型中的偏好学习
Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora
AI总结：
针对偏好学习数据集中的标注不一致问题，提出一种利用代理模型自动检测并筛选一致标注的自我筛选方法，在指令遵循任务中性能提升达33%，无需依赖启发式规则。
摘要：
训练语料中的不一致标注，尤其是在偏好学习数据集中，对开发高级语言模型构成了挑战。这些不一致通常源于标注者之间的差异以及偏好固有的多维特性。为了解决这些问题，我们引入了一种自我筛选方法，该方法通过利用直接在标注数据集上训练的代理模型来预处理这些数据集。我们的方法通过自动检测并选择一致的标注来增强偏好学习。我们通过广泛的指令遵循任务验证了所提出的方法，结果表明在各种学习算法和代理能力下，性能提升高达 33%。这项工作提供了一种简单可靠的解决方案来应对偏好不一致问题，而无需依赖启发式方法，可作为迈向更高级偏好学习方法论的第一步。代码可在 https://github.com/Self-Curation/ 获取。
英文摘要：
Inconsistent annotations in training corpora, particularly within preference learning datasets, pose challenges in developing advanced language models. These inconsistencies often arise from variability among annotators and inherent multi-dimensional nature of the preferences. To address these issues, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them. Our method enhances preference learning by automatically detecting and selecting consistent annotations. We validate the proposed approach through extensive instruction-following tasks, demonstrating performance improvements of up to 33\% across various learning algorithms and proxy capabilities. This work offers a straightforward and reliable solution to address preference inconsistencies without relying on heuristics, serving as an initial step toward the development of more advanced preference learning methodologies. Code is available at https://github.com/Self-Curation/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
