用于高效偏好数据选择与诊断的对齐数据图
Alignment Data Map for Efficient Preference Data Selection and Diagnosis
AI总结：
本文提出对齐数据图，通过评估偏好数据的对齐分数，选择高质量低变异性样本，提升对齐效果并提高标注准确性。
摘要：
人类偏好数据对于对齐大语言模型（LLMs）与人类价值观至关重要，但收集此类数据通常成本高且效率低，因此需要高效的数据选择方法以降低标注成本并保持对齐效果。为此，我们提出了对齐数据图，一种用于识别和选择有效偏好数据的数据分析工具。我们首先通过LLM作为判断者、显式奖励模型和基于参考的方法评估偏好数据的对齐分数。对齐数据图基于对齐分数考虑响应质量和跨响应变异性。从实验结果来看，仅训练33%的高质量低变异性样本，在MT-Bench、Evol-Instruct和AlpacaEval上实现了与完整数据集相当或更优的对齐性能。此外，对齐数据图通过分析标注标签与对齐分数之间的相关性，检测潜在的标签误标注，提高标注准确性。实现代码见https://github.com/01choco/Alignment-Data-Map。
英文摘要：
Human preference data is essential for aligning large language models (LLMs) with human values, but collecting such data is often costly and inefficient-motivating the need for efficient data selection methods that reduce annotation costs while preserving alignment effectiveness. To address this issue, we propose Alignment Data Map, a data analysis tool for identifying and selecting effective preference data. We first evaluate alignment scores of the preference data by LLM-as-a-judge, explicit reward model, and reference-based approaches. The Alignment Data Map considers both response quality and inter-response variability based on the alignment scores. From our experimental findings, training on only 33% of samples that exhibit high-quality and low-variability, achieves comparable or superior alignment performance on MT-Bench, Evol-Instruct, and AlpacaEval, compared to training with the full dataset. In addition, Alignment Data Map detects potential label misannotations by analyzing correlations between annotated labels and alignment scores, improving annotation accuracy. The implementation is available at https://github.com/01choco/Alignment-Data-Map.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
