arXiv 2409.009352024-09-02cs.CL

Self-Judge：基于对齐自评估的选择性指令遵循

Self-Judge: Selective Instruction Following with Alignment Self-Evaluation

National University of Singapore（新加坡国立大学）

Hai Ye, Hwee Tou Ng

备注：: Under review

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Self-J自训练框架，利用模型自评估能力和黄金参考答案构建判断模型，实现选择性指令遵循，实验表明其与GPT-4相关性优于强基线，并可作为奖励模型提升WizardLM在AlpacaEval上的表现。

摘要：

预训练大语言模型（LLMs）可以通过指令微调来遵循人类指令。然而，由于测试时数据分布的变化，它们可能并不总能准确执行指令，在作为聊天助手时可能产生事实错误或与人类意图不一致的内容。为了提高LLMs在遵循指令方面的可靠性，我们提出研究选择性指令遵循，即当预期响应质量较低时，系统拒绝执行指令。我们训练能够预测模型响应数值质量分数的判断模型。为解决数据稀缺问题，我们引入Self-J，一种新颖的自训练框架，用于在无需人工标注质量分数的情况下开发判断模型。我们的方法利用模型固有的自我评估能力，从带标签的指令微调数据中提取响应质量信息。该方法引入黄金参考答案以促进自我评估，并通过评估响应样本与黄金参考答案之间的语义相似度进行重新校准。在训练阶段，我们采用自蒸馏作为正则化技术，以增强无参考估计的能力。为验证通用指令遵循任务上的对齐评估效果，我们从Hugging Face收集大规模高质量指令用于模型训练和评估。在五个开源模型上的大量实验表明，我们的方法与GPT-4的相关性远高于强基线方法，例如从GPT-4和GPT-3.5-turbo蒸馏得到的监督模型。我们的分析表明，我们的模型在不同领域具有很强的泛化能力。此外，我们的判断模型还可作为良好的奖励模型，例如，使用我们的判断模型进行best-of-32采样，在AlpacaEval的v1和v2版本中分别将WizardLM-13B-V1.2从89.17提升至92.48，以及从12.03提升至15.90。

英文摘要：

Pre-trained large language models (LLMs) can be tailored to adhere to human instructions through instruction tuning. However, due to shifts in the distribution of test-time data, they may not always execute instructions accurately, potentially generating factual errors or misaligned content when acting as chat assistants. To enhance the reliability of LLMs in following instructions, we propose the study of selective instruction following, whereby the system declines to execute instructions if the anticipated response quality is low. We train judge models that can predict numerical quality scores for model responses. To address data scarcity, we introduce Self-J, a novel self-training framework for developing judge models without needing human-annotated quality scores. Our method leverages the model's inherent self-evaluation capability to extract information about response quality from labeled instruction-tuning data. It incorporates a gold reference answer to facilitate self-evaluation and recalibrates by assessing the semantic similarity between the response sample and the gold reference. During the training phase, we implement self-distillation as a regularization technique to enhance the capability of reference-free estimation. To validate alignment evaluation on general instruction-following tasks, we collect large-scale high-quality instructions from Hugging Face for model training and evaluation. Extensive experiments on five open-source models show that our method correlates much more with GPT-4 than strong baselines, e.g., supervised models distilled from GPT-4 and GPT-3.5-turbo. Our analysis shows our model's strong generalization across domains. Additionally, our judge models serve as good reward models, e.g., boosting WizardLM-13B-V1.2 from 89.17 to 92.48 and from 12.03 to 15.90 in version v1 and v2 of AlpacaEval respectively using best-of-32 sampling with our judge models.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文