超越标签：将大型语言模型与类人推理对齐
Beyond Labels: Aligning Large Language Models with Human-like Reasoning
AI总结：
本文提出DFAR伦理数据集和L+R微调方法，通过注入伦理标签及对应理由对齐大型语言模型，显著提升了伦理分类准确率并降低了理由生成错位率，使模型生成更具类人特征的响应。
摘要：
将大型语言模型（LLMs）与人类推理方法对齐，可确保LLMs产生道德正确且类人的决策。由于当前模型容易产生假阳性并提供恶意回复，因此引发了伦理担忧。为解决这一问题，我们策划了一个名为Dataset for Aligning Reasons（DFAR）的伦理数据集，旨在帮助对齐语言模型以生成类人理由。该数据集包含带有伦理-非伦理标签的陈述及其对应理由。在本研究中，我们采用了一种独特且新颖的微调方法，该方法利用伦理标签及其对应理由（L+R），这与仅使用标签（L）的现有微调方法形成对比。随后，我们在伦理-非伦理分类任务和理由生成任务上，评估了LLMs的原始预训练版本、现有微调版本以及我们提出的微调版本。我们提出的微调策略在两项任务中均显著优于其他方法，在分类任务中获得了更高的准确率，在理由生成任务中实现了更低的错位率。分类准确率的提高和错位率的降低表明，L+R微调模型更符合人类伦理。因此，本研究表明，注入理由显著改善了LLMs的对齐，从而产生更具类人特征的回复。我们已在 https://github.com/apurba-nsu-rnd-lab/DFAR 公开提供了DFAR数据集及相关代码。
英文摘要：
Aligning large language models (LLMs) with a human reasoning approach ensures that LLMs produce morally correct and human-like decisions. Ethical concerns are raised because current models are prone to generating false positives and providing malicious responses. To contribute to this issue, we have curated an ethics dataset named Dataset for Aligning Reasons (DFAR), designed to aid in aligning language models to generate human-like reasons. The dataset comprises statements with ethical-unethical labels and their corresponding reasons. In this study, we employed a unique and novel fine-tuning approach that utilizes ethics labels and their corresponding reasons (L+R), in contrast to the existing fine-tuning approach that only uses labels (L). The original pre-trained versions, the existing fine-tuned versions, and our proposed fine-tuned versions of LLMs were then evaluated on an ethical-unethical classification task and a reason-generation task. Our proposed fine-tuning strategy notably outperforms the others in both tasks, achieving significantly higher accuracy scores in the classification task and lower misalignment rates in the reason-generation task. The increase in classification accuracies and decrease in misalignment rates indicate that the L+R fine-tuned models align more with human ethics. Hence, this study illustrates that injecting reasons has substantially improved the alignment of LLMs, resulting in more human-like responses. We have made the DFAR dataset and corresponding codes publicly available at https://github.com/apurba-nsu-rnd-lab/DFAR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
