针对伦理模糊性的语言模型微调：与人类回答对齐的比较研究
Fine-Tuning Language Models for Ethical Ambiguity: A Comparative Study of Alignment with Human Responses
AI总结：
本研究针对道德模糊场景下LLM与人类判断对齐较差的问题，利用Scruples项目的数据集对Llama-3.1-8b等模型进行微调，发现微调显著提升了交叉熵和Dirichlet分数，使Mistral-7B性能比肩GPT-4o，但仍不及BERT和RoBERTa。
摘要：
语言模型常因对歧义的处理不当而误解人类意图，这一局限性在NLP研究中已得到广泛认可。虽然LLM对道德明确的场景具有较高的辨识度，但在道德模糊的语境中却面临更大的困难。在本研究中，我们探讨了LLM的校准问题，表明在这些场景下人类与LLM的判断存在严重的不一致。我们使用了来自Scruples项目的两个精选数据集进行评估：DILEMMAS包含成对的不同道德场景，用于评估模型比较和对比伦理情境的能力；ANECDOTES提供个体叙事，用于评估模型提取细节、解释和分析不同道德场景的技能。我们提取了所有可能选项的模型答案概率，并与人类标注进行比较，以对Llama-3.1-8b、Zephyr-7b-beta和Mistral-7b三个模型的对齐程度进行基准测试。微调后观察到了显著改善，在交叉熵和Dirichlet分数上均有明显提升，尤其是在后者上。值得注意的是，微调后Mistral-7B-Instruct-v0.3的性能与GPT-4o相当。然而，在交叉熵分数方面，所考察的实验模型仍然全部被BERT和RoBERTa模型超越。我们的微调方法以文本到文本格式改善了模型对文本分布的理解，有效提升了在复杂决策语境下的性能和对齐度，凸显了进一步研究以完善伦理推理技术并捕捉人类判断细微差别的必要性。
英文摘要：
Language models often misinterpret human intentions due to their handling of ambiguity, a limitation well-recognized in NLP research. While morally clear scenarios are more discernible to LLMs, greater difficulty is encountered in morally ambiguous contexts. In this investigation, we explored LLM calibration to show that human and LLM judgments are poorly aligned in such scenarios. We used two curated datasets from the Scruples project for evaluation: DILEMMAS, which involves pairs of distinct moral scenarios to assess the model's ability to compare and contrast ethical situations, and ANECDOTES, which presents individual narratives to evaluate the model's skill in drawing out details, interpreting, and analyzing distinct moral scenarios. Model answer probabilities were extracted for all possible choices and compared with human annotations to benchmark the alignment of three models: Llama-3.1-8b, Zephyr-7b-beta, and Mistral-7b. Significant improvements were observed after fine-tuning, with notable enhancements in both cross-entropy and Dirichlet scores, particularly in the latter. Notably, after fine-tuning, the performance of Mistral-7B-Instruct-v0.3 was on par with GPT-4o. However, the experimental models that were examined were all still outperformed by the BERT and RoBERTa models in terms of cross-entropy scores. Our fine-tuning approach, which improves the model's understanding of text distributions in a text-to-text format, effectively enhances performance and alignment in complex decision-making contexts, underscoring the need for further research to refine ethical reasoning techniques and capture human judgment nuances.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
