大型语言模型中的双重用途困境：增强伦理能力是否会降低实用性？
The Dual-use Dilemma in LLMs: Do Empowering Ethical Capacities Make a Degraded Utility?
AI总结：
本文提出基于直接偏好优化的对齐框架，通过平衡安全与实用性，在化学领域构建数据集并评估，使模型整体性能超越Claude-3等领先LLM，同时揭示长链思维推理带来的伦理风险。
摘要：
近年来，各领域在增强大型语言模型（LLM）方面投入了大量努力，同时其伦理影响也日益受到关注。然而，一个关键挑战在很大程度上仍被忽视：LLM必须在拒绝有害请求以确保安全与接受合法请求以保持实用性之间取得平衡。本文提出了一种基于直接偏好优化（DPO）的对齐框架，通过解决这一伦理-实用性权衡问题，实现了更好的整体性能，并以化学领域应用作为概念验证。我们的对齐流程始于一个GPT辅助的三阶段数据生成方案，据此创建了LibraChemQA，一个包含31.6k三元组实例的化学问答数据集。通过在数据生成过程中引入创新的平衡种子，我们的框架系统性地兼顾了合法与非法请求。该框架还引入了一种重述机制，用于高效的数据增强，以提升模型的化学理解能力。我们进一步开发了一种新颖的混合评估方案，利用LLM裁判对安全性和实用性进行精确评估。实验结果表明，我们的模型在同时考虑安全性和实用性的整体性能上取得了显著提升——在发布基准上，所得模型分别以13.44%、7.16%和7.10%的优势超越了包括Claude-3、GPT-4o和LLaMA-3在内的领先LLM。在论文末尾，我们分析了在基准上测试DeepSeek-R1所获得的实验结果，并揭示了这一备受赞誉的模型所引发的关键伦理问题。我们强调，DeepSeek-R1及其蒸馏得到的其他LLM所采用的长链思维（CoT）推理过程，在面向用户时引入了显著的伦理脆弱性。
英文摘要：
Recent years have witnessed extensive efforts to enhance Large Language Models (LLMs) across various domains, alongside growing attention to their ethical implications. However, a critical challenge remains largely overlooked: LLMs must balance between rejecting harmful requests for safety and accommodating legitimate ones for utility. This paper presents a Direct Preference Optimization (DPO) based alignment framework that achieves better overall performance by addressing this ethical-utility trade-off, using chemical domain applications as a proof-of-concept. Our alignment pipeline starts with a GPT-assisted three-phase data generation scheme, in which we create LibraChemQA, a chemical question-answering dataset comprising 31.6k triplet instances. By incorporating an innovative balanced seed in the data generation process, our framework systematically considers both legitimate and illegitimate requests. The framework also introduces a rephrasing mechanism for efficient data augmentation that enhances the model's chemical comprehension. We further develop a novel hybrid evaluation scheme with LLM judges for precise assessment of both safety and utility. Experimental results demonstrate our model's substantial improvements in overall performance where both safety and utility are considered - the resulting model outperforms leading LLMs including Claude-3, GPT-4o, and LLaMA-3 by margins of 13.44%, 7.16%, and 7.10% respectively on our released benchmark. At the end of this paper, we analyze experimental results obtained from testing DeepSeek-R1 on our benchmark and reveal the critical ethical concerns raised by this highly acclaimed model. We highlight that the long Chain-of-Thought (CoT) reasoning process employed by DeepSeek-R1, as well as other LLMs distilled from it, introduces significant ethical vulnerabilities when exposed to users.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
