HonestLLM：迈向诚实且有帮助的大语言模型
HonestLLM: Toward an Honest and Helpful Large Language Model
AI总结：
针对大语言模型需兼顾诚实性与有用性的需求，该研究制定诚实原则、构建HoneSet评估数据集，提出免训练提示和两阶段微调两种增强方法，在9款主流LLM上验证了显著的诚实对齐提升效果。
摘要：
大语言模型（LLM）凭借其卓越的生成能力在各行业取得了显著成功。然而，为实现安全有效的实际部署，确保其诚实性与有用性至关重要。本文探讨了如下问题：我们能否在保留LLM诚实性的同时，优先提升其有用性？\n首先，我们制定了旨在保障LLM诚实性的详尽原则。此外，我们引入了一个名为HoneSet的新型数据集，包含横跨六大类的930条精心设计的查询，用于评估LLM保持诚实的能力。\n随后，我们提出了两种增强LLM诚实性与有用性的方法：免训练增强法和基于微调的改进法。免训练方法基于好奇心驱动提示（curiosity-driven prompting），能让LLM表达出对查询的内部困惑与不确定性，从而优化其回复。而基于微调的方法则采用受课程学习（curriculum learning）启发的两阶段流程：首先指导LLM区分诚实与不诚实的回复，随后优化其训练以提升有用性。\n在9个主流LLM上开展的实验表明，通过实施我们提出的增强方案，所有模型的诚实对齐程度均得到显著提升。尤其值得注意的是，经H²（诚实与有用性）评估测得，Llama3-8b的性能提升了65.3%，Mistral-7b的提升幅度更是达到124.7%。我们相信，本研究可为开发适用于实际应用的更可信LLM铺平道路。
英文摘要：
Large Language Models (LLMs) have achieved remarkable success across various industries due to their exceptional generative capabilities. However, for safe and effective real-world deployments, ensuring honesty and helpfulness is critical. This paper addresses the question: Can we prioritize the helpfulness of LLMs while preserving their honesty? To begin with, we establish exhaustive principles aimed at guaranteeing the honesty of LLM. Additionally, we introduce a novel dataset, referred to as HoneSet, comprising 930 queries spanning six categories meticulously crafted to assess an LLM's capacity for maintaining honesty. Subsequently, we present two approaches to augmenting honesty and helpfulness in LLMs: a training-free enhancement and a fine-tuning-based improvement. The training-free approach, which is based on curiosity-driven prompting, empowers LLMs to articulate internal confusion and uncertainty regarding queries, thereby optimizing their responses. Conversely, the fine-tuning-based method employs a two-stage process inspired by curriculum learning: initially instructing LLMs to discern between honest and dishonest responses, then refining their training to enhance helpfulness. Experiments conducted on nine prominent LLMs demonstrate a significant improvement in alignment with honesty across all models through the implementation of our proposed enhancements. Particularly noteworthy is the 65.3% enhancement observed in Llama3-8b and the remarkable 124.7% improvement in Mistral-7b, as measured by the H$^{2}$ (honest and helpful) assessment. We believe that our work can pave the way for developing more trustworthy LLMs for real-world applications.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
