面向诚实的对齐
Alignment for Honesty
AI总结：
本文研究LLM的诚实对齐问题，提出受《论语》启发的诚实定义、衡量指标与可由高效微调实例化的训练框架，实验表明模型诚实度显著提升且不损害其他任务性能。
摘要：
近期研究在使大型语言模型（LLM）与有用性和无害性保持对齐方面取得了显著进展。在本文中，我们主张面向诚实的对齐的重要性，即确保LLM在缺乏知识时主动拒绝回答问题，同时又不过度保守。然而，面向诚实的对齐的一个关键方面在于识别LLM的知识边界，这需要在指标开发、基准构建和训练方法方面提出全面的解决方案。为应对这些挑战，我们首先建立精确的问题定义，并受《论语》启发定义“诚实”。这构成了开发相关指标的基石，这些指标通过量化LLM在对齐后的进步来有效衡量其诚实度。此外，我们提出一个灵活的训练框架，并通过若干高效微调技术进一步实例化该框架，这些技术强调诚实，同时不牺牲其他任务上的性能。大量实验表明，按照我们提出的指标，这些经过对齐的模型在诚实度上表现出显著提升。我们在 https://github.com/GAIR-NLP/alignment-for-honesty 开源所有相关资源，以促进未来研究。
英文摘要：
Recent research has made significant strides in aligning large language models (LLMs) with helpfulness and harmlessness. In this paper, we argue for the importance of alignment for \emph{honesty}, ensuring that LLMs proactively refuse to answer questions when they lack knowledge, while still not being overly conservative. However, a pivotal aspect of alignment for honesty involves discerning an LLM's knowledge boundaries, which demands comprehensive solutions in terms of metric development, benchmark creation, and training methodologies. We address these challenges by first establishing a precise problem definition and defining ``honesty'' inspired by the Analects of Confucius. This serves as a cornerstone for developing metrics that effectively measure an LLM's honesty by quantifying its progress post-alignment. Furthermore, we introduce a flexible training framework which is further instantiated by several efficient fine-tuning techniques that emphasize honesty without sacrificing performance on other tasks. Our extensive experiments reveal that these aligned models show a marked increase in honesty, as indicated by our proposed metrics. We open-source all relevant resources to facilitate future research at \url{https://github.com/GAIR-NLP/alignment-for-honesty}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
