arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.00474cs.LGcs.AIcs.CLstat.ML
收藏

长文本生成的语言校准

Linguistic Calibration of Long-Form Generations

Stanford University(斯坦福大学)

Neil Band, Xuechen Li, Tengyu Ma, Tatsunori Hashimoto

AI总结:

针对语言模型幻觉导致用户决策次优的问题,提出通过监督微调与强化学习训练框架实现长文本生成的语言校准,使模型输出带置信度声明,显著提升了下游决策的校准性。

摘要:

当语言模型(LM)自信地产生幻觉时,可能会导致其用户做出次优的下游决策。通过让LM用语言传达其主张正确的概率,可以缓解这一问题,但现有模型无法生成带有校准置信度声明的长文本。从决策的角度出发,我们定义了长文本生成的语言校准:如果一个LM的生成内容能让其用户做出校准的概率预测,则该LM是语言校准的。这一定义促成了一种训练框架,其中监督微调步骤引导LM生成长文本及置信度声明,如“我估计有30%的概率……”或“我确信……”,随后的强化学习步骤则奖励那些能让用户对相关问题给出校准答案的生成内容。我们对Llama 2 7B进行了语言校准,并在长文本生成的自动和人工评估中发现,与具有可比准确度的强微调事实性基线相比,其校准效果显著更好。这些发现在显著的领域偏移下依然适用,包括科学和生物医学问题,以及一个完全留出的人物传记生成任务。我们的结果表明,通过在用户于下游决策中做出的预测空间内构建目标,可以端到端地校准长文本生成。

英文摘要:

Language models (LMs) may lead their users to make suboptimal downstream decisions when they confidently hallucinate. This issue can be mitigated by having the LM verbally convey the probability that its claims are correct, but existing models cannot produce long-form text with calibrated confidence statements. Through the lens of decision-making, we define linguistic calibration for long-form generations: an LM is linguistically calibrated if its generations enable its users to make calibrated probabilistic predictions. This definition enables a training framework where a supervised finetuning step bootstraps an LM to emit long-form generations with confidence statements such as "I estimate a 30% chance of..." or "I am certain that...", followed by a reinforcement learning step which rewards generations that enable a user to provide calibrated answers to related questions. We linguistically calibrate Llama 2 7B and find in automated and human evaluations of long-form generations that it is significantly more calibrated than strong finetuned factuality baselines with comparable accuracy. These findings generalize under significant domain shifts to scientific and biomedical questions and to an entirely held-out person biography generation task. Our results demonstrate that long-form generations may be calibrated end-to-end by constructing an objective in the space of the predictions that users make in downstream decision-making.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑