通过潜变量推断训练思维链
Training Chain-of-Thought via Latent-Variable Inference
AI总结：
本文提出一种基于潜变量推断的MCMC-EM微调方法，通过边际化思维链推理过程来训练LLM，在GSM8K和BIG-Bench Hard上取得优于STaR和提示微调的准确率。
摘要：
大型语言模型（LLMs）在被指示使用“思维链”（chain-of-thought, CoT）提示逐步求解答案时，能够更准确且更具可解释性地解决问题。此外，还可以通过监督微调来提升LLMs在特定任务上的表现，即通过对某些可调参数进行梯度上升，以最大化来自带标签训练集的正确答案的平均对数似然。将CoT与监督微调简单结合，不仅需要对正确答案进行监督，还需要对导致这些答案的详细推理过程进行监督；而这些推理过程的人工标注成本很高。为此，我们提出了一种微调策略，该策略尝试在使用CoT提示时最大化生成正确答案的边际对数似然，并近似地对所有可能的推理过程进行平均。核心挑战在于从以正确答案为条件的推理过程后验分布中进行采样；我们使用一种简单的马尔可夫链蒙特卡洛（MCMC）期望最大化（EM）算法来解决该问题，该算法受到自学习推理器（STaR）、记忆化醒睡算法、马尔可夫得分爬升和持续对比散度的启发。该算法还引入了一种新颖的控制变量技术，随着模型的改进，该技术能够将梯度估计的方差降至零。我们将该技术应用于GSM8K和BIG-Bench Hard中的任务，发现这种MCMC-EM微调技术在留出样例上的准确率通常优于STaR或带或不带CoT的提示微调。
英文摘要：
Large language models (LLMs) solve problems more accurately and interpretably when instructed to work out the answer step by step using a ``chain-of-thought'' (CoT) prompt. One can also improve LLMs' performance on a specific task by supervised fine-tuning, i.e., by using gradient ascent on some tunable parameters to maximize the average log-likelihood of correct answers from a labeled training set. Naively combining CoT with supervised tuning requires supervision not just of the correct answers, but also of detailed rationales that lead to those answers; these rationales are expensive to produce by hand. Instead, we propose a fine-tuning strategy that tries to maximize the \emph{marginal} log-likelihood of generating a correct answer using CoT prompting, approximately averaging over all possible rationales. The core challenge is sampling from the posterior over rationales conditioned on the correct answer; we address it using a simple Markov-chain Monte Carlo (MCMC) expectation-maximization (EM) algorithm inspired by the self-taught reasoner (STaR), memoized wake-sleep, Markovian score climbing, and persistent contrastive divergence. This algorithm also admits a novel control-variate technique that drives the variance of our gradient estimates to zero as the model improves. Applying our technique to GSM8K and the tasks in BIG-Bench Hard, we find that this MCMC-EM fine-tuning technique typically improves the model's accuracy on held-out examples more than STaR or prompt-tuning with or without CoT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
