使用无标签数据进行思维链提示的解释选择
Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting
AI总结：
针对思维链提示中解释差异影响下游任务准确率的问题，提出一种黑盒两阶段框架：利用留一法生成候选解释，并基于对数似然和准确率等代理指标结合银标签开发集搜索最优解释组合，有效提升了文本推理任务的提示效果。
摘要：
最近的研究展示了如何用解释提示大型语言模型以在文本推理任务上获得出色表现，即思维链范式。然而，细微不同的解释会导致下游任务准确率大相径庭。未经任务“调优”的解释，如非专家编写的现成解释，可能导致平庸的表现。本文解决如何以黑盒方式优化注入解释的提示的问题。我们首先使用留一法方案为提示中的每个示例生成候选解释集，然后通过两阶段框架寻找这些解释的有效组合。我们首先根据两个代理指标（对数似然和在新示例上的准确率）孤立地评估每个上下文示例的解释。然后，我们搜索解释组合，以找到在银标签开发集上表现优异的组合。在涵盖问答、数学推理和自然语言推理的四个文本推理任务中，结果表明我们的代理指标与真实准确率相关，且整体方法能有效改进提示，优于众包标注和朴素搜索策略。
英文摘要：
Recent work has shown how to prompt large language models with explanations to obtain strong performance on textual reasoning tasks, i.e., the chain-of-thought paradigm. However, subtly different explanations can yield widely varying downstream task accuracy. Explanations that have not been "tuned" for a task, such as off-the-shelf explanations written by nonexperts, may lead to mediocre performance. This paper tackles the problem of how to optimize explanation-infused prompts in a blackbox fashion. We first generate sets of candidate explanations for each example in the prompt using a leave-one-out scheme, then find an effective combination of these explanations with a two-stage framework. We first evaluate explanations for each in-context example in isolation according to two proxy metrics, log likelihood and accuracy on new examples. Then, we search over combinations of explanations to find one that yields high performance against a silver-labeled development set. Across four textual reasoning tasks spanning question answering, mathematical reasoning, and natural language inference, results show that our proxy metrics correlate with ground truth accuracy and our overall method can effectively improve prompts over crowdworker annotations and naive search strategies
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
