LLM 的元认知能力：数学问题解决中的探索
Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
AI总结：
本文通过提示引导的技能标注、语义聚类及示例检索，验证强大 LLM 在数学推理中具备元认知能力，并提升多模型在 GSM8K 与 MATH 上的表现。
摘要：
元认知知识指人类对自身思维和推理过程的直觉性知识。当今最优秀的 LLM 显然具备某些推理过程。本文给出证据表明，它们也具有元认知知识，包括在给定任务时说出可应用的技能和程序的能力。我们主要在数学推理语境中探索这一点，开发了一种提示引导的交互程序，使强大的 LLM 为数学问题分配合理的技能标签，随后让其执行语义聚类，以获得更粗粒度的技能标签族。这些粗粒度技能标签对人类而言具有可解释性。为验证这些技能标签是有意义的，并且与 LLM 的推理过程相关，我们进行了以下实验。(a) 我们要求 GPT-4 为数学数据集 GSM8K 和 MATH 中的训练问题分配技能标签。(b) 在使用 LLM 解决测试问题时，我们向其展示完整的技能标签列表，并要求它识别所需技能。然后向其展示与该技能标签相关的、随机选择的已解答示例问题。这提高了若干强大 LLM 在 GSM8K 和 MATH 上的准确率，其中包括代码辅助模型。尽管本文将该方法应用于数学问题，但所提出的方法论是领域无关的。
英文摘要：
Metacognitive knowledge refers to humans' intuitive knowledge of their own thinking and reasoning processes. Today's best LLMs clearly possess some reasoning processes. The paper gives evidence that they also have metacognitive knowledge, including ability to name skills and procedures to apply given a task. We explore this primarily in context of math reasoning, developing a prompt-guided interaction procedure to get a powerful LLM to assign sensible skill labels to math questions, followed by having it perform semantic clustering to obtain coarser families of skill labels. These coarse skill labels look interpretable to humans. To validate that these skill labels are meaningful and relevant to the LLM's reasoning processes we perform the following experiments. (a) We ask GPT-4 to assign skill labels to training questions in math datasets GSM8K and MATH. (b) When using an LLM to solve the test questions, we present it with the full list of skill labels and ask it to identify the skill needed. Then it is presented with randomly selected exemplar solved questions associated with that skill label. This improves accuracy on GSM8k and MATH for several strong LLMs, including code-assisted models. The methodology presented is domain-agnostic, even though this article applies it to math problems.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
