Trace-of-Thought Prompting：通过问题分解探究基于提示的知识蒸馏
Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition
AI总结：
提出Trace-of-Thought Prompting框架，通过问题分解将高资源大模型推理能力蒸馏给低资源小模型，在GSM8K和MATH上显著提升准确率，减少对专有大模型的依赖。
摘要：
知识蒸馏允许较小的神经网络以更低的计算需求模拟大型教师模型的性能。针对大型语言模型（LLMs）的传统方法通常需要大量的微调，这限制了其可及性。为解决此问题，我们提出了Trace-of-Thought Prompting，这是一种新颖的框架，旨在将关键推理能力从高资源教师模型（超过80亿参数）蒸馏到低资源学生模型（不超过80亿参数）中。该方法利用问题分解来增强可解释性，并促进人在回路的干预。在GSM8K和MATH数据集上的实证评估表明，学生模型在GSM8K上的准确率提升了113%，在MATH上提升了21%，其中在Llama 2和Zephyr等较小模型上的提升尤为显著。我们的结果为开源、低资源模型最终同时充当学生和教师模型提供了一条有前景的途径，有望减少对高资源专有模型的依赖。
英文摘要：
Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
