COFFEQA：面向大语言模型问答的反事实引导少样本蒸馏
COFFEQA: Counterfactual-Guided Few-Shot Distillation for LLM Question Answering
AI总结：
COFFEQA是一种用于大语言模型问答的反事实引导少样本蒸馏策略，通过引入CFD指标验证事实-反事实对，在多个问答基准上优于仅事实的蒸馏基线。
摘要：
知识蒸馏旨在训练一个更小的学生模型，以模仿更大的教师模型，从而适配资源受限的环境。然而，蒸馏过程通常需要大量数据且计算成本高昂，尤其对于生成式任务而言更是如此。少样本蒸馏提供了一种替代方案，它利用有限的训练样本传递教师模型的知识。但我们发现，少样本蒸馏的性能高度依赖于这些训练样本的选择。实际上，我们证明，使用教师模型决策边界附近的不确定性较高的样本会显著降低蒸馏性能。受这一观察结果的启发，我们提出了面向开卷问答的反事实引导少样本蒸馏策略（COFFEQA），这是一种利用反事实为具有挑战性的事实样本提供额外监督的新方法。在生成式问答场景中，我们将反事实定义为经过最小程度修改的上下文，该上下文会促使教师模型将其响应更改为替代答案。我们提供了几何视角，说明事实-反事实对如何约束教师模型与学生模型决策边界之间的偏差。我们还引入了一种新指标——上下文遵循难度（CFD），该指标在蒸馏前基于与预期替代答案的一致性以及对修改后上下文的依赖性，进一步验证反事实的有效性。COFFEQA使用经CFD验证的事实-反事实对进行少样本蒸馏。在使用T5和Qwen2.5系列模型的5个域内和3个域外问答基准上开展的实验表明，在相同的少样本预算下，COFFEQA的性能优于仅使用事实的蒸馏基线。例如，在T5设置下，当使用32个少样本对时，COFFEQA在TriviaQA上较仅使用事实的基线提升了8.77的平均精确匹配（EM）点数和9.98的平均F1点数。
英文摘要：
Knowledge distillation aims to train a smaller student model to mimic a larger teacher model for resource-constrained environments. However, the process of distillation is typically data-intensive and computationally expensive, particularly for generative tasks. Few-shot distillation offers an alternative by transferring teacher knowledge using limited training samples. However, we observe that few-shot distillation performance is heavily dependent on the choice of these training samples. In fact, we demonstrate that using more uncertain samples near the teacher's decision boundary can significantly deteriorate distillation performance. Motivated by this observation, we propose Counterfactual-guided Few-shot Distillation in Open-book Question Answering (COFFEQA), a novel strategy that aptly uses counterfactuals to provide additional supervision around challenging factual samples. In the generative QA setting, we define a counterfactual as a minimally modified context that induces the teacher to change its response to an alternative answer. We provide geometric insights on how factual-counterfactual pairs can constrain the deviation between the teacher's and student's decision boundaries. We further introduce a new metric, Context Following Difficulty (CFD) that further validates counterfactuals prior to distillation based on agreement with the intended alternative answer and dependence on the modified context. COFFEQA uses CFD-validated factual-counterfactual pairs for few-shot distillation. Experiments across five in-domain and three out-of-domain QA benchmarks using T5 and Qwen2.5 families show that COFFEQA outperforms factual-only distillation baselines under the same few-shot budget. For instance, on TriviaQA, COFFEQA improves over the factual-only baseline by 8.77 average EM points and 9.98 average F1 points under the T5 setting with 32 few-shot pairs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
