稳健失败，保守修复：来自跨模型失败的文本知识蒸馏
Robust Failure, Conservative Repair: Textual Knowledge Distillation from Cross-Model Failures
浏览论文内容
中文总结 AI 辅助
针对基于失败的知识蒸馏不稳定问题，提出RFCR方法，从跨模型共享失败中提取规则并严格界定应用边界，在BIG-Bench Hard上提升准确率2.75个百分点。
中文摘要 AI 辅助
基于失败文本知识蒸馏旨在通过检查模型的任务错误来发现模型知识的空白。蒸馏出的知识可用于该模型（“源模型”）和其他模型的推理。然而，这种知识迁移可能不稳定。我们将规则原子定义为在推理时注入模型文本输入的独立规则。规则原子可以编码可迁移的任务知识或特定于模型的推理补丁，这些补丁可能混淆其他模型。此外，注入的规则原子可能被误用于不相关的情况，导致模型基于无关信息错误地翻转其答案。基于一个将训练示例蒸馏为特定任务速查表以辅助模型推理的流水线，我们研究了从失败中派生的规则何时能改进这些速查表。我们的早期实验表明，从单一模型的失败中进行的规则蒸馏在非源模型家族上表现不如基线速查表。这促使我们提出稳健失败，保守修复（RFCR），一种文本蒸馏程序，它从跨模型共享的失败中派生规则，使用边界案例锐化其应用边界，并在未找到有用规则时弃权（不执行）。在400项BIG-Bench Hard任务集上，RFCR将基线速查表从68.50%提升至71.25%（+2.75个百分点；95%置信区间[+1.25,+4.50]），且未在先前正确案例上造成性能下降。消融研究和跨模型诊断支持准确率提升既来自新知识的注入，也来自严格的规则应用控制。
英文摘要
Failure-based textual knowledge distillation aims to discover gaps in a model's knowledge by examining its task errors. The distilled knowledge can be useful for the reasoning of both this model ("source model") and other models. However, this transfer of knowledge may not be stable. We define a rule atom to be a standalone rule injected into a model's textual input at inference time. A rule atom can encode transferable task knowledge or model-specific reasoning patches that can confuse other models. Also, the injected rule atoms can be misapplied to unrelated cases, causing the model to incorrectly flip its answer based on irrelevant information. Building on a pipeline that distills training examples into task-specific cheat sheets that aid model reasoning, we examine when failure-derived rules can improve these cheat sheets. Our early experiment shows rule distillation from a single model's failures underperforms the baseline cheat sheet on non-source model families. This motivates Robust Failure, Conservative Repair (RFCR), a textual distillation procedure that derives rules from failures shared across models, sharpens their application boundaries using boundary cases, and abstains when no useful rule is found. On a 400-item BIG-Bench Hard task set, RFCR improves the baseline cheat sheets from 68.50% to 71.25% (+2.75 pp; 95% CI [+1.25,+4.50]) without performance degradation on previously correct cases. Ablations and cross-model diagnostics support that accuracy gains come from both new knowledge injection and strict rule-application control.
发表机构
- University of Chicago（芝加哥大学）
机构由 AI 辅助整理，请以论文原文为准。
