InheritSumm:通过从GPT蒸馏得到的通用、多功能且紧凑的摘要器
InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT
AI总结:
本文提出InheritSumm,通过从GPT-3.5蒸馏得到一个通用且紧凑的摘要模型,在保持与GPT-3.5相当的零样本和少样本性能的同时,支持高效微调并超越现有最优小模型。
摘要:
尽管GPT-3等大型模型在零样本和少样本摘要任务中表现出色,但其高昂的部署与微调成本阻碍了其在各类应用中的使用。相反,以往研究发现,尽管自动评估指标倾向于偏好较小的微调模型,但人工评估表明其生成摘要的质量逊于GPT-3等大模型。为解决此问题,我们提出InheritSumm,一种通过从GPT-3.5蒸馏获得的通用且紧凑的摘要模型。InheritSumm不仅展现出与GPT-3.5相当的零样本和少样本摘要能力,且足够紧凑以用于微调。实验结果表明,InheritSumm在零样本和少样本设置下取得与GPT-3.5相似或更优的性能。此外,在prefix-tuning和全数据微调场景中,它超越了此前最优的小模型。
英文摘要:
While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely, previous studies have found that although automatic metrics tend to favor smaller fine-tuned models, the quality of the summaries they generate is inferior to that of larger models like GPT-3 when assessed by human evaluators. To address this issue, we propose InheritSumm, a versatile and compact summarization model derived from GPT-3.5 through distillation. InheritSumm not only exhibits comparable zeroshot and fewshot summarization capabilities to GPT-3.5 but is also sufficiently compact for fine-tuning purposes. Experimental results demonstrate that InheritSumm achieves similar or superior performance to GPT-3.5 in zeroshot and fewshot settings. Furthermore, it outperforms the previously established best small models in both prefix-tuning and full-data fine-tuning scenarios.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。