arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.13083cs.CL
收藏

InheritSumm:通过从GPT蒸馏得到的通用、多功能且紧凑的摘要器

InheritSumm: A General, Versatile and Compact Summarizer by Distilling from GPT

Microsoft Cognitive Services Research Group(微软认知服务研究组)

Yichong Xu, Ruochen Xu, Dan Iter, Yang Liu, Shuohang Wang, Chenguang Zhu, Michael Zeng

AI总结:

本文提出InheritSumm,通过从GPT-3.5蒸馏得到一个通用且紧凑的摘要模型,在保持与GPT-3.5相当的零样本和少样本性能的同时,支持高效微调并超越现有最优小模型。

摘要:

尽管GPT-3等大型模型在零样本和少样本摘要任务中表现出色,但其高昂的部署与微调成本阻碍了其在各类应用中的使用。相反,以往研究发现,尽管自动评估指标倾向于偏好较小的微调模型,但人工评估表明其生成摘要的质量逊于GPT-3等大模型。为解决此问题,我们提出InheritSumm,一种通过从GPT-3.5蒸馏获得的通用且紧凑的摘要模型。InheritSumm不仅展现出与GPT-3.5相当的零样本和少样本摘要能力,且足够紧凑以用于微调。实验结果表明,InheritSumm在零样本和少样本设置下取得与GPT-3.5相似或更优的性能。此外,在prefix-tuning和全数据微调场景中,它超越了此前最优的小模型。

英文摘要:

While large models such as GPT-3 demonstrate exceptional performance in zeroshot and fewshot summarization tasks, their extensive serving and fine-tuning costs hinder their utilization in various applications. Conversely, previous studies have found that although automatic metrics tend to favor smaller fine-tuned models, the quality of the summaries they generate is inferior to that of larger models like GPT-3 when assessed by human evaluators. To address this issue, we propose InheritSumm, a versatile and compact summarization model derived from GPT-3.5 through distillation. InheritSumm not only exhibits comparable zeroshot and fewshot summarization capabilities to GPT-3.5 but is also sufficiently compact for fine-tuning purposes. Experimental results demonstrate that InheritSumm achieves similar or superior performance to GPT-3.5 in zeroshot and fewshot settings. Furthermore, it outperforms the previously established best small models in both prefix-tuning and full-data fine-tuning scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑