arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.13671cs.CLcs.AI
收藏

让我们逐步合成:通过从小模型外推误差来使用大语言模型迭代合成数据集

Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models

HKUST(香港科技大学);AIWaves Inc.(AIWaves公司);ETH Zürich(苏黎世联邦理工学院)

Ruida Wang, Wangchunshu Zhou, Mrinmaya Sachan

AI总结:

本文提出数据合成框架 S3,利用大语言模型迭代外推小模型在验证集上的误差,缩小合成数据与真实数据的分布差距,显著提升小模型在多个 NLP 任务上的性能。

摘要:

数据合成是一种在极少标注数据下训练小模型的有前景的方法。数据合成的一种方法是利用大语言模型的丰富知识为小模型合成伪训练样本,从而同时实现数据和计算效率。然而,数据合成的一个关键挑战是,合成数据集往往与真实任务数据分布存在较大的分布差异。因此,在本文中,我们提出了 Synthesis Step by Step(S3),这是一种数据合成框架,通过使用大语言模型在小规模真实世界验证集上迭代外推小模型在合成数据集上训练时产生的误差,从而缩小这种分布差距。在多个 NLP 任务上的大量实验表明,我们的方法通过缩小合成数据集与真实数据之间的差距,提升了小模型的性能,与几个基线相比取得了显著提升:相比 ZeroGen 提升了 9.48%,相比 GoldGen 提升了 2.73%,并且与在人工标注数据上训练的小模型相比最多提升了 15.17%。

英文摘要:

*Data Synthesis* is a promising way to train a small model with very little labeled data. One approach for data synthesis is to leverage the rich knowledge from large language models to synthesize pseudo training examples for small models, making it possible to achieve both data and compute efficiency at the same time. However, a key challenge in data synthesis is that the synthesized dataset often suffers from a large distributional discrepancy from the *real task* data distribution. Thus, in this paper, we propose *Synthesis Step by Step* (**S3**), a data synthesis framework that shrinks this distribution gap by iteratively extrapolating the errors made by a small model trained on the synthesized dataset on a small real-world validation dataset using a large language model. Extensive experiments on multiple NLP tasks show that our approach improves the performance of a small model by reducing the gap between the synthetic dataset and the real data, resulting in significant improvement compared to several baselines: 9.48% improvement compared to ZeroGen and 2.73% compared to GoldGen, and at most 15.17% improvement compared to the small model trained on human-annotated data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑