arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.11968cs.CLcs.LG
收藏

合成数据作为基准的有效性

Efficacy of Synthetic Data as a Benchmark

Diabolocom;University of Mons(蒙斯大学)

Gaurav Maheshwari, Dmitry Ivanov, Kevin El Haddad

AI总结:

该研究探究LLM生成的合成数据作为NLP任务基准的有效性,发现其效果随任务复杂度变化,还提出偏差因子指标并发现小模型对自生成数据存在偏差,建议从业者尽量用多模型大模型生成的数据。

摘要:

大语言模型(LLMs)已在零样本和少样本学习场景中催生了一系列应用,包括生成用于训练和测试的合成数据集。然而,要可靠地使用这些合成数据集,必须了解它们对真实世界数据的代表性如何。我们通过评估利用LLM生成合成数据并将其用作各类自然语言处理(NLP)任务基准的有效性,对此展开了研究。我们在6个数据集、3种不同任务上开展的实验表明,尽管合成数据能有效捕捉各类方法在意图分类等较简单任务上的性能,但在命名实体识别等更复杂的任务上表现不足。此外,我们提出了一种名为偏差因子(bias factor)的新指标,用于评估当使用同一个LLM既生成基准数据又执行任务时引入的偏差。我们发现,较小的LLM会对自身生成的数据表现出偏差,而更大的模型则不会。总体而言,我们的研究结果表明,合成数据作为基准的有效性因任务而异,从业者应尽可能依赖多个更大模型生成的数据。

英文摘要:

Large language models (LLMs) have enabled a range of applications in zero-shot and few-shot learning settings, including the generation of synthetic datasets for training and testing. However, to reliably use these synthetic datasets, it is essential to understand how representative they are of real-world data. We investigate this by assessing the effectiveness of generating synthetic data through LLM and using it as a benchmark for various NLP tasks. Our experiments across six datasets, and three different tasks, show that while synthetic data can effectively capture performance of various methods for simpler tasks, such as intent classification, it falls short for more complex tasks like named entity recognition. Additionally, we propose a new metric called the bias factor, which evaluates the biases introduced when the same LLM is used to both generate benchmarking data and to perform the tasks. We find that smaller LLMs exhibit biases towards their own generated data, whereas larger models do not. Overall, our findings suggest that the effectiveness of synthetic data as a benchmark varies depending on the task, and that practitioners should rely on data generated from multiple larger models whenever possible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑