arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00322cs.CL
收藏

LLM生成的自然语言与标度律相遇:新探索与数据增强方法

LLM-Generated Natural Language Meets Scaling Laws: New Explorations and Data Augmentation Methods

School of Information Resource Management, Renmin University of China(中国人民大学信息资源管理学院);School of Computing, National University of Singapore(新加坡国立大学计算机学院)

Zhenhua Wang, Guang Xu, Ming Ren

AI总结:

本文通过标度律分析LLM生成自然语言与人类自然语言的差异,并提出基于模糊计算的少样本文本分类数据增强方法ZGPTDA,显著提升分类性能。

摘要:

随着大语言模型(LLM)的兴起,自然语言处理取得了诸多进展,例如基于LLM的数据增强。然而,先前的研究存在两个主要问题:首先,缺乏对LLM生成的自然语言(LLMNL)是否真正与人类自然语言(HNL)一致这一关键基础性问题的思考;其次,忽视了增强数据是由LLM随机生成的这一事实,这意味着并非所有数据都具有同等的训练价值,这可能会妨碍分类器的性能。为应对这些挑战,我们引入标度律来从本质上计算LLMNL和HNL。通过大量实验,我们揭示了LLMNL与Mandelbrot定律存在轻微偏差(约0.2的Mandelbrot指数),强调了HNL在复杂度上的优势,并补充了关于语言风格的解释性讨论。这为LLM的扩展奠定了坚实基础。此外,我们提出了一种用于少样本文本分类的新型数据增强方法,称为ZGPTDA,该方法利用由符合标度律驱动的模糊计算机制来对GPT-4增强数据做出决策。在真实场景中进行的大量实验证实了ZGPTDA的有效性(将Bert和RoBerta的F1提高了7-10%)和竞争力(在DeBerta上比最近的AugGPT和GENCO方法的准确率高出约2%)。此外,我们还揭示了一些有趣的见解,例如Hilberg定律和Taylor定律可以为文本分类带来更多益处等。

英文摘要:

With the ascent of large language models (LLM), natural language processing has witnessed enhancements, such as LLM-based data augmentation. Nonetheless, prior research harbors two primary concerns: firstly, a lack of contemplation regarding whether the natural language generated by LLM (LLMNL) truly aligns with human natural language (HNL), a critical foundational question; secondly, an oversight that augmented data is randomly generated by LLM, implying that not all data may possess equal training value, that could impede the performance of classifiers. To address these challenges, we introduce the scaling laws to intrinsically calculate LLMNL and HNL. Through extensive experiments, we reveal slight deviations (approximately 0.2 Mandelbrot exponent) from Mandelbrot's law in LLMNL, underscore a complexity advantage in HNL, and supplement an interpretive discussion on language style. This establishes a solid foundation for LLM's expansion. Further, we introduce a novel data augmentation method for few-shot text classification, termed ZGPTDA, which leverages fuzzy computing mechanisms driven by the conformity to scaling laws to make decisions about GPT-4 augmented data. Extensive experiments, conducted in real-world scenarios, confirms the effectiveness (improving F1 of Bert and RoBerta by 7-10%) and competitiveness (surpassing recent AugGPT and GENCO methods by about 2% accuracy on DeBerta) of ZGPTDA. In addition, we reveal some interesting insights, e.g., Hilberg's law and Taylor's law can impart more benefits to text classification, etc.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑