Natural Language Processing (NLP) has undergone transformative changes with the advent of deep learning methodologies. One challenge persistently confronting researchers is the scarcity of high-quality, annotated datasets that drive these models. This paper explores the nuances of synthetic data generation in NLP, with a focal point on template-based question generation. By assessing its advantages, including data augmentation potential and the introduction of structured variety, we juxtapose these benefits against inherent limitations, such as the risk of overfitting and the constraints posed by pre-defined templates. Drawing from empirical evaluations, we demonstrate the impact of template-based synthetic data on the performance of modern transformer models. We conclude by emphasizing the delicate balance required between synthetic and real-world data, and the future trajectories of integrating synthetic data in model training pipelines. The findings aim to guide NLP practitioners in harnessing synthetic data's potential, ensuring optimal model performance in diverse applications.
AI "News" Content Farms Are Easy to Make and Hard to Detect: A Case Study in Italian
Istituto di Scienza e Tecnologia dell’Informazione “A. Faedo”(A. Faedo信息科学与技术研究所);IT University of Copenhagen(哥本哈根信息技术大学);Istituto di Linguistica Computazionale “Antonio Zampolli”(Antonio Zampolli计算语言学研究所)
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment
Allen Institute for AI(艾伦人工智能研究所)
指令学习范式--模型仅从任务描述中学习执行新任务--在通用模型研究中变得流行。然而,大型Transformer模型作为指令学习者的能力仍然知之甚少。我们使用受控的合成环境来表征这种能力。具体来说,我们使用决定给定字符串是否匹配正则表达式(视为指令)的任务来识别使指令学习具有挑战性的任务、指令和实例的属性。例如,我们发现我们的模型,一个微调的基于T5的text 2 text Transformer,与大型正则语言斗争,这表明不太精确的指令对模型来说是一个挑战。另外,需要跟踪先前步骤的较长上下文的指令执行也更困难。我们使用我们的研究结果系统地构建了一个具有挑战性的指令学习数据集,我们称之为硬RegSet。通过对硬RegSet进行微调,我们的大型Transformer只能正确解释65.6%的测试指令(至少90%的准确率),以及分布外泛化设置中11%-24%的指令。我们提出了硬RegSet作为一个具有挑战性的指令学习任务,和一个可控的环境研究指令学习。
Does a Tool Result Carry More Authority Than Plain Text? Three Prospective Studies of False-Claim Adoption in a Synthetic Assignment Task with Claude Opus 5
Corabo(科拉博公司)
研究Claude Opus 5在合成任务中,工具结果与纯文本对虚假主张采纳的影响,发现工具结果的权威性未高于已宣布的内联文本。
Marginal Alignment Does Not Guarantee Joint-Distribution Fidelity: An Official-Reference Audit of Nemotron-Personas-Korea with Cross-Locale Replication
Korea Advanced Institute of Science and Technology (KAIST)(韩国科学技术院)
Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure