arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.12368cs.CL
收藏

一种用于NLI模型领域泛化的合成数据方法

A synthetic data approach for domain generalization of NLI models

Google Deepmind(谷歌DeepMind)

Mohammad Javad Hosseini, Andrey Petrov, Alex Fabrikant, Annie Louis

AI总结:

本文提出一种生成多样化领域和长度的高质量合成NLI数据方法,使模型在零样本下游任务中实现更好的领域泛化,并在TRUE基准上显著提升性能。

摘要:

自然语言推理(NLI)仍然是大语言模型(LLMs)的一项重要基准任务。NLI数据集是向其他语义任务进行迁移学习的跳板,NLI模型也是识别模型生成文本忠实度的标准工具。如今已有多个大规模NLI数据集,模型通过在这些集合上进行“爬山式”优化已取得巨大进步。然而,它们在分布外/领域外数据上的实际表现仍不太清楚。我们探索了利用高质量合成数据集使NLI模型能够在新颖且未见过的文本领域中零样本应用于下游任务的机会。我们展示了一种在多样领域和长度上生成NLI数据的新方法,这些领域和长度目前尚未被现有训练集覆盖。所生成的示例具有有意义的前提,假设以创造性方式形成,而不是对少数前提词元进行简单编辑,并且标签具有高准确率。我们表明,在此数据(685K个合成示例)上训练的模型对全新的下游测试环境具有最佳泛化能力。在TRUE基准上,使用我们的数据训练的T5-small模型与在最佳替代数据集上训练相比,平均提升约7%。对于更小的模型,提升更为显著,而在T5 XXL模型上仍然具有意义。我们还证明,当域内训练数据与我们的领域通用合成数据结合增强时,在测试集上也能获得增益。

英文摘要:

Natural Language Inference (NLI) remains an important benchmark task for LLMs. NLI datasets are a springboard for transfer learning to other semantic tasks, and NLI models are standard tools for identifying the faithfulness of model-generated text. There are several large scale NLI datasets today, and models have improved greatly by hill-climbing on these collections. Yet their realistic performance on out-of-distribution/domain data is less well-understood. We explore the opportunity for synthetic high-quality datasets to adapt NLI models for zero-shot use in downstream applications across new and unseen text domains. We demonstrate a new approach for generating NLI data in diverse domains and lengths, so far not covered by existing training sets. The resulting examples have meaningful premises, the hypotheses are formed in creative ways rather than simple edits to a few premise tokens, and the labels have high accuracy. We show that models trained on this data ($685$K synthetic examples) have the best generalization to completely new downstream test settings. On the TRUE benchmark, a T5-small model trained with our data improves around $7\%$ on average compared to training on the best alternative dataset. The improvements are more pronounced for smaller models, while still meaningful on a T5 XXL model. We also demonstrate gains on test sets when in-domain training data is augmented with our domain-general synthetic data.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑