arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.15941cs.CL
收藏

这不是一个数据集:一个挑战大型语言模型的大型否定基准

This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models

HiTZ Center - Ixa, University of the Basque Country UPV/EHU(巴斯克大学HiTZ中心-Ixa);LoRea Group, University of the Basque Country UPV/EHU(巴斯克大学LoRea研究组)

Iker García-Ferrero, Begoña Altuna, Javier Álvez, Itziar Gonzalez-Dios, German Rigau

AI总结:

本文针对LLMs难以理解否定的问题,构建了包含约40万条常识描述句的大型否定基准,通过零样本测试和微调实验发现,LLMs在处理否定句时依赖表面线索且泛化能力持续不足。

摘要:

尽管大型语言模型(LLMs)显然已经获得了一定程度的语法知识和泛化能力,但它们无法解释否定,这是自然语言处理中的一个关键步骤。我们试图澄清LLMs理解否定表现欠佳的原因。我们引入了一个大型半自动生成的数据集,包含约400,000条关于常识知识的描述性句子,这些句子可为真或假,其中约2/3的语料以不同形式包含否定。我们在零样本方法下将我们的数据集与现有最大的开源LLMs一起使用,以掌握它们的泛化和推理能力,并且我们还对一些模型进行了微调,以评估是否可以训练对否定的理解。我们的研究结果表明,虽然LLMs擅长分类肯定句,但它们在否定句上存在困难,并且缺乏对否定的深刻理解,通常依赖于表面线索。尽管在否定句上微调模型提高了它们的性能,但在处理否定时缺乏泛化能力的问题依然存在,这凸显了LLMs在否定理解和泛化方面持续存在的挑战。数据集和代码是公开可用的。

英文摘要:

Although large language models (LLMs) have apparently acquired a certain level of grammatical knowledge and the ability to make generalizations, they fail to interpret negation, a crucial step in Natural Language Processing. We try to clarify the reasons for the sub-optimal performance of LLMs understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge that can be true or false in which negation is present in about 2/3 of the corpus in different forms. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability and we have also fine-tuned some of the models to assess whether the understanding of negation can be trained. Our findings show that, while LLMs are proficient at classifying affirmative sentences, they struggle with negative sentences and lack a deep understanding of negation, often relying on superficial cues. Although fine-tuning the models on negative sentences improves their performance, the lack of generalization in handling negation is persistent, highlighting the ongoing challenges of LLMs regarding negation understanding and generalization. The dataset and code are publicly available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑