arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2306.16638cs.CL
收藏

GPT的否定检测评估:基于xNot360数据集的分析

A negation detection assessment of GPTs: analysis with the xNot360 dataset

National Institute of Informatics(国立信息学研究所);University of Alberta(阿尔伯塔大学);Imperial College London(帝国理工学院);Royal Holloway, University of London(伦敦大学皇家霍洛威学院)

Ha Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas Stathis, Ken Satoh

AI总结:

本研究使用零样本预测和xNot360数据集评估了GPT系列模型的否定检测性能,发现GPT-4表现最佳但整体能力依然有限,揭示了其在高风险领域逻辑可靠性的重要性。

摘要:

否定是自然语言的基本方面,在沟通与理解中发挥着关键作用。本研究评估了生成式预训练Transformer(GPT)模型的否定检测性能,具体包括GPT-2、GPT-3、GPT-3.5和GPT-4。我们专注于使用应用于定制xNot360数据集的零样本预测方法来识别自然语言中的否定。我们的方法检查了带有标签的句子对,以指示第二个句子是否否定了第一个句子。研究结果揭示了GPT模型之间存在显著的性能差异,其中GPT-4超越了同类模型,而GPT-3.5则表现出明显的性能下降。GPT模型在否定检测方面的整体熟练程度仍然相对一般,表明该任务挑战了其自然语言理解能力的边界。我们不仅强调了GPT模型在处理否定时的局限性,还强调了在医疗、科学和法律等高风险领域逻辑可靠性的重要性。

英文摘要:

Negation is a fundamental aspect of natural language, playing a critical role in communication and comprehension. Our study assesses the negation detection performance of Generative Pre-trained Transformer (GPT) models, specifically GPT-2, GPT-3, GPT-3.5, and GPT-4. We focus on the identification of negation in natural language using a zero-shot prediction approach applied to our custom xNot360 dataset. Our approach examines sentence pairs labeled to indicate whether the second sentence negates the first. Our findings expose a considerable performance disparity among the GPT models, with GPT-4 surpassing its counterparts and GPT-3.5 displaying a marked performance reduction. The overall proficiency of the GPT models in negation detection remains relatively modest, indicating that this task pushes the boundaries of their natural language understanding capabilities. We not only highlight the constraints of GPT models in handling negation but also emphasize the importance of logical reliability in high-stakes domains such as healthcare, science, and law.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑