arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.01746cs.CL
收藏

ChatGPT是一个高流利度的语法错误纠正系统吗?一项全面评估

Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluation

University of Macau(澳门大学);The Chinese University of Hong Kong (Shenzhen)(香港中文大学(深圳));Soochow University(苏州大学)

Tao Fang, Shu Yang, Kaixin Lan, Derek F. Wong, Jinpeng Hu, Lidia S. Chao, Yue Zhang

AI总结:

本文全面评估了ChatGPT在语法错误纠正(GEC)任务中的能力,通过设计零样本和少样本思维链设置进行多语言及文档级测试,发现其错误检测与流利度表现出色,但在跨句错误纠正及遵循最小编辑原则上存在局限。

摘要:

ChatGPT是基于先进GPT-3.5架构的大规模语言模型,在各类自然语言处理(NLP)任务中展现出巨大潜力。然而,目前缺乏全面研究探讨其在语法错误纠正(GEC)领域的潜力。为展示其在GEC中的能力,我们利用上下文学习为ChatGPT设计了零样本思维链和少样本思维链设置。我们的评估包括测试ChatGPT在三种不同语言的五个官方测试集,以及三个英文文档级GEC测试集上的表现。实验结果与人工评估表明,ChatGPT具备出色的错误检测能力,能自由纠正错误使句子非常流利,这可能归因于其过度纠正倾向及未遵循最小编辑原则。此外,其在非英语和低资源设置下的表现突显了其在多语言GEC任务中的潜力。然而,对文档级各类错误的进一步分析表明,ChatGPT无法有效纠正跨句的协议、共指、时态错误及跨句边界错误。

英文摘要:

ChatGPT, a large-scale language model based on the advanced GPT-3.5 architecture, has shown remarkable potential in various Natural Language Processing (NLP) tasks. However, there is currently a dearth of comprehensive study exploring its potential in the area of Grammatical Error Correction (GEC). To showcase its capabilities in GEC, we design zero-shot chain-of-thought (CoT) and few-shot CoT settings using in-context learning for ChatGPT. Our evaluation involves assessing ChatGPT's performance on five official test sets in three different languages, along with three document-level GEC test sets in English. Our experimental results and human evaluations demonstrate that ChatGPT has excellent error detection capabilities and can freely correct errors to make the corrected sentences very fluent, possibly due to its over-correction tendencies and not adhering to the principle of minimal edits. Additionally, its performance in non-English and low-resource settings highlights its potential in multilingual GEC tasks. However, further analysis of various types of errors at the document-level has shown that ChatGPT cannot effectively correct agreement, coreference, tense errors across sentences, and cross-sentence boundary errors.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑