arXiv 2305.140912023-09-28cs.CLcs.AI

重新审视可接受性判断

Revisiting Acceptability Judgements

Shanghai Jiao Tong University（上海交通大学）；City University of Hong Kong（香港城市大学）；University of Pennsylvania（宾夕法尼亚大学）；Brigham Young University（杨百翰大学）；University of Washington（华盛顿大学）；Indiana University Bloomington（印第安纳大学伯明顿分校）

Hai Hu, Ziyin Zhang, Weifang Huang, Jackie Yan-Ki Lai, Aini Li, Yina Patterson, Jiahui Huang, Peng Zhang, Chien-Jer Charles Lin, Rui Wang

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文构建首个中文大规模语言可接受性数据集 CoLAC，评估发现大语言模型表现不及监督模型与人类，并证明可接受性知识可跨语言迁移且源于预训练。

摘要：

在本研究中，我们在大型语言模型的背景下重新审视语言可接受性。我们介绍了 CoLAC——Corpus of Linguistic Acceptability in Chinese（中文语言可接受性语料库），这是首个面向非印欧语系语言的大规模可接受性数据集。它经过母语者验证，也是首个附带两组标签的可接受性数据集：语言学家标签和大众标签。我们的实验表明，即使是最大的 InstructGPT 模型在 CoLAC 上的表现也仅处于随机水平，而 ChatGPT 的性能（48.30 MCC）也远低于监督模型（59.03 MCC）和人类（65.11 MCC）。通过跨语言迁移实验和细粒度语言分析，我们对模型预测进行了详细分析，并首次证明语言可接受性知识可以在类型学上不同的语言之间迁移，也可以追溯到预训练阶段。我们的数据集已公开发布于 \url{https://github.com/huhailinguist/CoLAC}。

英文摘要：

In this work, we revisit linguistic acceptability in the context of large language models. We introduce CoLAC - Corpus of Linguistic Acceptability in Chinese, the first large-scale acceptability dataset for a non-Indo-European language. It is verified by native speakers and is the first acceptability dataset that comes with two sets of labels: a linguist label and a crowd label. Our experiments show that even the largest InstructGPT model performs only at chance level on CoLAC, while ChatGPT's performance (48.30 MCC) is also much below supervised models (59.03 MCC) and human (65.11 MCC). Through cross-lingual transfer experiments and fine-grained linguistic analysis, we provide detailed analysis of the model predictions and demonstrate for the first time that knowledge of linguistic acceptability can be transferred across typologically distinct languages, as well as be traced back to pre-training. Our dataset is publicly available at \url{https://github.com/huhailinguist/CoLAC}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。