如何充分利用大语言模型的语法知识进行可接受性判断
How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgments
AI总结：
本文通过提示和模板比较九种判断方法，发现in-template LP和Yes/No概率计算两种方法在英汉语法可接受性判断上优于传统方法，且二者集成效果更佳。
摘要：
语言模型（LMs）的语法知识通常通过语言最小对基准来测量，在该基准中，语言模型被呈现一对可接受和不可接受的句子，并被要求判断哪个更可接受。传统方法直接比较语言模型赋予的句子概率，但近期的大语言模型（LLMs）被训练为通过提示来执行任务，因此它们赋予的原始概率可能无法完全反映其语法知识。在本研究中，我们尝试利用提示和模板从大语言模型中推导出更准确的可接受性判断。通过在英语和汉语中进行大量实验，我们比较了九种判断方法，并发现其中两种方法——一种概率读出方法“模板内语言模型概率（in-template LP）”和一种基于提示的方法“是/否概率计算（Yes/No probability computing）”——比传统方法取得了更高的准确率。我们的分析表明，这些方法在不同的语言现象上表现优异，表明它们获取了大语言模型知识的不同方面。我们还发现，将这两种方法集成使用优于单一方法。因此，我们推荐这些技术，无论是单独使用还是集成使用，作为评估大语言模型语法知识时比传统方法更有效的替代方案。
英文摘要：
The grammatical knowledge of language models (LMs) is often measured using a benchmark of linguistic minimal pairs, where the LMs are presented with a pair of acceptable and unacceptable sentences and required to judge which is more acceptable. Conventional approaches directly compare sentence probabilities assigned by LMs, but recent large language models (LLMs) are trained to perform tasks via prompting, and thus, the raw probabilities they assign may not fully reflect their grammatical knowledge. In this study, we attempt to derive more accurate acceptability judgments from LLMs using prompts and templates. Through extensive experiments in English and Chinese, we compare nine judgment methods and find two of them, a probability readout method -- in-template LP and a prompt-based method -- Yes/No probability computing, achieve higher accuracy than the conventional ones. Our analysis reveals that these methods excel in different linguistic phenomena, suggesting they access different aspects of LLMs' knowledge. We also find that ensembling the two methods outperforms single methods. Consequently, we recommend these techniques, either individually or ensembled, as more effective alternatives to conventional approaches for assessing grammatical knowledge in LLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
