arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2602.17848cs.CL
收藏

关于闭合概率与语言模型下一个词预测之间的缩放关系

On the scaling relationship between cloze probabilities and language model next-token prediction

University at Buffalo(布法罗大学);Université Paris Nanterre(巴黎南泰尔大学)

Cassandra L. Jacobs, Morgan Grobol

AI总结:

本研究探讨了语言模型在闭合概率与下一个词预测之间的缩放关系,发现更大模型在语义上更符合人类响应,但对低层信息不敏感。

摘要:

近期研究表明,更大的语言模型在预测眼动和阅读时间数据方面具有更好的预测能力。尽管即使最好的模型在人类响应上也低估了概率质量,但更大的模型在闭合数据中对下一个词及其生成可能性的估计质量更高,因为它们在词汇共现统计上不敏感,但更符合人类闭合响应的语义。这些结果支持了更大模型更大的记忆容量有助于猜测更符合语义的词,但使它们对与词识别相关低层信息不敏感的主张。

英文摘要:

Recent work has shown that larger language models have better predictive power for eye movement and reading time data. While even the best models under-allocate probability mass to human responses, larger models assign higher-quality estimates of next tokens and their likelihood of production in cloze data because they are less sensitive to lexical co-occurrence statistics while being better aligned semantically to human cloze responses. The results provide support for the claim that the greater memorization capacity of larger models helps them guess more semantically appropriate words, but makes them less sensitive to low-level information that is relevant for word recognition.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑