arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2603.17475cs.CL
收藏

人类与变换器语言模型:抽象驱动语言学习

Humans and transformer LMs: Abstraction drives language learning

Stanford University(斯坦福大学)

Jasper Jian, Christopher D. Manning

AI总结:

研究比较变换器语言模型在训练过程中对语言类别的学习行为,发现抽象层面的行为早于词汇层面,揭示抽象对语言学习的重要性。

摘要:

分类是人类语言能力的核心组成部分。本文研究变换器语言模型如何通过比较其训练过程中的行为,与抽象特征基础和具体实例基础的人类语言学习理论进行对比,探讨词汇语义和句法类别如何通过基于分歧的度量方法显现。在GPT-2 small实验中发现,当构造被学习时,抽象类别的行为早于词汇项特定行为,并且不同的语言行为在训练过程中以不同点突然出现,揭示抽象在语言学习中的关键作用。此结果为人类语言学习模型提供了存在证明。

英文摘要:

Categorization is a core component of human linguistic competence. We investigate how a transformer-based language model (LM) learns linguistic categories by comparing its behaviour over the course of training to behaviours which characterize abstract feature-based and concrete exemplar-based accounts of human language acquisition. We investigate how lexical semantic and syntactic categories emerge using novel divergence-based metrics that track learning trajectories using next-token distributions. In experiments with GPT-2 small, we find that (i) when a construction is learned, abstract class-level behaviour is evident at earlier steps than lexical item-specific behaviour, and (ii) that different linguistic behaviours emerge abruptly in sequence at different points in training, revealing that abstraction plays a key role in how LMs learn. This result informs the models of human language acquisition that LMs may serve as an existence proof for.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑