arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2106.00737cs.CL
收藏

神经语言模型中意义的内隐表征

Implicit Representations of Meaning in Neural Language Models

Massachusetts Institute of Technology(麻省理工学院)

Belinda Z. Li, Maxwell Nye, Jacob Andreas

摘要:

神经语言模型的有效性是否完全来自于对表面词共现统计数据的准确建模,或者这些模型是否代表和推理它们所描述的世界?在BART和T5 Transformer语言模型中,我们识别了上下文单词表示,这些单词表示在整个话语中演变时充当实体和情况的模型。这些神经表征与动态语义的语言模型具有功能相似性:它们支持每个实体当前属性和关系的线性读出,并且可以对语言生成产生可预测的影响进行操纵。我们的研究结果表明,在预训练的神经语言模型的预测支持,至少部分地,通过动态表示的意义和实体状态的隐式模拟,这种行为可以学习只有文本作为训练数据。代码和数据可在https://github.com/belindal/state-probes上获得。

英文摘要:

Does the effectiveness of neural language models derive entirely from accurate modeling of surface word co-occurrence statistics, or do these models represent and reason about the world they describe? In BART and T5 transformer language models, we identify contextual word representations that function as models of entities and situations as they evolve throughout a discourse. These neural representations have functional similarities to linguistic models of dynamic semantics: they support a linear readout of each entity's current properties and relations, and can be manipulated with predictable effects on language generation. Our results indicate that prediction in pretrained neural language models is supported, at least in part, by dynamic representations of meaning and implicit simulation of entity state, and that this behavior can be learned with only text as training data. Code and data are available at https://github.com/belindal/state-probes .

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑