arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2408.03062cs.CL
收藏

深度循环语言模型中论元结构构式的分析

Analysis of Argument Structure Constructions in a Deep Recurrent Language Model

University Erlangen-Nuremberg（埃尔朗根-纽伦堡大学）；University Hospital Erlangen（埃尔朗根大学医院）

Pegah Ramezani, Achim Schilling, Patrick Krauss

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究通过训练LSTM网络处理四种论元结构构式，并利用MDS、t-SNE和GDV分析隐藏层表征，发现循环神经语言模型能够自发形成与不同构式对应的聚类，表明其具有模拟人脑语言加工的潜力。

摘要：

理解语言和语言构式在大脑中的加工方式是认知计算神经科学中的一个基本问题。本研究探讨了循环神经语言模型中论元结构构式（Argument Structure Constructions, ASCs）的表征与加工。我们在一个自定义数据集上训练了一个长短期记忆（LSTM）网络，该数据集包含2000个句子，由GPT-4生成，代表四种不同的论元结构构式：及物构式、双及物构式、致使移动构式和结果构式。我们使用多维尺度分析（MDS）和t分布随机邻域嵌入（t-SNE）对LSTM模型隐藏层的内部激活进行了分析，以可视化句子表征。计算广义判别值（GDV）以量化这些表征中的聚类程度。结果表明，在所有隐藏层中，句子表征形成了与四种论元结构构式相对应的不同聚类，其中在输出层之前的最后一个隐藏层中观察到的聚类最为明显。这表明，即使是相对简单、受大脑约束的循环神经网络也能有效区分不同的构式类型。这些发现与先前研究一致，先前研究表明，在基于下一词预测任务训练的循环语言模型中会出现词类和句法规则表征。在未来的工作中，我们旨在使用更大的语言模型验证这些结果，并将其与连续言语感知过程中获得的神经影像数据进行对比。本研究突显了循环神经语言模型在反映人脑语言加工方面的潜力，为理解语言理解背后的计算和神经机制提供了有价值的见解。

英文摘要：

Understanding how language and linguistic constructions are processed in the brain is a fundamental question in cognitive computational neuroscience. In this study, we explore the representation and processing of Argument Structure Constructions (ASCs) in a recurrent neural language model. We trained a Long Short-Term Memory (LSTM) network on a custom-made dataset consisting of 2000 sentences, generated using GPT-4, representing four distinct ASCs: transitive, ditransitive, caused-motion, and resultative constructions. We analyzed the internal activations of the LSTM model's hidden layers using Multidimensional Scaling (MDS) and t-Distributed Stochastic Neighbor Embedding (t-SNE) to visualize the sentence representations. The Generalized Discrimination Value (GDV) was calculated to quantify the degree of clustering within these representations. Our results show that sentence representations form distinct clusters corresponding to the four ASCs across all hidden layers, with the most pronounced clustering observed in the last hidden layer before the output layer. This indicates that even a relatively simple, brain-constrained recurrent neural network can effectively differentiate between various construction types. These findings are consistent with previous studies demonstrating the emergence of word class and syntax rule representations in recurrent language models trained on next word prediction tasks. In future work, we aim to validate these results using larger language models and compare them with neuroimaging data obtained during continuous speech perception. This study highlights the potential of recurrent neural language models to mirror linguistic processing in the human brain, providing valuable insights into the computational and neural mechanisms underlying language understanding.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑