arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2406.04289cs.CL
收藏

哪些语言易于进行语言建模?来自学习概率正则语言的视角

What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages

Københavns Universitet(哥本哈根大学);ETH Zürich(苏黎世联邦理工学院);Universität Zürich(苏黎世大学)

Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, Ryan Cotterell

AI总结:

本研究从概率正则语言学习视角出发,通过实验探究RNN与Transformer语言模型对正则语言模型的可学习性,发现RLM秩、采样字符串期望长度是两类模型可学习性的强显著预测因子,其余预测因子的显著性模式存在差异。

摘要:

大语言模型能够学习什么?根据定义,语言模型(LM)是字符串上的概率分布。因此,解决上述问题的一个直观思路是将其形式化为字符串分布类的可学习性问题。该方向的已有研究聚焦于评估理论极限,而我们则试图理解经验层面的可学习性。与以往的实证研究不同,我们在神经语言模型的核心任务——学习概率语言——上对其进行评估,而非将其作为形式语言的分类器。具体而言,我们研究了循环神经网络(RNN)和Transformer语言模型对正则语言模型(RLM)的可学习性。我们通过实验测试了RLM的可学习性与RLM的各类复杂度参数、神经语言模型隐藏状态大小之间的函数关系。研究发现,对应于RLM条件分布logits张成的线性空间大小的RLM秩,以及采样字符串的期望长度,是RNN和Transformer可学习性的强显著预测因子。其他几个预测因子也达到了显著性水平,但在RNN和Transformer之间呈现出不同的模式。

英文摘要:

What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formalize it as a matter of learnability of classes of distributions over strings. While prior work in this direction focused on assessing the theoretical limits, in contrast, we seek to understand the empirical learnability. Unlike prior empirical work, we evaluate neural LMs on their home turf-learning probabilistic languages-rather than as classifiers of formal languages. In particular, we investigate the learnability of regular LMs (RLMs) by RNN and Transformer LMs. We empirically test the learnability of RLMs as a function of various complexity parameters of the RLM and the hidden state size of the neural LM. We find that the RLM rank, which corresponds to the size of linear space spanned by the logits of its conditional distributions, and the expected length of sampled strings are strong and significant predictors of learnability for both RNNs and Transformers. Several other predictors also reach significance, but with differing patterns between RNNs and Transformers.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑