arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2310.14840cs.CL
收藏

源头透明:在可访问真实分布下评估和解释语言模型

Transparency at the Source: Evaluating and Interpreting Language Models With Access to the True Distribution

Institute for Logic, Language and Computation(逻辑、语言与计算研究所);University of Amsterdam(阿姆斯特丹大学)

Jaap Jumelet, Willem Zuidema

AI总结:

提出一种基于大规模概率语法生成人工类语言数据的设置,用于训练、评估和解释神经语言模型,通过访问真实分布计算困惑度下界并直接比较学习表示与符号规则,揭示不同架构和训练目标在学习动态上的显著差异。

摘要:

我们提出了一种用于训练、评估和解释神经语言模型的设置,该设置使用人工的、类语言的数据。该数据使用大规模概率语法(基于状态分割 PCFGs)生成,该语法本身源自大型自然语言语料库,但也使我们对生成过程拥有完全的控制权。我们描述并发布了语法和语料库,并测试了生成数据的自然性。这种方法使我们能够定义闭式表达式,以高效计算使用因果和掩码语言建模可获得的困惑度的精确下界。我们的结果表明,神经语言建模架构和训练目标在多大程度上允许近似困惑度下界方面存在显著差异。我们的方法还允许将学习到的表示直接与底层源中的符号规则进行比较。我们尝试了多种解释模型行为和学习动态的技术。通过访问底层真实源,我们的结果表明不同类别词在学习动态上存在显著差异和结果。

英文摘要:

We present a setup for training, evaluating and interpreting neural language models, that uses artificial, language-like data. The data is generated using a massive probabilistic grammar (based on state-split PCFGs), that is itself derived from a large natural language corpus, but also provides us complete control over the generative process. We describe and release both grammar and corpus, and test for the naturalness of our generated data. This approach allows us to define closed-form expressions to efficiently compute exact lower bounds on obtainable perplexity using both causal and masked language modelling. Our results show striking differences between neural language modelling architectures and training objectives in how closely they allow approximating the lower bound on perplexity. Our approach also allows us to directly compare learned representations to symbolic rules in the underlying source. We experiment with various techniques for interpreting model behaviour and learning dynamics. With access to the underlying true source, our results show striking differences and outcomes in learning dynamics between different classes of words.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑