To make sense of massive data, we often fit simplified models and then interpret the parameters; for example, we cluster the text embeddings and then interpret the mean parameters of each cluster. However, these parameters are often high-dimensional and hard to interpret. To make model parameters directly interpretable, we introduce a family of statistical models -- including clustering, time series, and classification models -- parameterized by natural language predicates. For example, a cluster of text about COVID could be parameterized by the predicate "discusses COVID". To learn these statistical models effectively, we develop a model-agnostic algorithm that optimizes continuous relaxations of predicate parameters with gradient descent and discretizes them by prompting language models (LMs). Finally, we apply our framework to a wide range of problems: taxonomizing user chat dialogues, characterizing how they evolve across time, finding categories where one language model is better than the other, clustering math problems based on subareas, and explaining visual features in memorable images. Our framework is highly versatile, applicable to both textual and visual domains, can be easily steered to focus on specific properties (e.g. subareas), and explains sophisticated concepts that classical methods (e.g. n-gram analysis) struggle to produce.
Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules?
IDLab, Ghent University - imec(根特大学-imec IDLab实验室)
当前,在不同的基于神经网络的自然语言处理算法中使用了语义级特征。然而,人们对这些模型学习的字符级模式知之甚少。此外,模型往往只是定量比较,而缺乏定性分析。在本文中,我们研究了神经网络学习哪些字符级模式,以及这些模式是否与手动定义的分词和注释一致。为此,我们将上下文分解技术(Murdoch et al. 2018)扩展到卷积神经网络,这使我们能够比较卷积神经网络和双向长短期记忆网络。我们评估和比较这些模型的任务,形态标记三种不同的语言,并表明这些模型隐含地发现可理解的语言规则。我们的实现可以在https://github.com/FredericGodin/ContextualDecomposition-NLP上找到。
Using natural language processing techniques to extract information on the properties and functionalities of energetic materials from large text corpora
University of Maryland, College Park(马里兰大学帕克分校);Energetics Technology Center(能量技术中心)