arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.12440cs.CLcs.AI
收藏

增量式且数据高效的概念形成以支持掩码词预测

Incremental and Data-Efficient Concept Formation to Support Masked Word Prediction

Georgia Institute of Technology(佐治亚理工学院);Teachable AI Lab(可教人工智能实验室)

Xin Lian, Nishant Baglodi, Christopher J. MacLellan

AI总结:

本文提出Cobweb4L,一种基于Cobweb的增量式概念学习系统,通过多概念预测机制实现数据高效的掩码词预测,性能优于先前机制及Word2Vec,并在少数据下超越BERT。

摘要:

本文介绍了Cobweb4L,一种支持掩码词预测的高效语言模型学习的新方法。该方法基于Cobweb构建,Cobweb是一个增量式系统,学习概率概念的层次结构。每个概念存储出现在带有该概念标签的实例中的词的频率。该系统利用属性值表示将词及其周围上下文编码为实例。Cobweb4L使用类别效用的信息论变体,以及一种利用多个概念生成预测的新性能机制。我们证明,通过这些扩展,它显著优于仅使用单个节点生成预测的先前Cobweb性能机制。此外,我们证明Cobweb4L学习迅速,并达到与Word2Vec相当甚至更优的性能。接下来,我们展示在相同任务中,Cobweb4L和Word2Vec在较少训练数据下优于BERT。最后,我们讨论未来工作,以使我们的结论更加稳健和包容。

英文摘要:

This paper introduces Cobweb4L, a novel approach for efficient language model learning that supports masked word prediction. The approach builds on Cobweb, an incremental system that learns a hierarchy of probabilistic concepts. Each concept stores the frequencies of words that appear in instances tagged with that concept label. The system utilizes an attribute value representation to encode words and their surrounding context into instances. Cobweb4L uses the information theoretic variant of category utility and a new performance mechanism that leverages multiple concepts to generate predictions. We demonstrate that with these extensions it significantly outperforms prior Cobweb performance mechanisms that use only a single node to generate predictions. Further, we demonstrate that Cobweb4L learns rapidly and achieves performance comparable to and even superior to Word2Vec. Next, we show that Cobweb4L and Word2Vec outperform BERT in the same task with less training data. Finally, we discuss future work to make our conclusions more robust and inclusive.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑