arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2102.01223cs.CLcs.LG
收藏

利用动态容量槽注意力从字符序列中归纳有意义单元

Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention

École Polytechnique Fédérale de Lausanne (EPFL)(洛桑联邦理工学院);Idiap Research Institute(伊迪亚普研究所)

Melika Behjati, James Henderson

AI总结:

本文提出动态容量槽注意力模型,以无监督方式从字符序列中发现连续的有意义单元表示,并在多语言探测实验中被验证能捕获与先前单元相似且更高抽象层次的信息。

摘要:

字符本身不传达意义,但字符序列可以。我们提出一种无监督的分布方法,用于学习字符序列中抽象的有意义单元。不同于对序列进行切分,我们的动态容量槽注意力(Dynamic Capacity Slot Attention)模型发现序列中对象的连续表示,扩展了一种用于图像中对象发现的架构。我们在不同语言上训练模型,并使用正向和反向探测分类器评估所获得表示的质量。这些实验表明,我们的模型成功发现了在形式、内容和抽象层次上与先前提出的单元相似的单元,并且有望在更高抽象层次上捕获有意义的信息。

英文摘要:

Characters do not convey meaning, but sequences of characters do. We propose an unsupervised distributional method to learn the abstract meaningful units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images. We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers. These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑