通过自对比 Bregman 散度学习实现高效的文档嵌入
Efficient Document Embeddings via Self-Contrastive Bregman Divergence Learning
AI总结:
针对长文档嵌入的效率与质量问题,本文提出结合 SimCSE 自对比孪生网络与基于 Bregman 散度的神经网络的混合方法,在法律和生物医学长文档分类任务上优于基线方法。
摘要:
学习高质量的文档嵌入是自然语言处理(NLP)、信息检索(IR)、推荐系统和搜索引擎中的一个基本问题。尽管近年来基于 Transformer 的模型在通过自对比学习生成句子嵌入方面取得了进展,但在效率和质量方面,长文档(数千个词)的编码仍然具有挑战性。因此,我们使用最先进的无监督对比学习方法(SimCSE)训练基于 Longformer 的文档编码器。此外,我们用基于函数 Bregman 散度的额外凸神经网络补充了基线方法——孪生神经网络,旨在提高输出文档表示的质量。我们展示了自对比孪生网络与我们提出的神经 Bregman 网络的组合在法律和生物医学领域的三个长文档主题分类任务的两个线性分类设置中总体上优于基线方法。
英文摘要:
Learning quality document embeddings is a fundamental problem in natural language processing (NLP), information retrieval (IR), recommendation systems, and search engines. Despite recent advances in the development of transformer-based models that produce sentence embeddings with self-contrastive learning, the encoding of long documents (Ks of words) is still challenging with respect to both efficiency and quality considerations. Therefore, we train Longfomer-based document encoders using a state-of-the-art unsupervised contrastive learning method (SimCSE). Further on, we complement the baseline method -- siamese neural network -- with additional convex neural networks based on functional Bregman divergence aiming to enhance the quality of the output document representations. We show that overall the combination of a self-contrastive siamese network and our proposed neural Bregman network outperforms the baselines in two linear classification settings on three long document topic classification tasks from the legal and biomedical domains.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。