三重编码器:共同激活的表征,共同连接
Triple-Encoders: Representations That Fire Together, Wire Together
AI总结:
针对基于搜索的对话模型每轮重编码历史成本高、双编码器独立编码忽略语境化的问题,提出三重编码器,通过赫布启发的无权重局部共现学习目标自组织计算分布式话语混合表征,相比双编码器显著提升,零样本泛化优于单向量模型且无需重编码。
摘要:
基于搜索的对话模型通常在每一轮都重新编码对话历史,导致成本高昂。曲线对比学习(Curved Contrastive Learning)是一种通过双编码器将话语之间的相对距离编码到嵌入空间的表征学习方法,近期在对话建模中展现出可观效果,且效率远超传统方法。虽然通过独立编码话语实现了高效率,但这忽略了语境化的重要性。为解决这一问题,本研究提出三重编码器(triple-encoders),该模型受赫布理论启发,通过一种新颖的共现学习目标,以自组织方式从这些独立编码的话语中高效计算分布式话语混合表征,无需使用任何权重,即仅通过局部交互实现。实验结果表明,三重编码器相比双编码器有显著提升,甚至在无需重新编码的情况下,零样本泛化能力优于单向量表征模型。我们的代码(https://github.com/UKPLab/acl2024-triple-encoders)和模型(https://huggingface.co/UKPLab/triple-encoders-dailydialog)均已公开。
英文摘要:
Search-based dialog models typically re-encode the dialog history at every turn, incurring high cost. Curved Contrastive Learning, a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder, has recently shown promising results for dialog modeling at far superior efficiency. While high efficiency is achieved through independently encoding utterances, this ignores the importance of contextualization. To overcome this issue, this study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances through a novel hebbian inspired co-occurrence learning objective in a self-organizing manner, without using any weights, i.e., merely through local interactions. Empirically, we find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models without requiring re-encoding. Our code (https://github.com/UKPLab/acl2024-triple-encoders) and model (https://huggingface.co/UKPLab/triple-encoders-dailydialog) are publicly available.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。