arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2410.17161cs.CLcs.LGcs.LO
收藏

可互换词元嵌入:用于可扩展词汇表与Alpha等价性

Interchangeable Token Embeddings for Extendable Vocabulary and Alpha-Equivalence

Department of Computer Engineering, Middle East Technical University, Ankara, Turkey(中欧技术大学计算机工程系);Microsoft, İstanbul, Turkey(微软公司)

İlker Işık, Ramazan Gokberk Cinbis, Ebru Aydin Gol

AI总结:

针对语言模型缺乏可互换词元概念的问题,提出双部分词元嵌入策略,在多个形式逻辑任务上提升了对未见词元的泛化能力,并为alpha等价性提供了有利的归纳偏置。

摘要:

语言模型缺乏可互换词元的概念:即语义等价但不同的符号,例如形式逻辑中的约束变量。这一局限性阻碍了模型向更大词汇表的泛化,并妨碍了模型识别alpha等价性的能力,即重命名约束变量不改变语义。我们形式化了这一机器学习问题,并引入了alpha协方差,一种用于评估对此类变换鲁棒性的度量。为解决该任务,我们提出了一种双部分词元嵌入策略:共享组件确保语义一致性,而随机化组件保持词元的可区分性。与依赖alpha重命名进行数据增强的基线相比,我们的方法在解决线性时序逻辑、命题逻辑赋值预测以及可扩展词汇表复制任务中,对未见词元展现出更好的泛化能力,同时为alpha等价性引入了有利的归纳偏置。我们的发现为设计能够学习可互换词元表示的语言模型奠定了基础,这是向形式领域更灵活、更系统推理迈出的关键一步。我们的代码和项目页面可在 https://necrashter.github.io/interchangeable-token-embeddings 获取。

英文摘要:

Language models lack the notion of interchangeable tokens: symbols that are semantically equivalent yet distinct, such as bound variables in formal logic. This limitation prevents generalization to larger vocabularies and hinders the model's ability to recognize alpha-equivalence, where renaming bound variables preserves meaning. We formalize this machine learning problem and introduce alpha-covariance, a metric for evaluating robustness to such transformations. To tackle this task, we propose a dual-part token embedding strategy: a shared component ensures semantic consistency, while a randomized component maintains token distinguishability. Compared to a baseline that relies on alpha-renaming for data augmentation, our approach demonstrates improved generalization to unseen tokens in linear temporal logic solving, propositional logic assignment prediction, and copying with an extendable vocabulary, while introducing a favorable inductive bias for alpha-equivalence. Our findings establish a foundation for designing language models that can learn interchangeable token representations, a crucial step toward more flexible and systematic reasoning in formal domains. Our code and project page are available at https://necrashter.github.io/interchangeable-token-embeddings

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑