arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2402.10575cs.LGcs.AI
收藏

用于自监督序列学习的符号自编码

Symbolic Autoencoding for Self-Supervised Sequence Learning

EPFL(洛桑联邦理工学院);Univ. Grenoble Alpes, CNRS, Grenoble INP, LIG(格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔理工学院、信息与信号处理实验室(LIG))

Mohammad Hossein Amani, Nicolas Mario Baldwin, Amin Mansouri, Martin Josifoski, Maxime Peyrard, Robert West

AI总结:

本文提出符号自编码($Σ$AE)自监督框架,通过离散瓶颈层连接两个生成模型,利用非并行数据提升符号系统转换任务在并行数据稀缺时的性能。

摘要:

传统语言模型擅长文本序列的下一个 token 预测,但在不同符号系统之间的转换任务中常常遇到困难,尤其是在并行数据稀缺的情况下。为了解决这个问题,我们引入了符号自编码($Σ$AE),这是一个自监督框架,利用大量非并行数据和有限的并行数据。$Σ$AE 通过一个离散瓶颈层连接两个生成模型,并通过最小化重建损失(同时与并行数据的监督损失一起)进行端到端优化,使得由离散瓶颈生成的序列可以被读取为转换后的输入序列。我们还开发了基于梯度的方法,尽管瓶颈存在离散性,仍能实现高效的自监督序列学习。我们的结果表明,$Σ$AE 显著提升了转换任务的性能,即使在并行数据极少的情况下,也为弱监督学习场景提供了有前景的解决方案。

英文摘要:

Traditional language models, adept at next-token prediction in text sequences, often struggle with transduction tasks between distinct symbolic systems, particularly when parallel data is scarce. Addressing this issue, we introduce \textit{symbolic autoencoding} ($Σ$AE), a self-supervised framework that harnesses the power of abundant unparallel data alongside limited parallel data. $Σ$AE connects two generative models via a discrete bottleneck layer and is optimized end-to-end by minimizing reconstruction loss (simultaneously with supervised loss for the parallel data), such that the sequence generated by the discrete bottleneck can be read out as the transduced input sequence. We also develop gradient-based methods allowing for efficient self-supervised sequence learning despite the discreteness of the bottleneck. Our results demonstrate that $Σ$AE significantly enhances performance on transduction tasks, even with minimal parallel data, offering a promising solution for weakly supervised learning scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑