有限监督与非一致输入下的双向英汉翻译学习
Learning Bidirectional Action-Language Translation with Limited Supervision and Incongruent Input
摘要：
人类婴儿的学习发生在探索环境的过程中，通过与物体的互动，以及通过随意地倾听和重复话语，这类似于无监督学习。只有偶尔，一个正在学习的婴儿会收到一个匹配的口头描述，描述他正在做的一个动作，这类似于监督学习。这种学习机制可以用深度学习来模仿。我们使用配对门控自动编码器（PGAE）模型对这种弱监督学习范式进行建模，该模型结合了动作和语言自动编码器。在观察到减少监督训练比例时的性能下降后，我们引入了配对转换自编码器（PTAE）模型，使用基于转换器的跨模态注意力。PTAE在语言到动作和动作到语言的翻译中实现了显著更高的准确性，特别是在现实但困难的情况下，只有很少的监督训练样本可用。我们还测试了训练后的模型是否真实地表现出冲突的多模态输入。根据心理学中不一致的概念，冲突恶化了模型的输出。排除动作输入比冲突语言输入的影响更大，并且冲突特征越多，干扰越大。PTAE可以在标记数据很少的情况下在大多数未标记的数据上进行训练，并且在使用不一致的输入进行测试时表现得很好。
英文摘要：
Human infant learning happens during exploration of the environment, by interaction with objects, and by listening to and repeating utterances casually, which is analogous to unsupervised learning. Only occasionally, a learning infant would receive a matching verbal description of an action it is committing, which is similar to supervised learning. Such a learning mechanism can be mimicked with deep learning. We model this weakly supervised learning paradigm using our Paired Gated Autoencoders (PGAE) model, which combines an action and a language autoencoder. After observing a performance drop when reducing the proportion of supervised training, we introduce the Paired Transformed Autoencoders (PTAE) model, using Transformer-based crossmodal attention. PTAE achieves significantly higher accuracy in language-to-action and action-to-language translations, particularly in realistic but difficult cases when only few supervised training samples are available. We also test whether the trained model behaves realistically with conflicting multimodal input. In accordance with the concept of incongruence in psychology, conflict deteriorates the model output. Conflicting action input has a more severe impact than conflicting language input, and more conflicting features lead to larger interference. PTAE can be trained on mostly unlabelled data where labeled data is scarce, and it behaves plausibly when tested with incongruent input.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
