Transformer工作记忆中符号表征的复杂度与任务复杂度相关
Complexity of Symbolic Representation in Working Memory of Transformer Correlates with the Complexity of a Task
AI总结：
针对Transformer缺乏显式记忆的问题，研究了其解码器新增的符号工作记忆的特性，发现该记忆可提升机器翻译质量，其内容与文本相关，且存储的词元、词性多样性与语料复杂度正相关。
摘要：
尽管Transformer已被广泛应用于自然语言处理任务（尤其是机器翻译），但它们缺乏用于存储已处理文本核心概念的显式记忆。本文探究了添加至Transformer模型解码器的符号工作记忆的内容特性。这类工作记忆提升了模型在机器翻译任务中的预测质量，可作为对模型做出正确翻译而言重要的信息的神经-符号表征。对记忆内容的研究显示，翻译文本的关键词被存储在工作记忆中，表明记忆内容与所处理文本具有相关性。此外，记忆中存储的词元和词性的多样性与机器翻译任务的语料库复杂度相关。
英文摘要：
Even though Transformers are extensively used for Natural Language Processing tasks, especially for machine translation, they lack an explicit memory to store key concepts of processed texts. This paper explores the properties of the content of symbolic working memory added to the Transformer model decoder. Such working memory enhances the quality of model predictions in machine translation task and works as a neural-symbolic representation of information that is important for the model to make correct translations. The study of memory content revealed that translated text keywords are stored in the working memory, pointing to the relevance of memory content to the processed text. Also, the diversity of tokens and parts of speech stored in memory correlates with the complexity of the corpora for machine translation task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
