arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.15030cs.LG
收藏

神经网络在通用多模态推理过程中的泛化能力

On the generalization capacity of neural networks during generic multimodal reasoning

T.J. Watson Research Center, IBM Research(IBM研究院T.J.沃森研究中心)

Takuya Ito, Soham Dan, Mattia Rigotti, James Kozloski, Murray Campbell

AI总结:

本文构建多模态问答基准gCOG,系统比较RNN、Transformer、Perceiver等架构在三类OOD泛化上的表现,揭示注意力机制的作用与局限。

摘要:

Transformer的出现推动了大型语言模型(LLM)的发展,这类模型似乎展现出类似人类的能力。为评估此类模型以及多种其他基础神经网络架构在多模态领域的通用性,我们评估并比较了它们的多模态泛化能力。我们引入了一个多模态问答基准,用于评估三种特定类型的分布外(OOD)泛化性能:干扰项泛化(在存在干扰项时的泛化)、系统性组合泛化(对新任务排列的泛化)以及生产性组合泛化(对更复杂任务结构的泛化)。我们发现,在各种模型架构(例如RNN、Transformer、Perceiver等)中,具有多个注意力层的模型,或在输入域之间利用交叉注意力机制的模型,表现更好。我们的积极结果表明,对于多模态干扰项泛化和系统性泛化,跨模态注意力或具有更深注意力层的模型,是整合多模态输入所需的关键架构特征。另一方面,这两种架构特征都没有带来生产性泛化,这表明现有架构在特定类型的多模态泛化方面存在根本局限。这些结果展示了现代神经模型用于多模态推理的特定架构组件的优势与局限。最后,我们提供了Generic COG(gCOG),这是一个包含多个多模态泛化划分的可配置基准,供未来研究探索。

英文摘要:

The advent of the Transformer has led to the development of large language models (LLM), which appear to demonstrate human-like capabilities. To assess the generality of this class of models and a variety of other base neural network architectures to multimodal domains, we evaluated and compared their capacity for multimodal generalization. We introduce a multimodal question-answer benchmark to evaluate three specific types of out-of-distribution (OOD) generalization performance: distractor generalization (generalization in the presence of distractors), systematic compositional generalization (generalization to new task permutations), and productive compositional generalization (generalization to more complex tasks structures). We found that across model architectures (e.g., RNNs, Transformers, Perceivers, etc.), models with multiple attention layers, or models that leveraged cross-attention mechanisms between input domains, fared better. Our positive results demonstrate that for multimodal distractor and systematic generalization, either cross-modal attention or models with deeper attention layers are key architectural features required to integrate multimodal inputs. On the other hand, neither of these architectural features led to productive generalization, suggesting fundamental limitations of existing architectures for specific types of multimodal generalization. These results demonstrate the strengths and limitations of specific architectural components underlying modern neural models for multimodal reasoning. Finally, we provide Generic COG (gCOG), a configurable benchmark with several multimodal generalization splits, for future studies to explore.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑