利用多模态深度神经网络从视觉美学中解耦语言
Using Multimodal Deep Neural Networks to Disentangle Language from Visual Aesthetics
AI总结：
本研究通过对单模态视觉、单模态语言及多模态DNN模型表征进行线性解码预测人类美感评分，发现美感体验主要源于前馈感知计算，语言相关知识贡献有限。
摘要：
当我们将一种视觉刺激体验为美的时候，这种体验中有多少来自我们无法描述的感知计算，又有多少来自我们可以轻易转化为自然语言的概念知识？通过行为范式或神经影像学在视觉诱发的情感和美学体验中解耦感知与语言，在实证上通常是难以实现的。在此，我们通过对单模态视觉、单模态语言以及多模态（语言对齐的）深度神经网络（DNN）模型的习得表征进行线性解码，来预测人类对自然图像的美感评分，从而规避了这一挑战。我们发现，单模态视觉模型（例如SimCLR）解释了这些评分中绝大多数的可解释方差。与单模态视觉模型相比，语言对齐的视觉模型（例如SLIP）带来的增益很小。以视觉嵌入为条件、通过CLIPCap生成字幕的单模态语言模型（例如GPT2）并未带来进一步的增益。仅使用字幕嵌入得到的预测准确性低于图像和字幕嵌入结合（拼接）的情况。综上所述，这些结果表明，无论我们最终能找到怎样的词语来描述自身的美感体验，前馈感知的不可言说的计算可能为这种体验提供了充分的基础。
英文摘要：
When we experience a visual stimulus as beautiful, how much of that experience derives from perceptual computations we cannot describe versus conceptual knowledge we can readily translate into natural language? Disentangling perception from language in visually-evoked affective and aesthetic experiences through behavioral paradigms or neuroimaging is often empirically intractable. Here, we circumnavigate this challenge by using linear decoding over the learned representations of unimodal vision, unimodal language, and multimodal (language-aligned) deep neural network (DNN) models to predict human beauty ratings of naturalistic images. We show that unimodal vision models (e.g. SimCLR) account for the vast majority of explainable variance in these ratings. Language-aligned vision models (e.g. SLIP) yield small gains relative to unimodal vision. Unimodal language models (e.g. GPT2) conditioned on visual embeddings to generate captions (via CLIPCap) yield no further gains. Caption embeddings alone yield less accurate predictions than image and caption embeddings combined (concatenated). Taken together, these results suggest that whatever words we may eventually find to describe our experience of beauty, the ineffable computations of feedforward perception may provide sufficient foundation for that experience.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
