在多模态上下文中预测句子可接受性判断
Predicting Sentence Acceptability Judgments in Multimodal Contexts
AI总结：
研究探讨了在多模态上下文中，视觉上下文对人类和LLM句子可接受性判断的影响，发现LLM在去除视觉上下文时表现更优，且不同模型的判断分布各异。
摘要：
先前的研究已经考察了深度神经网络（DNNs），特别是Transformer，在预测人类句子可接受性判断方面的能力，无论是独立于上下文还是在文档上下文中。我们考虑了人类和大语言模型（LLMs）在先前接触视觉图像（即视觉上下文）对这些判断的影响。我们的结果表明，与文本上下文相比，视觉图像似乎对人类可接受性评分几乎没有或没有影响。然而，LLMs在文档上下文中显示出与之前在人类判断上的压缩效应相同的现象。不同种类的LLMs能够以较高的准确性预测人类可接受性判断，但总体而言，当去除视觉上下文时，它们的性能略好。此外，LLM判断的分布因模型而异，Qwen与人类模式相似，而其他模型则偏离。LLM生成的句子可接受性预测在一般情况下与它们的标准化对数概率高度相关。然而，当存在视觉上下文时，相关性会降低，这表明在存在视觉上下文时，LLM内部表示与生成预测之间存在更大的差距。我们的实验工作表明，人类和LLM在多模态上下文中处理句子时存在有趣的相似性和差异性。
英文摘要：
Previous work has examined the capacity of deep neural networks (DNNs), particularly transformers, to predict human sentence acceptability judgments, both independently of context, and in document contexts. We consider the effect of prior exposure to visual images (i.e., visual context) on these judgments for humans and large language models (LLMs). Our results suggest that, in contrast to textual context, visual images appear to have little if any impact on human acceptability ratings. However, LLMs display the compression effect seen in previous work on human judgments in document contexts. Different sorts of LLMs are able to predict human acceptability judgments to a high degree of accuracy, but in general, their performance is slightly better when visual contexts are removed. Moreover, the distribution of LLM judgments varies among models, with Qwen resembling human patterns, and others diverging from them. LLM-generated predictions on sentence acceptability are highly correlated with their normalised log probabilities in general. However, the correlations decrease when visual contexts are present, suggesting that a higher gap exists between the internal representations of LLMs and their generated predictions in the presence of visual contexts. Our experimental work suggests interesting points of similarity and of difference between human and LLM processing of sentences in multimodal contexts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
