视觉与语言模型中短语定位与任务性能的联合研究
A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models
AI总结：
提出联合研究任务性能与短语定位的框架及三个基准，发现当代模型在定位短语与解决任务间存在不一致，并通过暴力训练解决该问题。
摘要：
在视觉语境中推理自然语言的任务，关键在于将单词和短语定位到图像区域。然而，观察当代模型中的这种定位十分复杂，即使通常预期，若以利于泛化的方式解决任务，这种定位会发生。我们提出了一个联合研究任务性能和短语定位的框架，并提出了三个基准来研究两者之间的关系。结果表明，当代模型在定位短语的能力与解决任务的能力之间存在不一致。我们展示了如何通过对短语定位标注进行暴力训练来解决这一问题，并分析了由此产生的动态。代码和资料可在 https://github.com/lil-lab/phrase_grounding 获取。
英文摘要：
Key to tasks that require reasoning about natural language in visual contexts is grounding words and phrases to image regions. However, observing this grounding in contemporary models is complex, even if it is generally expected to take place if the task is addressed in a way that is conductive to generalization. We propose a framework to jointly study task performance and phrase grounding, and propose three benchmarks to study the relation between the two. Our results show that contemporary models demonstrate inconsistency between their ability to ground phrases and solve tasks. We show how this can be addressed through brute-force training on ground phrasing annotations, and analyze the dynamics it creates. Code and at available at https://github.com/lil-lab/phrase_grounding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
