多语言视觉推理中缺失了什么以及如何解决
What Is Missing in Multilingual Visual Reasoning and How to Fix It
AI总结：
针对多语言视觉推理任务中开源模型性能落后于专有系统的问题，分析出多语言性等三大挑战，提出三种针对性干预措施，大幅提升开源模型零样本性能并略微改善GPT-4V性能。
摘要：
如今的NLP模型致力于支持多种语言和模态，以提高不同用户的可访问性。在本文中，我们通过视觉推理任务测试来评估它们的多语言、多模态能力。我们观察到，像GPT-4V这样的专有系统目前在该任务上性能最佳，但开源模型相比之下较为落后。令人惊讶的是，GPT-4V在英语和其他语言之间表现出相似的性能，这表明跨语言公平系统开发具有潜力。我们对模型失败案例的分析揭示了使该任务具有挑战性的三个关键方面：多语言性、复杂推理和多模态性。为应对这些挑战，我们提出了三种针对性干预措施，包括解决多语言性的“翻译-测试”方法、分解复杂推理的视觉编程方法，以及利用图像描述来解决多模态性的方法。我们的干预措施在零样本设置下实现了该任务的最佳开源性能，将开源模型LLaVA-v1.5-13B提升13.4%、LLaVA-v1.6-34B提升20.3%、Qwen-VL提升16.7%，同时还略微提高了GPT-4V的性能。
英文摘要：
NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users. In this paper, we evaluate their multilingual, multimodal capabilities by testing on a visual reasoning task. We observe that proprietary systems like GPT-4V obtain the best performance on this task now, but open models lag in comparison. Surprisingly, GPT-4V exhibits similar performance between English and other languages, indicating the potential for equitable system development across languages. Our analysis on model failures reveals three key aspects that make this task challenging: multilinguality, complex reasoning, and multimodality. To address these challenges, we propose three targeted interventions including a translate-test approach to tackle multilinguality, a visual programming approach to break down complex reasoning, and a method that leverages image captioning to address multimodality. Our interventions achieve the best open performance on this task in a zero-shot setting, boosting open models LLaVA-v1.5-13B by 13.4%, LLaVA-v1.6-34B by 20.3%, and Qwen-VL by 16.7%, while also minorly improving GPT-4V's performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
