VisLingInstruct：通过自主指令优化提升多模态语言模型的零样本学习能力
VisLingInstruct: Elevating Zero-Shot Learning in Multi-Modal Language Models with Autonomous Instruction Optimization
AI总结：
针对多模态语言模型零样本性能依赖指令质量的问题，提出VisLingInstruct方法，通过上下文学习自主优化指令并改进视觉特征提取模块，在TextVQA、HatefulMemes数据集上准确率较SOTA分别提升13.1%、9%。
摘要：
本文提出VisLingInstruct，这是一种推动多模态语言模型（MMLMs）零样本学习发展的新方法。当前MMLMs在多模态任务中展现出出色的零样本能力，但其性能高度依赖指令质量。VisLingInstruct通过上下文学习（In-Context Learning）自主评估并优化指令文本，以此解决该问题，提升MMLMs中视觉感知与语言表达的协同性。在优化指令的同时，我们还优化了MMLMs的视觉特征提取模块，进一步增强其对文本内容的响应能力。我们基于FlanT5和Vicuna对MMLMs开展了全面实验，结果表明VisLingInstruct显著提升了视觉多模态任务的零样本性能。值得注意的是，在TextVQA和HatefulMemes数据集上，其准确率相比现有最优水平分别提升了13.1%和9%。我们的核心代码可在https://github.com/Zhudongsheng75/VisLingInstruct获取。
英文摘要：
This paper presents VisLingInstruct, a novel approach to advancing Multi-Modal Language Models (MMLMs) in zero-shot learning. Current MMLMs show impressive zero-shot abilities in multi-modal tasks, but their performance depends heavily on the quality of instructions. VisLingInstruct tackles this by autonomously evaluating and optimizing instructional texts through In-Context Learning, improving the synergy between visual perception and linguistic expression in MMLMs. Alongside this instructional advancement, we have also optimized the visual feature extraction modules in MMLMs, further augmenting their responsiveness to textual content. Our comprehensive experiments on MMLMs, based on FlanT5 and Vicuna, show that VisLingInstruct significantly improves zero-shot performance in visual multi-modal tasks. Notably, it achieves a 13.1% and 9% increase in accuracy over the prior state-of-the-art on the TextVQA and HatefulMemes datasets. Our main code is available at https://github.com/Zhudongsheng75/VisLingInstruct.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
