Chain-of-Caption: 无需训练提升多模态大语言模型的指称表达理解
Chain-of-Caption: Training-free improvement of multimodal large language model on referring expression comprehension
AI总结：
本文提出无需训练的Chain-of-Caption框架，通过结合多种上下文提升多模态大语言模型在指称表达理解任务中的性能。
摘要：
给定一个文本描述，指称表达理解（REC）任务涉及在图像中定位被提及的对象。多模态大语言模型（MLLMs）通过扩大模型规模和训练数据，在REC基准测试中实现了高精度。此外，通过链式思维和技术如工具使用，可以进一步提高MLLMs的性能，提供额外的视觉或文本上下文。在本文中，我们分析了通过工具使用提供额外视觉和文本上下文的各种技术对MLLM及其REC任务的影响。此外，我们提出了一种无需训练的框架，名为Chain-of-Caption，以提升MLLMs的REC性能。我们在RefCOCO/RefCOCOg/RefCOCO+和Ref-L4数据集上进行了实验，并展示了单独的文本或视觉上下文可以在无需微调的情况下提高REC性能。通过结合多种上下文，我们的无需训练框架在不同交集过联（IoU）阈值下，相对于基线模型在准确性上实现了5%到30%的性能提升。
英文摘要：
Given a textual description, the task of referring expression comprehension (REC) involves the localisation of the referred object in an image. Multimodal large language models (MLLMs) have achieved high accuracy on REC benchmarks through scaling up the model size and training data. Moreover, the performance of MLLMs can be further improved using techniques such as Chain-of-Thought and tool use, which provides additional visual or textual context to the model. In this paper, we analyse the effect of various techniques for providing additional visual and textual context via tool use to the MLLM and its effect on the REC task. Furthermore, we propose a training-free framework named Chain-of-Caption to improve the REC performance of MLLMs. We perform experiments on RefCOCO/RefCOCOg/RefCOCO+ and Ref-L4 datasets and show that individual textual or visual context can improve the REC performance without any fine-tuning. By combining multiple contexts, our training-free framework shows between 5% to 30% performance gain over the baseline model on accuracy at various Intersection over Union (IoU) thresholds.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
