GPT-4V和Gemini Pro等多模态大语言模型（MLLM）在视觉问答（VQA）中实现人类水平感知方面面临挑战，尤其是在以对象为中心的感知任务中；正如实证结果所示，这类任务要求对对象身份、位置或属性进行细粒度理解。这主要源于它们有效整合复杂视觉线索与文本信息的能力有限，以及潜在的对象幻觉问题。本文提出一种名为联合视觉与文本提示（Joint Visual and Text Prompting，VTPrompt）的新方法，利用细粒度视觉信息增强MLLM在VQA中的能力，尤其是面向以对象为中心的感知。VTPrompt融合视觉提示与文本提示，从文本问题中提取关键概念，并采用检测模型在图像中突出相关对象，将其作为视觉提示。处理后的图像与文本提示随后被输入MLLM，以生成更准确的答案。我们在MME、MMB和POPE三个基准上使用GPT-4V和Gemini Pro进行实验，结果表明性能显著提升。特别是，该方法使GPT-4V在MME上的分数最高提升183.5，并使GPT-4V和Gemini Pro在MMB上的性能分别提升8.17%和15.69%。
英文摘要：
Multimodal Large Language Models (MLLMs) such as GPT-4V and Gemini Pro face challenges in achieving human-level perception in Visual Question Answering (VQA), particularly in object-oriented perception tasks which demand fine-grained understanding of object identities, locations or attributes, as indicated by empirical findings. This is mainly due to their limited capability to effectively integrate complex visual cues with textual information and potential object hallucinations. In this paper, we present a novel approach, Joint Visual and Text Prompting (VTPrompt), that employs fine-grained visual information to enhance the capability of MLLMs in VQA, especially for object-oriented perception. VTPrompt merges visual and text prompts to extract key concepts from textual questions and employs a detection model to highlight relevant objects as visual prompts in images. The processed images alongside text prompts are subsequently fed into MLLMs to produce more accurate answers. Our experiments with GPT-4V and Gemini Pro, on three benchmarks, i.e., MME , MMB and POPE, demonstrate significant improvements. Particularly, our method led to a score improvement of up to 183.5 for GPT-4V on MME and enhanced MMB performance by 8.17\% for GPT-4V and 15.69\% for Gemini Pro.
Keyword-Centric Prompting for One-Shot Event Detection with Self-Generated Rationale Enhancements
State Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University（一般人工智能国家重点实验室，智能科学与技术学院，北京大学）
Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models
Center for Information and Language Processing (CIS)（信息与语言处理中心）；Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）；ScaDS.AI and TU Dresden（ScaDS.AI 和 梅克伦堡-前波美拉尼亚技术大学）；Department of Statistics, LMU Munich（统计学系，慕尼黑大学）；University of Maryland, College Park（马里兰大学 College Park 分校）
National University of Singapore（国立新加坡大学）；Salesforce AI Research（Salesforce人工智能研究）；Institute for Infocomm Research (I 2 R), A*STAR（信息与通信研究 institute（I2R），A*STAR）
EgoDyn-Bench: Evaluating Ego-Motion Understanding in Vision-Centric Foundation Models for Autonomous Driving
Professorship of Autonomous Vehicle Systems, Technical University of Munich, Munich, Germany（自动驾驶车辆系统教授职位，慕尼黑技术大学，德国慕尼黑）；Bayerische Motoren Werke AG, Munich, Germany（巴伐利亚发动机有限公司，德国慕尼黑）；Data Analytics and Machine Learning Group, Technical University of Munich, Munich, Germany（数据分析与机器学习小组，慕尼黑技术大学，德国慕尼黑）
VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
Om AI Research（Om AI研究机构）；Binjiang Institute of Zhejiang University（浙江大学滨江研究院）；College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院）
