大规模预训练的最新进展推动了先进的视觉语言模型(VLM)的发展,这些模型在理解和生成多模态内容方面表现出卓越的能力。尽管VLM具有执行复杂推理的出色能力,但当前模型往往难以有效且精确地捕捉图像和文本两侧的组合信息。为了解决这一问题,我们提出了FineMatch,一个新的基于方面的细粒度文本与图像匹配基准,专注于文本和图像的不匹配检测与纠正。该基准引入了一项新任务,用于提升和评估VLM在基于方面的细粒度文本与图像匹配中的组合能力。在此任务中,模型需要识别标题中不匹配的方面短语,确定方面的类别,并为可能包含0到3个不匹配的图像-文本对提出纠正建议。为了评估模型在此新任务上的性能,我们提出了一种新的评估指标,名为ITM-IoU,我们的实验表明该指标与人工评估高度相关。此外,我们还对现有主流VLM进行了全面的实验分析,包括全监督学习和上下文学习设置。我们发现,在FineMatch上训练的模型在检测细粒度文本与图像不匹配方面表现出更强的熟练度。此外,具有强大多模态上下文学习能力的模型(例如GPT-4V、Gemini Pro Vision)在细粒度组合图像与文本匹配分析方面并不那么熟练。借助FineMatch,我们能够构建一个用于文本到图像生成幻觉检测与纠正的系统。
英文摘要:
Recent progress in large-scale pre-training has led to the development of advanced vision-language models (VLMs) with remarkable proficiency in comprehending and generating multimodal content. Despite the impressive ability to perform complex reasoning for VLMs, current models often struggle to effectively and precisely capture the compositional information on both the image and text sides. To address this, we propose FineMatch, a new aspect-based fine-grained text and image matching benchmark, focusing on text and image mismatch detection and correction. This benchmark introduces a novel task for boosting and evaluating the VLMs' compositionality for aspect-based fine-grained text and image matching. In this task, models are required to identify mismatched aspect phrases within a caption, determine the aspect's class, and propose corrections for an image-text pair that may contain between 0 and 3 mismatches. To evaluate the models' performance on this new task, we propose a new evaluation metric named ITM-IoU for which our experiments show a high correlation to human evaluation. In addition, we also provide a comprehensive experimental analysis of existing mainstream VLMs, including fully supervised learning and in-context learning settings. We have found that models trained on FineMatch demonstrate enhanced proficiency in detecting fine-grained text and image mismatches. Moreover, models (e.g., GPT-4V, Gemini Pro Vision) with strong abilities to perform multimodal in-context learning are not as skilled at fine-grained compositional image and text matching analysis. With FineMatch, we are able to build a system for text-to-image generation hallucination detection and correction.
VLM-FO1: Bridging the Gap Between High-Level Reasoning and Fine-Grained Perception in VLMs
Om AI Research(Om AI研究机构);Binjiang Institute of Zhejiang University(浙江大学滨江研究院);College of Computer Science and Technology, Zhejiang University(浙江大学计算机科学与技术学院)
FINER: MLLMs Hallucinate under Fine-grained Negative Queries
Technical University of Munich(慕尼黑技术大学);Munich Center for Machine Learning(慕尼黑机器学习中心);Helmholtz Munich(海德堡-慕尼黑亥姆霍尔茨中心);Google(谷歌);LTCI, Télécom Paris, Institut Polytechnique de Paris(LTCI,巴黎电信学院,巴黎理工学院)
State-Conditioned Visual Evidence Retrieval for Fine-Grained Perception in Document Vision-Language Models
College of Computer Science and Artificial Intelligence, Fudan University(复旦大学计算机科学与人工智能学院);Shanghai Innovation Institute(上海创新研究院);Shanghai Artificial Intelligence Laboratory(上海人工智能实验室);ByteDance, LarkAI(字节跳动 LarkAI);Shanghai Key Lab of Intelligent Information Processing(上海智能信息处理重点实验室)
AeroRAG: Structured Multimodal Retrieval-Augmented LLM for Fine-Grained Aerial Visual Reasoning
Research Center for Space Computing System, Zhejiang Lab, Hangzhou(杭州浙大实验室空间计算系统研究中心);Hangzhou Institute for Advanced Study, University of Chinese Academy of Sciences, Hangzhou(中国科学院大学杭州高等研究院);School of Cyber Science and Engineering, Zhengzhou University(郑州大学计算机科学与工程学院);School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院)
E0: Expressive Fine-Grained Discrete Action Prediction for Vision-Language-Action Models via Tweedie Discrete Diffusion
Sun Yat-sen University(中山大学);Guangdong Key Laboratory of Big Data Analysis(广东大数据分析与处理重点实验室);X-Era AI Lab(X-Era人工智能实验室);Guangdong University of Technology(广东工业大学)
FGU3R: Fine-Grained Fusion via Unified 3D Representation for Multimodal 3D Object Detection
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications(北京邮电大学网络与交换技术国家重点实验室);School of Computer Science and Technology, Beijing Jiaotong University(北京交通大学计算机科学与技术学院)
Semantic Hardness Is Not Visual Hardness: Sign-Aware Hard Negative Mining for Sign Language Retrieval
School of Computing(计算机学院);Graduate School of Culture Technology(文化技术研究生院);Korea Advanced Institute of Science and Technology(韩国科学技术院);ETRI Medical Informatics Laboratory(电子通信研究院医学信息学实验室)
E4S: Fine-grained Face Swapping via Editing With Regional GAN Inversion
Tsinghua Shenzhen International Graduate School(清华大学深圳国际研究生院);South China University of Technology(华南理工大学);The University of Hong Kong(香港大学);Tencent AI Lab(腾讯AI实验室)
BoxTuning: Directly Injecting the Object Box for Multimodal Model Fine-Tuning
School of Computer Science and Technology, Tianjin University(天津大学计算机科学与技术学院);Faculty of Computer Science and Artificial Intelligence, Shenzhen University of Advanced Technology(深圳理工大学计算机科学与人工智能学院)
Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University(人工智能 MOE 实验室,上海交通大学人工智能学院);Ningbo Institute of Digital Twin, Eastern Institute of Technology, Ningbo, China(宁波数字孪生研究所,东技术研究所,宁波,中国);Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative, Ningbo, China(宁波空间智能与数字衍生关键实验室,宁波,中国);University of Science and Technology of China(中国科学技术大学);CASIA;Tsinghua University(清华大学)
CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment
Goethe University of Frankfurt(法兰克福歌德大学);MIT(麻省理工学院);IBM Research(IBM研究院);MIT-IBM Watson AI Lab(MIT-IBM沃森人工智能实验室);Tuebingen AI Center/University of Tuebingen(图宾根人工智能中心/图宾根大学)